I don't go to 20,000 feet, but on numerous Colorado hunts up to around 8,750 and camping at 7,500, I never felt the burning need for anything other than my cotton pants up there. It was just too hot for anything else during the day. Sure you can get wet, but the normal air up there is dry as heck and warm or even dang hot during most days. So far I have yet to see any issues with the cold (my opinion), and actually, I am going the other way this year. I bought extra light weight, uninsulated boots, I am considering short sleeve camo tops, no felted brimmed hat, but rather a do rag. I get too overheated up there. Of course first thing in the morning is chilly, but half mile up the trail I am very warm. In the evening, when the sun goes below the horizon, you can feel the temperature drop. Yes, it gets cool, but a decent wool vest that was in my pack always seems to do it. On one occasion that I recall I got cold, and I put on my rain gear and solved it. Again. . that's just my experience.
ChuckC