Camos, plaids, dull solids, I wear them all. One thing they have in common is I wear what is appropriate for the type of hunting I am planning (sitting vs. moving, warm vs. cold, etc.) so that I can be comfortable - which means keeping still or moving quietly.
If you are cold you'll be fidgety - and no matter what you are wearing unnecessary movement will blow your cover. A deer is just as afraid of moving stumps as they are humans.
Dress warm in the winter, but if you are still hunting that is much different than if you are sitting in a stand or blind. Layers can help you regulate your temperature.
And no matter what you are wearing - be conscious of the wind and keep the deer upwind of you. They sometime pause to get a better look but they ALWAYS bug out when they wind you.