I answered yes color and pattern both matter, but with certain reservations. This is only my opinion, but it is based on numerous things I have read over many years. Your problem may well have been "color" based on dyes with a high UV rating. Deer see further into the UV spectrum than people, and the brighteners in most clothes dyes contain UV enhancers which may make this an issue. Many of our clothes, due to these enhancers, glow in a deer's vision. Is brightness/darkness considered color or pattern? If the camo coloration strongly and "glowingly" contrasts with your surroundings, it might make you tend to stand out. I feel the pattern itself is less likely to be a problem as long as it is subdued.
In one article I read quite a long time ago, it said that military camo is specifically made with dyes that do not "glow". I believe it, because, for many years, I only used the old style military woodland camo from early season (green woods) till the leaves were completely off the trees (brown woods) with no problems. On the other hand, most commercial dyes, including those used in civilian produced camo, contain these color enhanced dyes.