Lots of options. Throwing some out to you. You are right, the resistance to wind can make or break your clothing warmth. I have a set of golf "wind shirt" and pants that I sometimes wear under a layer, to break the wind. A vest adds chest warmth with no additional arm bulk, which gets tough on bow shooting. Keep your chest warm and you hardly need sleeves.
I have, in past days, opted to wear layers of sweaters and vests with an XXL or larger camo T shirt over the top. Cheap, kept me warm, worked. Remember the thing about sleeves and bulk. You don't need a lot on your arms.
Of course, then there is good quality wool. . . .
BTW, an investment in good quality wool under drawers and shirt is really worth your while.
ChuckC