A good vest (or two), whether wool or other, keeps your torso warm and keeps your arms bulk free. Try it , it really works. Wear a decent hat, put a cloth face mask on, one that covers your neck and keeps the wind out. If it is windy, make use of golf shirts, the kind that they wear that keeps the wind out, not the short sleeved Izod kind, but wear it under an outer layer to keep noise at bay.
I like wool and wear it a lot, but other materials work fine too, especially if you are hunting down south.
ChuckC