Ian,
Being as you live in La. and it isn't Sub Zero cold outside.....(although I DO understand that it gets COLD down there)
Go to the Good-Will store and get a few 100% wool sweaters.Layer 2 or 3 under your regular cotton camo and you will be warm as a toaster waffle
My legs never get cold so I rarely wear more than a pair of cotton camo jeans or cargo pants. However, I will layer my upper body with wool sweaters and cover them with a cotton camo shirt. The tight weaved cotton also acts as a wind stopper. Up here in the NorthEast,I can NEVER remember having to wear more than 3 sweaters,and I have never been really cold.(just a little chilly on a few occasions)
Here's a helpfull hint.....
If you possibly can,eliminate the cotton T-shirt and wear your first wool sweater against your bare skin.It will make a BIG differance if you start sweating as it will "wick" the sweat away from your skin. Also the "itchy" feeling stimulates blood flow to the skin.