My shooting doesn't lose much speed at all. I lose quite a lot! :D
In all seriousness, it sounds like you've already heard the real, hidden issue: you probably aren't getting the last inch or so of draw. It will FEEL like you are (perceived effort), but you probably aren't due to extra clothing. Now, that's if you are using modern fglass bows. With a selfbow, esp a water-loving wood such as hickory or dogwood, it might actually be that the bow has absorbed moisture from the cold humidity and has grown a bit more sluggish.
This is preferable to the other extreem, if you heat with wood and keep your bows inside. In this case, the bow sometimes explodes in your face due to excessive dehydration from very dry wood heat. Don't ask...