Down when it's cold out, actually when it's cold enough for down you don't have to worry about getting it wet. A full day of sweating won't dampen down enough to fully destroy its abilities. If it's rainin out put your down away or put a rain coat over it to keep it dry. My daypack always has a down vest in a little tiny stuff sack, it weighs next to nothing and doesn't take much room. If you put a down layer under a quiet over layer the usual nylon shell isn't as noisy.
Wool stays warm even when it's wet but if you ever get it totally wet (soaking) it weighs a ton and is almost impossible to dry overnight in a camp situation. The one possibility is a wall tent with a woodstove but even then it takes a lot. In that situation down would totally let you down
as it's very difficult to dry without a tumble dryer. There is a new fangled down that's waterproofed now, I'm not sure How well it works though.
Fleece, holds odours in my experience, some of it is quite noisy especially the wind proof stuff but, it's easy to dry and can be wrung out and worn.
So like others have said a combination of the three in different layers works well. It's a huge mistake to simply discount any of these because of one drawback.
Don't forget about fur though, in cold dry circumstances it's the king of warmth and quiet.