The world is changing, every day, and has been since the beginning. That includes animal or plant populations and distribution. Changes in weather, feed availability, predator population, new predators, new diseases and many more will all affect the deer (and everything else). When the stars line up and several of those changes put deep pressure on the animals of choice, the population drops. If any have the traits that give them an edge over the adversity (and the others), and with a handful of good luck, they tend to prosper and multiply. Over not too many years much of the new population has those traits or even more fine tuned versions of those traits. Then, again the herd might prosper.
Until along comes yet another big change. That is the basis for evolution and is very discernible if you really look.
We are a part of it and a very major ( maybe THE) predator in many areas as well as "changer" or "cause" of changes. It was brought up above. Our quest for controlling the genetics of deer... making them "better" sure seems to cull for only one thing... antler growth. Of all the things a buck might do, I am gonna guess that is very low on the totem pole regarding survival, and ....
in fact.. in an area where there are minimum antler restrictions, say 8 point or better, a genetic anomaly of having a larger 6 point rack would be a distinct advantage for survival. Are we culling out the biggest and best ?