I don't even look at antlers much anymore. I can't even remember why I used to, except that it was the "mystique" of hunting when I was growing up; all the hunting stories and such, and the guys that always got the biggest racks were always the "most respected." Maybe it has so much to do with ours becoming a Leisure Society. We have time and money for things that no previous generation could have conceived. Instead of "success" meaning that a man fed his family, it means a whole gamut of other tings, so the only way to reward a man's (and to a leser degree, societally, a woman's) hunting success is with "trophy status." Look at sex, even -- it's been a long long time since it was almost solely for procreation, and occasionally there's been times in history when it has nearly gone back to being just that. So instead of the man with 15 kids being thought of as "virile," or a "stud" (I hear that today's kids use the term "pimp" in place of the older word, "stud"??), today it's usually the guy with no kids and twenty girlfriends, or it's measured by "size," or the gal that's got the hottest body (and WOW do definitions of "hott" change from generation to generation!!!) being the greatest trophy.
So for me, I hunt. And like you, I have nothing whatsoever against trophy hunting or trophy hunters. But I do resent the misguided attitude of those that judge others by their own ideas of what make for a successful hunter, or hunt for that matter. Would I pass up a trophy rack? Never. And it isn't a sour-grapes attitude driving my views, it just amazes me to watch the stampedes in every genre being driven by advertising campaigns. The whole "gotta have the newest-biggest-bestest-most expensive" keeping up with the Jones' thing has been going on for what seems like forever.
As for it upsetting The Natural Order, perhaps it's the natural order of man that stands to lose the most when all is said and done. As far as the game is concerned, it has an almost opposite effect on wild game that's hunted than it those species which we've domesticated. How many breeds of dogs and catle are there now, anyway? We can manipulate things our way. But as for wild game? Those that we hunt during the day, are tending toward nocturnal behaviors. Thos with big antlers? Well, we seem to instinctively KNOW where that might lead, so we inplement such programs as QMD, food plots, etc. in order to cause the swing to remain in what we regard as "favorable."
Am I on;y in favor of killing the old, the weak, and the young? No. And I'm not going to be an hypocrite about it, either... there's a ton of big bucks around here, and I've more often than not let pass smaller deer and does in favor of "The One" perchance ambling on by. And as for specifically targeting one animal and devoting one's attention to that critter alone, again - nothing wrong, in my opinion, but still it is the result of having time and resources that allow an individual to do that. Not needing to put meat in his family's belly allows a man the freedom of that type of pursuit.
So, as I started out saying, it's more that man's order of things has changed man, and the trickle-down naturally changes the world around him. Just my two cents with a few more thrown in for inflation...