Well I'd like to preface this by saying I don't really care personally. But I don't think peoples opposition has been very well articulated so I'll be devils advocate for the sake of argument...
In todays age of people being alienated from hunting, and nature in general, a question people are often left with is: "Why do people hunt?". People really don't know or understand anymore. The reasons people hunt are as diverse as the person hunting I suppose, but the average person is left to make up their mind from fairly little information on a passing stranger. Other than that he is wearing camo, an orange hat, and carrying a bow they don't know anything about you. They might assume something pretty negative, like that you enjoy hurting animals, or they might think something less outrageous like you enjoy a few hours in nature for peace and quiet after months of work in a factory or where ever.
The problem is that a fairly ignorant, but apathetic segment of society (nonhunters) controls all of our destiny as hunters by what they think and vote. When folks see guys dressed in tights and chain mail and referring to themselves in medieval character names people chuckle and think they are eccentric, but living out their fantasies isn't hurting anything but their social status. But when they find out they aren't going to archery ranges, but are going to hunt animals. Not only that but they have higher levels they reach by killing more animals like characters in a Role Playing Game. It starts looking like maybe it isn't harmless, maybe it's adults living out fantasies at the animals expense. Given what people see about canned hunting operations and so forth you can imagine what snowballs from this direction of thought about hunters and hunting. It's better they stay apathetic than to have them become anti hunters.
Hunters have a lot of pragmatic and realistic reasons why it is okay to hunt. In almost every instance wild animals lead a much nicer life than those raised to be slaughtered. Our dollars make it economically viable to preserve the habitat for these very creatures and thus actually protecting their herd numbers by being able to hunt them. A bulldozer can do much more damage in a month than the most unscrupulous hunter ever could in his lifetime. But these and the other more intellectual ideas take time to explain, and they can fall on deaf ears if people have set opinions and have something memorable to hold onto like a guy shooting squirrels out a tree with a home made cross bow while dressed like robin hood. When people see the average hunter they have to guess why he hunts, and maybe even ask him. When they see you dressed like that they have already made up their minds and may even purposely avoid talking to you, much less giving your opinions much credence if they do.