Sam, one of the main issues is that animals like cows and deer are ruminants. Unlike other mammals, they digest food by essentially fermenting (or ruminating) it in their stomachs. "Sweet" foods like grains turn sour in their stomachs and can cause health problems and change the composition of their meat (especially in terms of the balance of Omega 3/Omega 6 fatty acids). This is very well documented in cattle, and I'd wager it can be seen in farmed deer herds as well. These animals simply are meant to eat green, leaf-based diets (sometimes supplemented by grains etc.), not diets dominated by grains. This occurs whether or not GMO's are present.
There are many reasons why animals that eat grain-based diets have meat that is fattier and gain weight faster (and the opposite is generally true of wild game or grass-fed meat livestock). The availability of grains in the Midwest has certainly contributed to the monstrous bucks that live there.
The above is of course a simplification, and there are many factors one could consider when choosing what to eat or what not to eat. Wild deer are probably more wholesome than other options, but the quality of their meat is certainly affected by their feed.
Great conversation!