I think, first, you need to have a realistic expectation of what kind of impact you really have. 80 acres is small by management standards. Most say you need over 400 acres before you can effectively "control" a group of deer. Their range is just too broad. Second, there are a lot of things that affect antler growth; nutrition, pressures/stress (overpopulation), and even another dominant buck can suppress antler growth in sub-dominate bucks, and probably the #1 factor AGE, followed in a distant last - genetics. Most people don't have the resources (acreage, fences, restricted hunting) to effectively fiddle with genetics. I wouldn't worry about genetic "potential". There are plenty of examples of pen raised deer going from a spike to world class if given two main ingredients: time and nutrition.
I would concentrate on taking a certain age range if you really want to improve the bucks in your area, as much as you can on 80 acres. Learn to tell a 4.5 year buck from a 2.5 year buck, decide what age structure you're going to take out and let the antlers be what they are. Reducing the number of does in your area, if you have too many does, helps improve the habitat and the quality of nutrients available for the bucks that remain. Create areas that the bucks feel safe (1 acre thicket you NEVER enter - buck sanctuary) to attract and keep bucks on your property. You can improve nutrients by simply fertilizing native food species already on your property as well. Plenty of nutrients help a buck come out of the rut in better shape. The better shape he ends the rut the more energy he'll likely have to survive the winter and put into next year's antlers.
The main ingredient is age and the willingness to accept that come rut he can be a mile away and shot by someone who doesn't hold the same higher standard you do. Don't stress about it too much and just take the deer that you want to take and let the rest walk; to either be appreciated by another hunter or to live to see another year.
If you want more detailed information I'd suggest ordering and reading Dr. Kroll's
A Practical Guide to Producing and Harvesting White-tailed Deer
.