Some of my most memorable hunts of the past 50(!!!) years are the ones where I chose NOT to take a shot. You'll remember that evening fondly after you take your first deer later this season.
Also, years ago, I allowed a young man permission to hunt on my farm, even though his family owned property (and would not allow hunting). I told him I had only two rules he needed to follow:
1. The permission was for him only, and he was not to bring anyone else on our land, and
2. I did not want him to nail any homemade stands to the trees, since another friend had previously done so to three of the best stand trees... and each had died.
He agreed, and I wished him well.
10 days later, as I was busy harvesting a field, I saw him walking across the horizon with a friend. They were far away, but I could see both had bows, and they were carrying something. That evening when I had finished for the day, I went to check things out.
I found a new wooden stand nailed into my favorite stand tree.
The next morning I saw him walking across the field and was able to drive over and say "Hi." He looked very nervous, and replied honestly when I asked if he had brought a friend and nailed up a stand.
I told him I had asked him to follow only two rules, and he had ignored both. I told him to go ahead and hunt the rest of that morning, but that was it, and he would no longer had permission to hunt my place. He agreed, apologized, and I wished him well on the morning's hunt.
I tell this story simply to say you acted wisely in honoring the request of the guy who holds the "key" to your access. We never go wrong doing the right thing...
By the way, that tree died, as well...