Hi Tim,
As far as this topic goes, opinions are like As@@@@@s, we all have one! That being said, I would have killed this little buck. I, just this season, broke a long dryspell by killing a young button. Wonderful deer, I was SO tickled and the meat has been excellent. I needed the confidence boost badly.
I think that for your own confidence you should have taken the shot. Kill that 1st and then be a bit more selective if you want. But lets face it, at this time in our history, we have an overabundance of deer roaming around. And as mentioned, yes the deer would probably grow a nice (if typical) rack in a couple of years and provide, in many folks mind, "better" trpohy. But, realize that the bucks just dont "grow bigger" they grow exponentially "smarter", and the likely hood of you even seeing him again is very small in all reality. We hunt with trad-bows and it's not likely we could ever seriously impact the "herd" by "over-harvesting".
I, myself wouldn't sweat it, take the next deer (whatever size) and be thrilled(you should be as you did it "the hard way")don't over analyze this too much. Our time (for most of us) in the field is very limited. Enjoy it fully. Enough preaching!!
Good Luck
Harry.