Good call. I passed on four different bucks last weekend. I don't regret any. Two were fork horns. One was probably around 120 or so, but only about 2 1/2 years old...15 yards, broadside. I just couldn't take him because of his age.
The worst one was a huge, old buck, 150+ and VERY wide...but it was getting pretty dark, he was 25 yards out, and he was quartering slightly to me. I passed, reluctantly. The next morning, back in the same stand, I could see some pencil-sized limbs that very well could have deflected an arrow. I'm glad I didn't attempt the shot. It was the right choice and so was yours! The results could have been similar: a wounded and possibly unrecovered deer.