I "used to" have this same problem. It was to the point where my hunting buddies were about to get physical haha! The idea was: If the oldest or most beautiful showed up; it was all or nothing because I'd rather have taken the chance than wondered the rest of my life, "What if?"
But in the last few months ... creating opportunity was so much hard work that the "idea" changed. It is now : "If this other monster didn't exist right before my eyes, would I still shoot this smaller one?" If yes, nocked and drawn. If no, get the camera and snap some pics!
Hope you get that big boy!