The ability to handle the bow, to me, is the deciding factor. If I can draw and hold for 10 to 12 seconds with good form and shoot comfortably for 25 arrows in a session, I feel that I am not over bowed.
As Rob stated, that poundage is not always totally static. In warm weather, standing upright at the target line does not equal the ability to pull that same bow in an odd position ( as might happen in an elevated stand) in freezing weather.
I can now, following shoulder surgery, shoot my 65# bow just fine for about 20 shots, but I can't do it nearly as well in the cold. In the past I could shoot it and my 78# bow for many, many shots. Now, I can't even string the 78# bow. My 53# bows are just about perfect for me now, so that is now what I consider a proper maximum hunting weight for me.