Yours are probably ready for replacement, but Bill's got it right. Some materials need replacement more often than others. As for wearing on the bow, I suspect you'd have to shoot an awful lot of arrows for that to become anything other than a cosmetic issue. The biggest issue I see with what you've described is that your arrows are contacting a whole lot of the rest and side plate material. A single point of contact is best as it will affect arrow flight less. If you're rubbing all over the arrow, you could be adversely affecting your arrow flight. On the other hand, if you're hitting what you're aiming at, why fix it if it ain't broke! If it sounds like I'm saying there's no absolutely right or wrong answer to your question, you're right, but personally, I'd replace the rest material and the side plate.