As you say, it doesn't affect accuracy, since the feathers were conveniently engineered to fold in when they contact the bow. Black Widow recommends orienting the cock feather up, and having a hen feather slide into the corner between the riser and the shelf. Fred Asbell taught us to nock the arrows without looking at them, as it was his feeling also that feather contact wasn't really that important. So one possibility would be to just shoot with cock feather in and spread the wear around.
Assuming you are now shooting with your nock point in the correct position, you could raise your nock point a little (maybe 1/8"?) without hurting accuracy, as the feathers will stabilize the arrows quickly after they leave the string. I don't know if that would solve your problem or not.
The only real problem with feathers, other than wear, is if you have quill contact, which will cause an arrow bounce and will affect accuracy. You can tell if you have quill contact by trimming the feathers on one of your arrows back to maybe 1/8" and marking the edges with lipstick or something to see if you have any contact with the bow. The arrow you trim will fly essentially like a bare shaft, so might be useful in the future for tuning purposes.