I understand what you are saying about accuracy. Shoot the most accurate set up, period? Should we drill a hole through a yew self-bow, put in a brass bushing for a Berger button and cavalier rest? Probably not, but someone has probably tried it. Not fitting to that style of equipment, and really probably not going to improve the accuracy of that style of equipment.
Some like an elevated rest and are more accurate with that. Really there are other issues than the rest to tune a bow and arrow combination. Longbows are shot off the shelf, by and large. Rest are more an exception than the rule in longbows. Longbows lends itself to that method of holding the arrow.
For target accuracy look at some of the scores that Horace Ford shot that stood for so long in the FITA. He was shooting off his knuckle with a string follow english yew self bow. He was an exceptional archer, but accuracy can be obtained if the archer is willing to work at the tune of their bow and arrow combination regardless of arrowrest.
We all should strive towards an accurate bow and arrow set-up. As a archer, whether that is at game or on the course. It is just so much more enjoyable of a sport to shoot a properly tuned bow and arrow combination, and you limit yourself so much without it.
On the other hand, a recurve can be shot either way, and for many including myself, it is much easier to find an accurate tune with a flipper or brush rest on that style of bow.
You can achieve great arrow flight and accuracy with both methods.