I look at sound much like sent. The deer in the areas I hunt are use to some of both in the areas I hunt. When relaxed, they are not bothered by a little of either. But a big dose of one will get a reaction. I don't need to have a bow so quiet the deer hats nothing, but I don't want to drop a fire cracker on them.
I see no down side in a quiet bow unless you kill the performance to much and get to much arrow drop at your effective range. That is not an issue for me most of the time. If my arrow does fall short, I would rather the deer was still standing not alarmed.
I doubt the performance level of most bows used by hunters long ago shooting stone points was such that they needed silencers. Many modern long bows don't need them.
And, who says all the glass bows in recent history were that loud? I have some old glass recurve bows I hunted with 35 years ago. With original strings and no silencers, and wood arrows from the time, they are as quiet as any modern well tuned recurve with fast flight string, silencers, good weight arrow, and good tune. I never used silencers in those days and would hunt with those bows without them now.