Sequoia: Maybe your ears aren't that good. I've stood behind backstops at 20 yards as well and listened to people shoot. Could always hear the bow go off. Regardless, a critter's ears are probably better than both of ours.
By the time you hear the arrow, it's at least half way to the target. If the critter doesn't start moving until then, you'll get him if your aim is true.
At longer distances, 30, 40, 50 yards, for example, feather noise can give a critter plenty of advance warning, but at close range, they'll be hearing and reacting to the bow going off before they hear the hissing.