I think a lot of us have that problem, Sirius. We play different games to get ourselves to hold at anchor for whatever time we think is best. They work for a while and then they don't. I have a metronome app on my iPhone that I use in practice to hold for 5 seconds, twice as long as I want to hold in general. I have no problem holding for the 5 seconds in practice, and then I go right back to rushing it when I'm not timing my shot with the metronome. You would think that I could just count out the seconds when I don't have the metronome; it wouldn't be exact, but it would be close enough. But for some reason, it's a lot harder for me to count out the seconds myself than it is to listen to the metronome. Not impossible, I can make myself do it most of the time, but a lot harder for sure.
It shouldn't be necessary to time the hold, whether by metronome or in my head, because the number of seconds is only indirectly related to why I'm holding, which is to stabilize my shot. Ideally, I should just be able to observe my sight picture and know that it's time to shoot when it's stable, regardless of the number of seconds that have gone by. However, whenever I try to do that, I end up rushing the shot for sure, so I seem to need some arbitrary control over it, or I don't end up holding at all.
The only consolation I can take is that my heroes seem to feel the same nervousness when they hold as I do. I've noticed Rick Welch flinch and recover, flinch and recover, and then settle down and shoot. You can observe Byron Ferguson do the same thing on more than one shot in his video, "Barebow 101." In fact, one of the things I like about that video is that he doesn't edit out all his bad shots, which gives me hope when I make bad shots of my own.