I would be willing to bet a dozen doughnuts that you do actually use a reference point. I know that I subconsciously use the tip of my arrow. I don't notice it, but as I am burning that hole in the bull, my secondary vision is picking up that tip and the coconut computer is calculating a gap.
If you were shooting at say, fifteen to twenty yards, and it were totally dark except for a flashlight pointed at the target, I bet you would find yourself looking for that arrow tip, especially a target tip that did not extend past the riser, silhouetted against the light on the target. Ask me how I found that out.
I would also wager that you can throw a rock pretty much where you want it to go, and that uses no reference point.
I am left eye dominant, and shoot right handed. Fortunately, my left eye is not too insistent, and is trainable. I spent a few sessions concentrating, and that seems to (mostly) put things into perspective. One thing you might try, if you wear glasses, is to put some matte-finish Scotch tape over the left lens. Right eye takes over as the left eye mopes and grumbles about life not being fair. If you, like me, do not wear glasses to shoot, try an eye patch, or close the left eye. The patch will be less work for you, and less distraction. Eventually the right eye will stand up and shoulder its share of the load.
Killdeer