My guess would be because you took yourself from "binocular" vision....to "monocular" vision. Are you right eye dominant?
If this is happening on a regular basis, I would try changing out your sideplate, to a softer type material. Something like female velcro, and see if your arrow doesn't move back to where you are looking. You could also go to a heavier weight tip, which would weaken your spine. I have this problem "occasionally" and when I do, just closing up my stance brings me back to the right. If it is a "regular" occurance, I would change one of the above, at a time.....going from the easiest first and see if the problem doesn't go away.
Taking an educated guess, I'm saying that you are shooting carbon arrows.
Winterhawk1960