I think you just answered your question. You are moving your eye off of the spot on the target. For instance... the brain cannot concentrate on THE SPOT while it is also searching for the arrow shaft in your perphipheral vision. Trying to look at two things at once is impossible.
What has helped me is pretty much what Jerry said. When I come to anchor, I pause, hold at full draw, focus on the spot, release, head and bow arm do not move until I see the fletch in the target.