Reddogge is right, but you have to force yourself to concentrate on the one spot you're trying to hit.
If you let your focus wander at all, your brain may well pick something more interesting to look at. I've had it happen when I was shooting through cover and realized from the shot that I hadn't been focused on the target completely, but also partially on some branch I was worried about clearing.