In my opinion:
The deer first see you as a lump that they normally don't see in that location - and they know what they expect to see. If you move before they have accepted you, or disregarded you -- well, you are busted because you moved. Once they accept you as nothing important to them, they will disregard you and you can move.
Pay close attention to their body language. You will notice that there is a point when they decide they are, for the most part, safe. You have to stay completely still until that point. Of course, if there are multiple deer they won't all relax at the same time.
Remember, a deer is a prey animal, and their eyes are made to pick up and interpret movement. They don't have the ability to reason out what you are if you don't move. So, don't move until they disregard you.