I agree with WVbowhunter that what you are experiencing fits the definition of target panic. However, there are many degrees of target panic, ranging from mild to severe enough to justify the name. Let's assume for the moment that you are experiencing one of the milder forms, and just deal specifically with the problem you describe. It won't hurt anything to try.
Arne Moe recommends not focusing on the target until you are at full draw and have satisfied yourself that everything in your form is as it should be. He recommends this for everyone, not just those with target panic. He recommends drawing the bow horizontally, with your bow arm and torso in a perfect T configuration, regardless of the position of the target, and then adjusting to bring the bow on target by tilting and twisting at the waist, maintaining the T configuration of your bow arm and torso.
I used to draw the bow while looking at the target, which put me pretty much on target when I reached full draw, but also led me to target panic. Drawing the bow without looking at the target helped me to avoid target panic, but then I experienced the jerkiness you experience in aiming. Drawing the bow horizontally, like Arne suggests, required me in most cases to make some pretty major movements to get on target, which actually turns out to be easier for me than making minor adjustments, since I'm more inclined to tilt and twist at the waist making major adjustments, and once I get that movement started, it moves smoothly enough. With minor adjustments, I'm more inclined to move my bow arm, throwing my T out of alignment. If I get to full draw and find that I'm already close enough to being on target that only minor adjustments are needed, sometimes it helps to make some fairly major adjustments back and forth across the target by tilting or twisting at the waist to break things loose so I'm moving smoothly before my final aiming focus on the target.
I don't think you need to draw horizontally for the rest of your life, or in hunting situations where you want to avoid any unnecessary movement, but it is a good exercise in maintaining the T and the major movements needed to get on target do seem to help in avoiding the jerkiness you describe.