This is Terry's Clock, which is often referred to in this forum, to illustrate proper shooting allignment:
What may be happening to cause your problem is that your shoulders are not properly alligned parallel to the arrow. Instead, they may be alligned so they point off at an angle to the left. As Old York said, if you set up with your feet perpendicular to the target, it helps you to get your shoulders into proper allignment. However, even with your feet in that position, you may tend to draw the bow, particularly when you start your draw, with your shoulders pointed off to the left. Shooting "in the bow" means that you end up your draw with your shoulders in proper allignment. Part of pushing with your bow arm is really rotating your shoulders into the bow, to get that proper allignment, and you can do that from either a closed or an open stance. Note that Terry is in an open stance in the picture, but his body is still properly alligned with the bow and arrow.
I prefer the open stance, and then twisting my upper body into proper allignment, since I don't have to worry about where my feet are in a hunting situation, and twisting my body into allignment doesn't make any noise, whereas shuffling my feet around does. However, this does add another variable, which you probably don't need if you're just in the process of learning proper body allignment.