I would guess you are doing something slightly different since putting the quiver on the bow. Perhaps you're putting a little thumb pressure on the grip to offset the weight of the quiver, thus torquing the bow a little. Maybe you're noticing the arrows in your quiver and that's causing you to move your head/eye a little away from being directly over the arrow. I would just go down the checklist of alignment and form, because whatever you're doing is evidently too subtle to notice. I doubt if it has anything to do with the quiver per se, since you don't have it mounted on any working part of the limbs.