Luke, short answer, no you're not an idiot (unless I am too, which is possible, cause I've had the same problem).
I have experienced some issues like that when I didn't ensure my shaft/insert were perfect. Ever since I started using the G5 arrow squaring device, this problem only occurs once in a blue moon. This little tool makes sure the ends of the shaft, whether carbon or alum are perfectly square. That said, if the insert is square, and the broadhead adapter is square at it's base (and I have seen both of these out of square from time to time), a true spinning broadhead should be capable of switching shafts without any change in it's true. As an aside, I've found the brass inserts and steel adapters to be more consistent in their straightness than aluminum.
Hope that helps.