YA can remove the first bunch of material pretty fast with the bench sander, but when you have the basic round shape it's time to S.L.O.W down . Its easy to remove too much material as you found out . I have 2 wooden blocks tied together with a door hinge and a 23/64 hole drilled down the middle. I stick the footed end in the block( along with 180 grit sandpaper) , the other end in a cordless drill and start spinning the shaft, continually moving the shaft back and forth throught the block. You will remove alot of material this way too, so go slow and check often to make sure you have not taken too much material off. I have a 4x4 steel plate with an 11/32 hole drilled in it that i test the footed shaft in. When it just fits into that hole i stop sanding( or go to some very fine sandpaper to eliminate any scratches) .
Hope this helps, i am not the best giving instructions. Bill