Patrick,
As long as the nock and the point are in line with each other, what happens in between doesn't matter all that much.
That being said, if I have a wood arrow in my hand, I am fiddling with it. I'm like you; they are never "straight enough".
I spend a lot of time on the raw shafts, straightening and smoothing out the kinks and bumps with opposite pressure and sometimes heat. After I stain and seal them, again straighten and smooth. When I finish cap dipping and cresting the shafts, I do more straightening, then again after fletching. I have been known to ignore deer while sitting in my stand because I am busy hand straightening an arrow or two. No, I am NOT obsessed!
I have no idea how straight is straight enough. I do know that within my self imposed range of 20 yards, (with wood arrows anyway) I am consistent on a target enough to be confident in the field after game. I have a dozen ash shafts that are still in the raw. I’ve had them for two years now and one day expect to have them straight enough to move on to the next step.
I don't use glue-on points anymore. The poor fit of the glue-on points accounted for more loss in accuracy than most of the bend, curve or kink that is left in the shaft. I leave the end un-tapered and epoxy on a short length of aluminum arrow with an insert. That way I can use screw-in points and broadheads.
Even carbon shafts aren't perfectly straight. I guess straight enough for wood shafts is when you get tired of fooling with them or they go where you want them to out of the bow. I settle for the latter.
OkKeith