I know a guy works with all kinds of wood.
He daren't go to his own shop when they are working with ipe and a couple of the guys working there get days off depending on the wood they are working on, even with some high-end dust extraction in place.
Wood and allergies seems to be a bit of a crap shoot.
Go carefully.