Do a goggle search on wood dust toxicity, you will come up with a bunch of lists rating the more hazardous woods.
Each person reacts differently to a toxic dust. Ipe might not bother some folk but a friend had such an allergic reaction to the dust that he went into anaphylactic shock.
Micky Lotz had to quit making bows because of an increased sensitivity to osage dust that made him very sick.
Osage dust gives me hay fever, same with walnut dust but a little worse.
According to the toxicity charts Ipe, yew, walnut and to a lesser degree osage can cause people problems.
More exotic woods from south America are much more toxic than our local woods.