I haven't experienced any allergy issues with the wood but it's just like you said Troy - different woods affect people in different ways.
I live in Madison, Alabama which is just to the west of Huntsville, Alabama. Huntsville (Monte Sano) is pretty much in the center of it's limited range in our pocket. Most of the land on Monte Sano and surrounding peaks are owned and protected by the local Wildlife Trust. While this is a good thing for the environment, it does make obtaining the wood extremely difficult. There is a select group of volunteers who collect fallen smokewood trees every year. Most of the plunger is in the form of fallen limbs and only good for very small projects. I wish I could get my hands on more material but it's just so hard to come by even here.