Try to seal the wood material after your sanding at a lower grit (a sealer such as super glue, high solids vynl sealer (my favorite on bows), or even shellac should work), then proceed to sand to your finer grits. The sealer will simply seal up the pores thus the contaminate cannot "get in".....of course if you sand too deep, you need to re-apply a sealer.
Also, a stiff backer material behind your sandpaper will make sure the hard aluminum, and softer wood surfaces stay flush with each other.