I shoot aluminums and for me, there is virtually no downside. They sure do bend if you hit the wrong thing, but the answer is to make sure you DON'T hit the wrong thing! I have ruined a lot more shafts by shooting them with other arrows, than I have by bending them.
I currently have about 15 dozen shafts of 3-4 different sizes, all of which I have bought (unused) pretty inexpensively at garage sales, clearance sales, and online. I fletch all my own arrows, and find them very easy to work with. My completed arrows - not including points - cost around $3-$4 on average. I shoot a lot, but I haven't bent many shafts in the past 3-4 years... probably fewer than a dozen. But then again...
1. I shoot a foam target, with a relatively soft backdrop, or
2. I stump shoot at clumps of grass, dirt, or muddy banks -- never wood or stumps unless they are severely rotted, or
3. I shoot at tennis balls or milk jugs or an old soccer ball in my backyard.
3-D shoots are where I bend the most shafts, because I can't control what is behind the target, in case of a miss.