Bill, I'm still fine tuning mine but have had the same experience so far as you. I've got the bow arm completely bare and working on a permanent solution (much like yours with elastic) to keep the extra material out of the way. The bow arm area is the only problem I've had with clearance. There was also so much extra mesh material on the arm, it bloused out at the cuff area actually obscuring the view of my draw.
I've got the seperate booniehat/hood and tuck the string behind my left ear so that when I turn my head, the hat turns too..otherwise the hat stays still and you are looking out the side pieces trying to shoot (don't ask me how I know :p ) I like your elastic idea there and may get some of the big wide stuff to get a firmer grip on my noggin and try that too. I also cut a good bit out from the front of the hat, again, because of obscured vision.
I am convinced this is an excellent way to hunt, my few trial runs with it in WY at Vance's drilled that into my thick skull. As soon as it gets cool enough here to try it, I'm gonna spend a good bit of time in it and actually kill something with it from the ground (in a bowhunting only area of course).
If I get this thing worked out and it turns out to be as good a tool as I think it will be, I'm gonna build one myself and add in a bunch of blaze orange for use during our cooler gun seasons. Bow hunting from the ground during gun season around here in a ghillie, even on private property, is rolling dice to say the least.
Good luck, let those of us still trying to figure it out know any thing you learn.