I would use only leather out of the butt for your washers and press it even firmer before punching it out. A old book binding press works good for this. The leather should be cased before going into the press.
If you don't do this the handle will get loose with time. ( it will anyway but this helps)
Start with 12 oz leather and press it down to about 7 oz. I've had good luck with Herman Oak skirting.