I haven't decided to make the jump yet, but I know what you mean by having more fun shooting. Since I've picked up a recurve (2 months ago) I've been shooting everyday. It gets frustrating sometimes, but is always fulfilling.
Every once in while I'll take out my compound and shoot a couple of dozen. Not fun, but instant accuracy. The only problem with that accuracy is fine balance it takes between the chain of mech. aids it requires, (i.e. Release, Peep, Sight, drop rest, Rangefinder, etc.)
This is my first year archery mule deer hunting and I'm just not confident enough yet to make the jump. It would feel great to get rid of all that stuff, just waiting to break or misalign, and just hunt but I don't know if my paltry 15-20 yds is good enough for the alpine mule deer hunting I want to do.