You could try scraping the glue away with a sharp knife or razor blade. I did that on a piece of cherry bark I glued with Titebond 3 on the back of the handle area. Most of it came off, but it took a lot of careful scraping and sanding.
Even if it's a stiff handle design, there's going to be stress at the handle fades when you're drawing it back. I'd want good, strong wood there.