I just did the same thing to a bow my Dad purchased in 1960. As long as the handle was made with enough laminations, as his was, then shaving a bit of wood here and there should not effect it in the least.
Agree with post above, don't touch the back fiberglass.