You can modify the shelf without too much stock removal, and without really effecting the bow other than fletching clearance. It doesn't seem like much but it can matter on fine tuning.
You want for the arrow to rest directly above the deepest part of the grip, as shown. Make a relieve cut down into the shelf first (second pic), very modest. Then work down with a cut lowering the shelf, working carefully to the kerf you established already. Then work the fillet (rounded corner of the shelf and riser) with a round rasp. Then work the surfaces down to even with the fillet.
You can also take a little width out of the bow at the arrow pass, increasing it's centershot. Not necessarily a good thing. A little center shot, versus being cut past center, actually helps make a finger shot bow more forgiving, imho. You want to be able to anticipate consistent flex in the shaft on the loose and a little proud of center adds and guides the forces into the flex. So, be gentle with the stock removal, in fact avoid the area.
Some tools. A 1/2 round double cut Nicholson and a #50 pattern makers rasp. I use a single cut mill bastard too for fine cut. The end grain of the arrow shelf in particular will sometimes develop chatter and tool marks so go slow, you'll need to leave proud to allow for generous sanding.