When you build your arrows, once the fletch are dry, take a dremel tool with a sanding drum. Mine is variable speed and I use it on a slower speed.
Sand down the leading edges of the fletch, at least the one facing downward. Once you smooth the transition from fletch to arrow shaft, go ahead and put a drop of glue on each to further blend it and hold it better.
That should solve some or all of the issue. Can't tell you how many pieces of feather I have pulled out of my hands over the years. There is actually scaring from the cuts. None since taking the time to do this.