Have you been doing it for a few months, years? Those tissues take a while to toughen up. Hands and fingers get sore from swinging a golf club or baseball bat or tennis racquet, etc. I've been shooting for more than 50 years and my fingers will still hurt for the first few arrows if I haven't shot for a while, say a week or two. Depends on bow weight, type of glove, callouses, etc. Same with your face, though a little change in anchor point or pressure on the side of your face might ameliorate that situation. Good luck.