I hear ya, buddy. I curtailed a lot of hunting when our kids were in the preschool stages. And some of my hunting friends less than understanding about my choices.
However, now the kids are 14, 11 and 7. Right at the beginning of hunting ages. So, now I just look at my wife, shrug my shoulders, and say "Gee, Honey, I HAVE to go hunting... it's for the kids". And all 3 of them enjoy coming out with me. I am not saying it is the same kind of fun as an elk camp, but when we are all out chasing rabbits with the bows, it doesn't get much better.
So, hang in there. As husbands and fathers we are called to happliy sacrifice for our families. It is not always easy, but if you cast your bread on the waters, it will return 10 fold.