I've been in very similar situations. Now I rarely ride anywhere with anyone. If any of the guys need to leave or want to leave, they are free to hop in their truck and go.
I am a big believer in hunting to satisfy myself. A couple of years ago, I realized that the only person that has to be satisfied is me...whether its a buck that many guys wouldn't shoot, a 12 hr drive to hunt Sika deer on the eastern shore, an 8 hr drive to GA to hoghunt or an evening trip at the family place...i am the only one that has to be satisfied with my decision to stay, leave, harvest an animal, etc.