Why not look at button up dress shirts or light weight flannels? I frequently hunt in plaid dress shirts in the summer time and flannel shirts in the fall/spring. I really only bust out the camo when it starts to get a little on the chilly side or when there is some weather concerns and I'm looking to take advantage of wind-blocking or water resistant technologies.
My wife even makes fun of me occasionally because I won't buy a dress shirt that I can't go hunting in. She says I don't wear many colors. In my mind I have tons of browns, oranges, greens, greys, reds, etc. All look right at home in the woods, office, or church.