Not using gloves is like not wearing a harness in a treestand. You might get away with it for a long time, some can get away with it for their whole life, but really what's the point of chancing it? One infected animal is all it takes to put you in the hospital. Gloves are cheap, don't take up much space, are light, and could save your life and your health. Really, what's the downside?
I use nitrile rubber gloves always, no matter what kind of animal I'm cleaning. A box of 100 pairs costs around $20.