I've done it before during firearms season. It's tough to do on public land in Michigan with all the gun hunters. I love the looks on other hunters faces when they see us toting traditional bows!!
Ignore the orange, the deer don't care.
One thing I do suggest is washing your orange in a detergent that removes UV brighteners, such as the brand made by Atsko. The hunter orange garments in stores are loaded with UV brightners and to a deer it looks like a bright glowing blue blob.
My daughter and I went out this past Sunday and had a turkey walk past us. I think that is the ultimate test!!