I'm far from an expert, but have shot a few critters over the last few decades. My experiences are with whitetails, hogs, black bear, and turkey. My limbs have always been some sort of natural wood, snake skins, or a dark brown or black.
I'd say that as long as you don't have something that has a shiny finish, which might cause a glare, or a color that a deer can see (blues and violets), you're far more likely to spook an animal with odors, movement, or noise, in that order. I don't ever recall spooking an animal solely with movement from a bow.
Most of the time, if a whitetail sees a slight movement or something out of place, they'll usually start bobbing their head around to get a better look. They'll stomp their feet to try and get whatever it is they saw to move, and if they're not satisfied, they'll maneuver themselves downwind.
That being said, I would say that hunting with a light colored bow would increase your odds of being noticed. It might not outright scare a deer, but it certainly might cause one to focus on you, which is something you definitely don't want. It's about impossible to draw on a deer that's looking at you without spooking it.