Shawn is right. He is aware of the doe groups in your area, he will be back. Maybe more frequently than you think. Be super careful of your scent, and remember your scent can linger long after you have left the area. If he detects you it may keep his movements to non-hunting hours, or cause him to stay away.
Also, keep in mind, he is a different deer than the rest. He may not bed in traditional bedding areas. He may actually be much closer than you think, or much farther away than you think.
The ultra sensitivity that big bucks have to human encounters is what keeps me out of certain areas that I hunt until the rut has the bucks about to bust.
Good luck, I hope you at least get to see him in the sunshine!