It is probably not a coincidence that the first time you saw the buck was during shooting hours, and after revisiting several more times the buck is coming at night. They can go nocturnal in an instance.
To give you a perspective; I drive by a field on the way to my house every day. During the spring and summer months, this field has 20-30 deer in it EVERY single evening an hour or so before dark.
This field borders a conservation area.
Come Sept 16th, the day after bow season starts: I NEVER once see any deer in that field again during shooting hours until the next year.
It only takes 1 day of hunting pressure to push these deer nocturnal.