It can certainly be enough time. My first bull was taken on the first day of a hunt, and I have also shot a cow on day four. Plenty of other close encounters early over the years as well.
Sure, it cuts down on your odds of success, but those of us who choose to hunt elk with bows generally don't like to look at those odds anyway.
There is a limit as to how many elk hunts we get in our lifetime. If you can do it now, and it isn't going to eliminate some other hunt, go for it! Four days in the mountains is priceless, and are four days you will never make up for if you don't do it.