I think the bowfit is a good idea. I have one, but now the bow I shoot is heavier than the bowfit, so it doesn't do that much good.
Even if you don't have time to shoot, get the bow out and do some draw and hold exercises. When I do this, I try to draw a bit past normal, and hold for a few seconds. I use a shaft with a clothespin on it set at about an inch past my normal draw. Draw until the pin touches my left hand. Hold, count, let down slowly and with control. A little bit is better than nothing. Pull ups and push ups are good exercises also.
An important part is setting goals. You have an end goal set - 50# by elk season. Set some milestone goals along the way, to keep you moving and prevent procrastination. A few years ago I used this to get myself to save money toward hunting. I used to have a whiteboard calendar and drew a deer on one end of it. I drew a little stick figure hunter on the other end - far away from the deer. In the space between I wrote my milestone goals at regular intervals: $100, $300, etc. Each time I achieved one of those milestones, I erased the stick figure hunter and redrew her at that spot. It's a funny, very visual way of "seeing" yourself getting closer to your goal, like stalking a deer. Sometimes you look at the calendar, and you think "wow, I'm still a long way from that deer!" and for me anyway, that was motivating.
Hope this helps, good luck.