Well, when I started in tradarchery, I jumped in with both feet. I walked in and bought a Martin Hunter in 50#. That's what I started with. It was never an issue drawing the bow because I worked in repair and used my hands always.
I found that some of the machines at the local gym can be adapted to exercise your "bow muscles". Basically, just make a mental note of the muscles that feel weak when pulling that 50#er. When you go to the gym, use the machines you feel cause that same sensation. Stick with those machines. Don't neglect the others, though, because you need a full-body exercise. Diet plays a big role here, too. If you're not feeding your body quality fuel while you're trying to build muscles, it just won't happen. Kick the "junk food" habit and eat real food. In other words, if your great-grandparents wouldn't recognize it as food, don't eat it. Plenty of protein (fish or meat), milk, and lots of fresh or frozen vegetables. No crap. No sodas. It's a committment. Believe me, diet is a big part of building those muscles. You don't need to look like a body builder because it isn't necessary. But you do need to build your muscles.
I'm barely 5'7" and weighed about 120lbs when I started in tradarchery. It isn't size as much as it is how often you're using those muscles and how well you're taking care of the overall body itself.