Only you can, and should, decide what you want to hunt with and why. Actually, it sounds more like your woodsmanship skills, rather than archery skills, are your major drawback. (One of the most consistent bow hunters I know kills most deer at 10 to 12 yards.) These skills can be improved. Read a lot of material, and you will notice similarities in what the various experts have to say.
Essentially, locate an area with a good deer population. Learn where the deer already want to go. Set up in that area with close attention to the wind - always the wind. BE STILL, BE QUIET! Let the deer come to you. Pick a spot. See, none of this is anything you have not heard before. You just need to get better at it. You will never be perfect, and you will always screw up some chances as we all do, no matter what weapon you hunt with.
Personally, I don't think changing to a compound bow will cause your problems to evaporate. Nonetheless, use what suits you. Spend time in the off season studying your hunting area and get the deer patterned. Then be patient.
Whatever your decision, I wish you luck. It will happen. I too had no hunting mentor, and it took me a long time to begin taking deer, and at that time I used a rifle. The average hunter will take fewer deer with a trad bow, because he accepts certain limitations as part of the process. For years, I hunted rifle in gun season and bow during archery season, and finally migrated to bow only. Maybe that will work for you, but whatever happens, do what suits you. As long as you hunt legally and ethically you are a sportsman.