Go for it! There are lots of ways to proceed. Twenty years ago a friend of mine bought 80 acres, then bought a camping trailer and put it out there.
I saw the wisdom in that and found 120 acres for sale (very reasonably) in 1995. I have a huntable population of deer, grouse, the occasional timber wolf and a bear every once in awhile. I rent out the field for hay and that helps a bit on the taxes.
Eventually, my friend upgraded to a small cabin and I bought his old trailer for $300. I have used the trailer for about 10 years now.
Not pretty, and it will never compare favorably to Shrewhaven or Little Big Man's abode. When it gets below zero you can burn through the wood pile, pretty quickly.
Not too nice, but even then someone had to break in and steal a twenty dollar microwave oven. (sad what some people will stoop to) My wife has no desire to use it as a weekend home, so I don't have to spend a bunch of money keeping it pretty. But it does the job, I have nothing in it and have never regretted it.
The only "negative" is my valuation has steadily increased and my property tax has tripled. One thing to consider: since my shack is a camping trailer, I don't pay more property task vs a permanent structure. You have to decide if it is worth it. It might be, to you.
The hunting shack:
The land:
Most days, we have a pretty good time and build a few memories as a by product. Not only hunting, but putting up wood, clearing trails; it is just fun spending time out there.