I like the scruffy look myself....that's pretty much the way me and the guys I hunt with do it, bud.
The scruffy look always reminds me of the beginning chapters of Jay Massey's book, 1000 Campfires where he talks about old wool pants, a wool shirt, suspenders, and watch caps...you don't need to look like a fashion plate...