When I lived in town, I shot in the backyard a lot. Hung a target in a corner between the dog kennel and the privacy fence. I was shooting down hill (especially from the deck) and toward a ditch -- no houses directly behind me. Any time there was anyone out I'd quit just to be safe.
It was illegal in my town (Lexington) as mentioned earlier, but I never had a problem. Neighbors were cool with it and I was very conscious of what I was doing and didn't make it an issue.
Good Luck with it.