I extend the height of the fence with some plywood, and tack old carpeting on my side of the fence.
Whatever you do, make sure your neighbor understands and approves of your shooting arrows against the fence, because s**t happens, and sooner or later, an arrow will go over the fence. So, as you did, only shoot whenever you're sure nobody is on the other side.
Once when my neighbors were gone, I was shooting and somehow an arrow went through the fence. I looked over and saw it on the other side, and thought I could reach it with a rake. So I went to get my rake, but by the time I got back, the neighbor's dog had come, and when I tried to rake the arrow over to me, the dog grabbed the arrow and raced off with it. As I take care of the dog sometimes when the neighbors are gone, I felt okay about going in their backyard to get the arrow away from the dog, but the dog had hidden the arrow and I couldn't find it. The next morning, after the neighbors had returned, I found the arrow laying across my newspaper in my driveway. You don't need too many of those kind of things to happen before your archery becomes an unpopular activity in your neighborhood. Fortunately for me, the neighbor is also a hunter and meant the gesture of putting the arrow on my newspaper in a friendly way. But with an implied message, I'm sure.