Yeah, depending on how rough it is, just start with a reasonable grit paper and work your way up.
As Roy said, you may want a small sanding block or something.
You can always stick PSA sandpaper to an appropriate sized piece of hardwood to make a simple sanding stick