I go to finer grits, from 400, to 800. A good buffer will then make it easy, I have a Baldor 1HP with 10" wheels, and use green Matchless compound.
If I get to polishing and find some scratches that will remain, then I go back to the grinder and get them out.
Dan