There's really not much difference in grinding hard or soft steel at the 80 grit level.
You should hardly be able to tell the difference.
Scale was probably your problem.
We really need to be more careful with this term "annealing", especially when we use it in reference to steel with more than .84% carbon, as well as less than .84% carbon.
As well, slow cooling from above critical and below critical.
That right there is four entirely different animals.
And what are we "annealing"?
Hardened martensite or pearlite?
There's a whole heck of a lot more to it than getting a piece of steel hot and cooling it off.
If not done from the right steel phase at the right rate and temp, grain growth can just destroy all of your time and effort.
There are no short-cuts in this stuff - if you want to do it right.