I have used 36, and it's about as coarse as the grooves my toothing plane leaves behind, and I think that they are a near perfect combo.(toothing plane grooves and Smooth On). Even in highly stressed, working, fade-to-nothing glue joints in the dips of say a tri-lam, they don't fail. For use under clear glass however, grooves left by 36 grit can sometimes be faintly visible if you look really close, depending on the lighting and color of the wood, so I try to use 40-50-60 grit or something like that. I dont see a reason to use anything finer than that. JMO.
Your question is really quite loaded as there are many other factors and applications.