Ron, you may be able to disassemble and add a washer so that it'll tighten up well. If you decide to go that route I can mail you some small pieces of vulcanized fiber that is used for washer and liner material. You can cut it with scissors.
Then you need to file it smooth, sand with progressively finer sandpapers until you get to about 800 grit, then you can buff if you want to.
I suspect the antler has shrunk over the years but that is a guess.
This may be completely the wront way to go about it so if anyone else has done this before (and I have not), please chime in.
Dan