I had a "specialist" get it out for me- it was a burr inside the hole holding it in.
I have been spoiled by not ever having any issues with the finish of the receiver in fitting parts.
Lesson learned- always check the holes and finger a drill bit to be sure.