I have done a few and never had a problem. The other day, I decided to take a railed upper and install a front cap so I could put some standard hand guards on the rifle. It was the most difficult one to get started, and after torquing the barrel nut, I was anxious to get the FSB back on, and forgot to install the cap. As I was attempting to remove the pins again, I was having an even harder time than at first, and the pins were not even as deeply seated as they originally were.
Long story short, I was mushrooming the punches I was using. I took them out to the grinder and put a sharp face back on them, and they worked like magic.