I drill them with a smaller bit about 1/4 inch through the rivet then use a pin punch to drive them out.  Use Kroil to for good measure.  They pop right out with this method.  Be sure to use a flat pin punch, I think the center punches push out the metal.

Kroil is your friend..  Don't leave home without it..

your local gunshop should carry it, if not..  I don't know if Wallyworld does??

I misread your original post, I thought you were having trouble with the pins not the rivets.

For the rivets, I prefer to sand them flush so I can get a good read on the center. Centerpunch them and then start small and slow and work your way up to just shy of the true rivet diameter.  I go about 2/3 the way and change to a larger bit.  When you are near the rivet diameter, insert a steel punch and give it a few hard taps.  It should come loose.  Make sure you are not resting the rear trunnion on the rivet you are trying to punch out or well, you know, it won't move....

Sorry for the confusion