Quoted:
Warren sights need to be fitted. Borrow a sight pusher to get it out. if you beat on it too hard you can break the tritium vial. Before you reinstall you need to pass the bottom of the sight over a file or stone. Once you remove enough metal to get it about 1/2 way in with light taps of your punch, use the pusher to move it the rest of the way.
But you tapped it in, so you can tap it out. You need a steel punch. and a bigger hammer The most important thing is to secure the slide in a solid vice. If the slide is moving it is much harder to tap out.
This. you need a solid vise.
since those sights have a set screw to hold them, just file to fit and install them. no need for a sight pusher. a sight pusher is needed for those sights that are friction fit, and with out a set screw