The front pivot pin channel has stop end on the channel to prevent the pivot pin from totally coming out of the receiver. If you try to punch the pivot pin past the end of dent stop point (not drifted over to just remove the upper, but totally out of the receiver), you are going to blow the detent receiver channel out of the lower receiver (read will need to replace the lower receiver).
If you need to remove the front take down pin from the receiver, insert a 1/32 punch threw the small hole in the pivot pin (detent pulled out as far as it will go) and press in on detent (threw the pivot pin to push it completely into the receiver), and turn the pivot pin to allow the detent to pop back out against a smooth side of the detent (not back into the pivoting pin’s dent groove). From there, you should be able to pull the pivot pin completely out of the lower receiver.
As stated, the detent and detent spring are under a great deal of tension, and will fly as soon as the pivot pin clears the receiver. As suggested, place the receiver into a clear bag to contain the flying parts.
P.S. Not to scare you, but pray that you only bent the end of the detent, and not oblong/forced over the detent channel itself in the receiver when you went from a soft drift to a steel punch and hammer.