I see no reason to have to depress the ejector while removing the roll pin, I have never needed to. When the roll pin pops out, the punch will be holding the ejector back. So you don't have to worry about the ejector zinging out upon removal. Just be sure to catch it when you remove the punch. However, the ejector WILL need to be held in when you go to reinstall the roll pin.