FYI, the spring gets loaded first into the channel first, then the buffer/detent installed on top of the spring.

As for getting it out (with the buffer loaded into the channel first) compressed air does the trick every time. Just remember to stick your hand over the channel to catch it, or the detent will never be found again.

Also, not a bad idea to clean up the bottom of the channel with a drill bit.  Often, burs and even traces of blasting media have been know to be on the bottom of the channel, and no cleaned out when the parts are assembled.