I've got a lower with Colt 9mm mag blocks installed.  It works, but the mags (also colt) are really tight, and are a bear to get out, usually requiring both hands.

It's definitely tight front to back.  I've tried filing a bit, but the front has a lip at the top to keep the mag from overinserting, which limits the travel of my files.

The rear has the ejector at the top, also limiting things.

I don't want to make it sloppy, but anyone got any ideas to make things a little easier?

It would seem to be much easier to make a few strokes with the file along the front of the mag. Put a pencil mark on the mag at the bottom of the magwell so you don't leave file marks below the part obscured by the magwell

Are ALL your mags doing the same?  Perhaps the mag is the problem?

No, all my mags (except 1) are the same way.  And I've bought a bunch of new production mags of different makes, hoping to find one that works better, and they're all at least as bad if not worse than the Colts.