Tried mounting a older Reflex/RX10 combo on the detachable carry handle of an M4 (factory Colt).  

For some reason, the mount doesn't go in all the way in the front of the handle, causing the mount to point up.  Same unit fits fine in A1 and A2 uppers.  I'm afraid to give it a hit with the mallet.  Anyone experience this?

Yep, I have noticed the same thing among my all-Bushmaster family.  For some odd reason, the A-3 groove is 0.006 inch narrower than the A-1 groove.  

I just measured my Bushamsters:
A-1 groove = 0.500"
A-3 groove = 0.494"

However, for my accesories, they will squezze into the narrow A-3 groove with a little additional force.  Whereas, the same accessories will fall into the A-1 groove with no force needed.