While I really like the RDB, the plastic lower is probably the weakest part of the gun; I suspect the magwell simply isn't made to that great of tolerances, so you see this variation in mag compatibility (for example, mine can't shoot metallic mags without dropping them). We really need to get someone making an aluminum or *quality* plastic lower.
I've never heard of people having feeding issues, though, so I'm inclined to question the mag itself. Now, I did have a single Pmag that would fall out...but examination showed that it's catch was cut .02" higher than the other Pmags, so that was a Magpul quality issue. I'd compare the Hexmags to your other mags that do work, and see if anything looks different.
If Pmags are tight in your magwell, I suspect the Hexmags are as well, so there may be some squeezing at the feed lips (or lack thereof) going on that is causing malfunctions.