Pop your pins, do they wobble when popped out? If so, again, my bet is your take down pins are junk. It's all fine and dandy to swear by colt products, but your upper and lower aren't colt products themselves. Plus even colt has manufacturing tolerances and variances.
So here's the deal with these AR's they mate up with the upper and lower, but will have some form of gap always. That means for the most part, your tightness is purely on the tension of the take down pin. The green dot pushes against the upper to bind with the takedown pin, that is how it's intended to work and is why its rubber, so u can put pressure on it. If your pin OD isn't too small and matches with your uppers takedown pin holes ID, it works great with taking up the tiny bit of slack and completely eliminates any slack.
However, if your pins OD are smaller than your uppers or lower ID, then excessive slack happens, and it allows the upper to rock side to side since it really is just floating on the take down pins. (smaller OD on pin = upper can move UP and side to side making a gap you wouldn't have with proper OD pins) And this happens from normal wear, or poor mix and matching of parts. Ton's of solutions available pending your situation. Active service days I'd use a disposable ear plug and a piece of plastic wrapper to bind in there to make it rock solid. However you with all new parts, matching manufacturer upper and lower, I'm sorry man, I bet your pins are junk even if they are colt.
But hey, I could be wrong. You could always remove your take down pins and use a quality calibrated bore gauge micrometer and check the ID of your take down pin holes on your upper and lower, then measure your takedown pins OD.
But if your upper wobbles side to side AT ALL, it means your pin has a smaller OD than your uppers ID. And if your upper and lower ID match, then your pins are to blame.