The issue is not the bare uppers and/or lowers. It's what goes inside those pieces. I've got lowers by Colt, Rock River, Bushie, and DPMS, and uppers by Colt, Rock River and one "no name", and don't feel handicapped by any of them. When it comes to the internals, tho, it's Colt (almost) all the way. Four Colt bolt carriers,one DPMS, and all Colt bolts. Triggers and hammers; all Colt (M16ers w/the tang ground off). Three Colt barrels, one Bushie, and one Rock River; and I wish the two non Colt barrels WERE Colt.
The aftermarket pieces are simply not as well finished or inspected like Colt stuff. Colt bolts and carriers are micropolished, so it'll be a cold day in hell when a Colt bolt cracks at the cam pin hole or scrapes off enough brass shavings from a case head to tie the piece up.
For me, that's one of the advantages of building my own ARs. I haunt the gun shows for GI surplus Colt parts, 'cause I'd rather have a used Colt piece in the rifle than a brand new anything else.[:D]