I asked the Boss, Mark Westrom, about this. He worked at AMCCOM at Rock Island and worked some M-16 stuff once in a while.
He says they believed the M-16 lower receiver was good for about 100,000 rounds, and that they were done in by enlargement of the hammer and trigger pins. They either wore out or enlarged each time the receiver was reanodized. That's why the rifle could only be run through rebuild twice.
So Troy's right. The pin holes are the limiting factor, but new Colt sized pins can be used to give another 100,000 rounds of life.
That's exactly why we made the AR-10 with standard government sized pins instead of larger ones. You get a second life on the receiver. The new AR-180 has a steel liner in the polymer lower to carry the pins and keep them from working on the polymer.