I got a ruler and took measurements on an M4 on the rifle rack in the office. (I work at a Joint command, but I think it was an Army-issued weapon.) The rifle has a Matech BUIS mounted on the rearmost slot. The sight radius is 370mm, and the aperture on the Matech is 2mm in diameter. Assuming people can pick up the center of the front sight reliably, the math would put the maximum spread of the different aiming lines through the rear aperture at about 18.6 MOA. In practice, it would probably be somewhat smaller, due to that natural tendency for your eye to look for the center of the hole. That seems consistent with the observations of deviations in a 6 in radius at 50 yds.
On the other hand, FM 3-22.9 states:
5-40. When standard zeroing procedures are followed, a properly zeroed weapon for one Soldier is close to the zero for another Soldier. When a straight line is drawn from the target's center to the tip of the front sightpost and through the center of the rear aperture, it makes little difference whose eye is looking along this line. There are many subtle factors that result in differences among individual zeros. Instructors/trainers should emphasize the similarity of individual zeros instead of the differences.
5-41. Most firers can fire with the same zeroed weapon if they properly apply marksmanship fundamentals. This information can be useful in three ways:
(1) If a Soldier has difficulty zeroing and the problem cannot be diagnosed, a good firer could zero the weapon to find the problem and eliminate the weapon as part of the problem.
(2) When a Soldier must fire another Soldier’s weapon without opportunity to verify the zero by firing for example, picking up another man’s weapon on the battlefield), the weapon will be closer to actual zero if the sights are left unchanged. This information is useful in deciding initial sight settings and recording zeros.
(3) All weapons in the arms room, even those not assigned, should have been previously zeroed by the last Soldier they were assigned to. Zeroing this newly assigned weapon should start with the sights left where they are.
(I should point out that when I have gone through combat training, they always had us start from mechanical zero, instead of where the sights were last set.)
I would recommend doing some testing with realistic scenarios (that reflect likely situations where the rifles would be employed) to check how much of a loss of accuracy is experienced when a bunch of different people use the same weapons.