Milspec* calls for the small hole lowers. When Colt released the semiauto AR15 to the public in '64 they changed the specification to a large pivot pin to make it more difficult to use M16 parts.
Only in the past decade or so has Colt switched to small pivot pins in their commercial models.
The vast majority of M4 uppers will be small hole. If there are any large hole M4 uppers they would represent a tiny percentage.
* - Milspec means much more than what size pins a receiver uses. It also includes having all the proper machining to accept all M16 parts. True milspec receivers are thus class 3 machinegun parts.
edited because I dunt allwaz spill so gud.