However, even though the 12 ga round has double the pellets, it spreads too quickly in the large 40 mm barrel and is therefore less effective than the M576 at 20-30 yards. Is this correct?
Yes. The shot from the M576 carrier is not released until it exits the 40mm barrel and the carrier slows enough from air resistance to rfelease them. The shot pattern will hold tight until that happens. When fired from an adapter with a length of 5", a 12ga OO buck will start spreading when it is released from the adapter, after 5" of travel.
Therefore, if ones plans to use buckshot wih his M16, would one be better with an undermounted SBS than an M203??
If you want to add the ability to launch shotgun payloads to the M16 platform, a "Masterkey" mount serves this purpose better than an M203. (You can carry FAR more 12ga rounds than you can carry M576 rounds, plus the cost is much cheaper!) To get the effectiveness of the shotgun, you need a longer barrel than an adapter. There are actually two ways you can do this. First is the Masterkey mount for an 870 type shotgun. Second is this:
Nick Ferris Adapter
This adapter transforms your 40mm barrel to a 20ga shotgun barrel (rifled for slugs). You can then shoot 20ga standard shotshells, including buckshot, slugs, and the exploding practice pictured on the site.
As far as weaponry goes, a shotgun is meant to fire shotgun shells, the M203 is meant to fire 40mm grenades. When you take ammo from one weapon system, and use an adapter to fire it in another, there is a trade off. Especially with shotgun shells in a 40mm with adapter. Imagine the difference between your standard length 12ga 870, and a sawed off 5" barrel single shot 12ga. For short range "oh crap" shots, the adapter will work fine. So will the 12ga. For the medium range shots... the adapter will not work so well. The shotgun will.