Correct me if I'm wrong, but one of the reasons a 37mm launcher is not classified as a DD is because it's a smoothbore, whereas the M203 has a rifled barrel.

However, if you load your own non-lethal 37mm ammo, and use a hull that is rifled, would that make the launcher (or the ordnance) a DD?

I'm about half awake, but...

You have a rifled hull, but a smooth bore. Without some sort of sabot setup (and, maybe not even then) how would you impart rotational motion?

It might be legal, but what would it do?

Here's what I imagine:

Say I'm looking to load my own smoke round or other non-lethal ordnance. Correct me if I'm wrong, but the hull still remains in the launcher tube after firing.

If I use a particularly long rifled hull (say, 5" hull), and use a smoke projectile that's only 2" long, The hull should impart rotational motion onto the projectile will it not?

And would this end up making the launcher (or the ordnance) a DD?

The verbage states smooth bore. Period. The other verbage states riffled barrel larger than. Period.
Don't read more into it than there is. Bore, barrel with no lands and grooves to impart spin is a smooth bore. Bore, barrel with lands and groves to impart spin is riffled. Ammo is ammo. If they wanted a restriction on type of projectile. It would be listed as such.
Bottom line... The cartridge is not part of the launcher. Separate definitions with their own rules depending on classifications.
Now...if you are planning on reloading cartridges in a manner counter to the intent of the rule... That have never been commercially avaible and normaly used in a launcher of that type.before. I would expect the ATF to consider it constructive possession.
So... If riffled cartridges were normally used in a launcher of that type then reloading them should be no problem, right..? NO...not nessassarly. you  may be reloading a cartridge originally designed for  a DD and never loaded as a less lethal cartridge for the purpose of imparting spin to increase range and accuracy. The ATF is going to use intent "as they see it" to come after you.
If... Less lethal cartridges were availble on the open market with extended hulls having riffiling then there should be no problem.
I would not reload any cartridge, hull, etc. in that size.  in a manner different than what is available commercially within the correct classification. If you stray from the norm to far I would expect trouble.
If you are planning on poking the bear you may well need an attorney to explain why you should not. Yes.. Get attorney advise.

 There's an Airsoft round like that the CASE is Rifled, the projectiles have molded-in rotating bands, when you load the projo into the launcher, you have to line up the band with the rifling, when the round is fired the gas pushes the round out of the case, imparting spin as it goes.   Linky