You are correct, the main difference is the barrel, the barrel thickness in front of the sight of the M4 is thicker and have a mounting notch for the M203 grenade launcher.
The real M4 have 14.5" long barrel, civilian versions are 16" or 14.5" with permanently attached long flash suppressor like a Phantom or Vortex.
M4s have taller 'F' marked front sight base.
The M4 usually have a side sling mount at the front sight mounted on the barrel.
The handguards of the M4 are oval section and fatter with double heat shields.
M4 have enhanced feedramps which are a different angle and longer which extends to the M4 upper receiver.
Colt which mainly supplies the US Military with the M4 have a 4 position M4 stock, other brands have the 6/ 5 position.
Colt Defense M4 website linkPic link of my LMT 16" M4gery