DallmannTodd,
If your rifle isn't full-auto, and/or you won't be using a suppressor with it, then you have no problems. While it IS true that some M4A1s (the full-auto version used by Spec-Ops) now use a medium-contour barrel, all of the standard M4s (the burst models) and many M4A1s still use the standard M4 barrel, and have no problems with it.
A BM M4 barrel will be capable of at least 1.5 MOA accuracy (many are much better). That is PLENTY of accuracy for the type of uses intended for an M4 (i.e., combat from CQB out to 200m), and the light weight allows for very fast target acquisition and keeps you from running out of steam by not having an overly-heavy carbine. If benchrest shooting is what you were looking for, than the M4 is the wrong configuration.
-Troy