CH444, No this would make a great rifle to M-4.
The one thing that you will want to do is just swap the barrel, fore arm, and butt stock. This will leave you with a A-1 upper w/o a forward assist. You will not have to worry about using a off-set pivot pin, nor the upper/ lower not color matching if you use the SP-1 upper.
The high points to this is the A-2 sight is a waste on the M-4, and most people never use the forward assist anyway.
The parts needed to convert are as follows:
Car Carrier-stripped(use your old bolt and key).
M-4 buttstock with buffer and buffer spring.
M-4 barrel with gas tube and Hand guards.
Note: You will need to file the new front sight(pin), due to the A-2 sight housing being a little higher, but only a few strokes will be needed to zero the rifle.
Hope this helps.