I can answer this from experience. I used a brand new Craftsman drill press to do some very light milling. I even had an X-Y table and had the workpiece firmly clamped down. By "light milling" I mean 3/16" carbide bit and a depth of cut of only about 0.025"
All I did was cut a depression 0.025" deep and about 4" x 5" in a piece of 6061 aluminum. By the time the project was done, I had destroyed all the bearings in the spindle & sleeve. Replacing them was inexpensive, but a huge pain in the butt and the machine never ran quite the same again. It's still suitable for normal drill press operation, but it's noisy and tends to vibrate some.
I have a mill now. With the mill, I could have done the same project with far greater accuracy in about 1/10th the time with no damage.