Glad you asked. I hate to see people seriously injured out of ignorance. See canuk's response - dead on.
You may get by with light cuts for a while, but sooner or later it will get you.
Edit for more info:
Drill chicks mount on a taper - they are a friction fit. the harder you press DOWN when drilling, the tighter they grip. Placing a high frequency side to side motion on them will loosen them.
Milling machines use colletts that are bolted on. They also have a taper that closes the collet around the end mill. The tighter you make the bolt, the tighter the tool is clamped. It ain't going anywhere with side loading.
High frequency side loading (called chatter) is a function of the rigidity of the machine. The heavier and more rigid a milling machine is, the bigger the cut you can take without chatter. Modified drill presses are not even in the same rigidity league with real mills of equal size.
In a mill, if you are getting chatter, it will affect accuracy, finish,tool life, etc. In a drill press it will affect the ability of the tool to stay attached.
Can you get away with using a drill press? Well, if you take extremely light cuts in soft stuff, you may think all is well. Part of the problem is that you just never know when the chuck is getting loose. The other part is that , by definition, you can not forsee an accidental "crash".
That's when you feed a little too far and hit a much taller surface. This increases the load tremendiously and causes the tool to take too big of a bite.
You can also have problems by getting greedy - taking too big of a cut or hitting a hard spot in the metal.
This is where TSHTF. With a mill using a collett, you damage the part and ruin the endmill. With a drill chuck, you may launch something the size of a 30mm round with a broadhead attached at about 500 fps. This is where luck has a special meaning.
Be safe.