The term you looking for is run out.
As stated, pull the check and see how much run out you have to the actual spindle via a dial indicator. Also put some side load by hand on spindle extended all the way down, and see how much you can increase the run out as well.
As stated, may be just the bearing as worn, or could be if an older unit with bronze bushing instead of ball bearings, that you destroyed taking a sled hammer to it.
Next on the chuck, and while most can be rebuild if really needed, its not had to true up the jaws in the chuck if they are not too bad.
Really, need to know the model of the drill press, and this will go a long way to resolving spindle run out back down to usable ranges.
Note, most drill presses between the spindle and the chuck will have anywhere from .002" to .010" run out. Its just the nature of the beast, since they don't use taper bearing, but just standard barrel barrels for only a Z axis load only.