In a mill, you wouldn't notice a difference. I doubt you would in a drill press either.
FWIW, if you shop around, you should be able to find 7075 for just as cheap if not cheaper than 6061 as the 7075 is more common, quantity>price. There are several recent posts about 80%'ers. Check them out, lots of good info and links. Ares has theirs on sale now for $60. As others have mentioned in similar posts, they do require a little clean up work as the finish milling is not perfect, but I've done a few and would certainly buy more. I also like Coats Carbine. I prefer billet to forged for asthetics only. I have a couple of Anderson(forged) and TR Enabling(billet) as well. All of the 80's I've bought were under $75, the anderson forged were under $50 at the time. Again, all 7075. While the difference between 6061 and 7075 may be minimal to neglegible, if you can get the 7075 for the same price, why not.