Non inpact will shatter and you'll put your eye out!
Seriously, though, other than the differences in types available and the possibility of a thicker wall not fitting into tight places, there is no practical difference. I do what you are thinking of doing: I have a set of impact sockets, which serves as a socket set with a relatively inexpensive ratchet drive handle. If, as I, you do not use the tools professionally, but just on a home repair level, I see no reason not to save an extra set like this. I try to minimize the overall accumulation of stuff, and try to condense like this. I would also add that, in the situation of a tight fit, or specialized size or type, buy just that one exact socket, which is the best course anyway. I know my computer uses a specific two sizes of torx, so I have only those two instead of a set.