It sounds to me like somebody in that situation needs to be the adult, and it looks like it will have to be you. Act like an adult and ignore him. By all means do not hesitate to defend yourself, but DO NOT go to him. If he comes to you and strikes you, then by all means defend yourself, but do not instigate an altercation.