Make contact with him under the guise of the whole thing being a misunderstanding.
Record the call.
Do not be the one to escalate anything. Just say " Hey, I wanted to get with you about this thing we had . Look, I know things are stressful sometimes, but there are kids involved and we need to try and keep all of this under the surface around the kids. " or some BS like that. Just get the ball rolling. Either he'll apologize and you know the threat isn't real, or, he'll blow up. If he blows up, then you take the tape to his PO.
Done