You need to make up a really cool excuse if you are going to decide to leave. Something like:
Dad, I just got a call from my boss. I have an important job that I have to complete by 22:00 two days from now. I can't say much more because of OPSEC, but lest just say it includes a 9 iron, stuffed horse-head, and 300lbs of lime. There is a helicopter waiting for me about 30 miles down the road. They will take us to HQ, where my wife and child will stay until the job complete. They will then take me to the insertion point. I will be on my own from there and won't be able to be contacted by cell phone.
If a man in a black truck with out of state plates comes to this location and askes any questions, tell him that the cheese is old and moldy. Hopefully that will keep him from asking any more questions and he'll leave you and mom alone. If he asks further questions, tell him that the situation has been compromised and that you kicked me out of the house when I told you about my profession.
I'm sorry that we can't spend more time with you, but business is business and I have to take care of my family.