I get CS. So it can happen.
You need a lawyer.
A dad with residential custody of a child (or children), and the child may already not want to visit mom, and mom may or may not pay child support is not as rare as you think. It's just not politically correct to put on government websites, news media, etc because we all want all moms to be perfect and the dads are the deadbeats.
But you DO need a full agreement in writing, written by an attorney and submitted and recognized (ordered) by a court.
Just you wait though...go ahead and mention that to your ex, and the tune will change. Mention child support and it will be like a Metallica concert.
You go get an attorney that specializes in divorce/custody representing the father. You get the BEST lawyer you can afford. It will cost you everything, but it will be worth it. Trust me. Courts are prejudice in favor of women. You need to protect yourself and your family.
I'll say it again. BEST lawyer. First time I got a "good" custody lawyer, and $10,000 later got nowhere. Then I got the best divorce lawyer I could (couldn't) afford and got somewhere. Just yesterday, during Round 2 which the ex started, my super expensive lawyer ate her lawyer for lunch.
Oh, and get ready for her to play the Restraining Order card. Meanwhile you behave PERFECTLY. No fights, no DUI, no nasty emails, no threats, nothing.