For cheap thrills, the original OM-D E-M1 would be great for your intended usage. That thing was a wailer for me before I moved over to Fuji for my small-ish sensor stable. OM-D EM-5 would also be a contender, but not as handy with a long lens in front of it.
I still like KEH, Adorama, or US/Japan/based > 99% feedback users on EBay for used stuff.
That old Nikon 1 system was supposed to be good for small size telephoto wildlife stuff as well.
I have 4 kids as well, and the one camera that was great when they were younger (instead of just a cellphone) was the Nikon Coolpix A. Fixed 28mm lens equivalent, super small, pop-up fill flash and a high quality DX sensor optimized for that fixed lens. That thing was awesome. I still think about picking up another one. The battery door eventually broke on mine. Had similar success with the Fuji X-100T, but nothing beat that Coolpix A for streamlined Dad shooting.
Now that the kids are older I usually go light with a Leica M10 or
Fuji X-Pro 3, both with 35 primes.