I live in Daytona (and my sister is in Ponce Inlet).
Ponce itself is pretty quiet. Mostly just houses. It has the beach (very nice), the lighthouse (fun once), and few restaurants at the inlet. The HIDDEN TREASURES restaurant is good for small kids because it is mostly outside (though I prefer their other location on the other side of the inlet).
North of you is Daytona Beach. Never heard anybody call it "dirtona", but everybody has an opinion. Definitely has some run down areas but we like the vibe and it isn't over built/commercialized.
In the summer (Memorial Day > Labor Day) there are free Saturday concerts at the bandshell on the beach (at Main Street) with fireworks at the end (9:45pm, maybe a bit late unless you make sure the kids got a late nap).
Minor league baseball team (Daytona Tortugas) is fun, cheap, and good for kids (for the price and the no-name players you don't feel guilty to leave before the end of the game). Sat night games end with fireworks.
Beach street has lots restaurants, stores and weekend events (https://www.riverfrontshopsofdaytona.com/events/). Check out ANGEL & PHELPS chocolate factory. The tours are neat and the store is great.