OP, my family started running beach stands in OC back in 1972. So you could say I've got a lot of history there. Two of my sons graduated from Worcester Prep, the third was pulled out when we moved to Florida.
The town has gone downhill in my opinion especially over the last decade. What was once a nice family town, has been over run by the thug element from DC, PG County, Baltimore City, etc. Stay off the boardwalk late night, it's a real shit show nowadays. No way in hell I'd allow kids to run around like we use to back in the day. Lots of fights, assaults, etc. especially downtown as stated above. It got to the point were we had to get out, property values have suffered as a result. I loved growing up in the town, but don't miss it one bit. We found a little slice of small beach town in FL, and have no plans to go back.
Frankly, you'd be better off going north to Delaware beaches.