Don't go to TJ with the family, only by yourself or with a couple of buddies. I always take the trolley down to the border and walk across, don't even think about driving.
Gaslight District. Several good brewpubs, mostly nighttime stuff but some family-type restaurants and so forth too.
Mission Beach, there is a great pizza place near the roller coaster that sells primo giant slices. Also lots of scenery around that area.
Balboa Park, all the museums and whatnot the other guy mentioned.
Anthony's Fish Grotto on the waterfront is great if you are a seafood fan. The ships down there, "Berkeley" and "Star of India," are worth a look.
S.D. is a great place, it's like all the good stuff in L.A. without all the accompanying B.S. If it wasn't in Kalifornia it would be even better.