You sound like me. I married at 23 right out of college. I'm 27, and will celebrate my 4th anniversary in July. We have 3 kids already (all three boys!) My wife is 18months younger than me. Both sets of parents are still married and we grew up in pretty traditional households (except for my father-in-law's alcoholism).
Marriage is work, and if you're not willing to work at it, don't get married to begin with. "Divorce" isn't even in our vocabulary, but "love, humility, and sacrifice" are.
I think age doesn't matter as much as an individual's maturity and responsibility. How many folks in the 1940s and 50s got married at age 18 and were married "unto death" some 50 years later? I think maturity and responsibility do not necessarily come with age (I've known some 35y.o guys who were about as mature as 15y.o. boys), but from how a person is raised.