Here is what I would do. Emphasis on I.
Buy your land that you think you want. Just sit on it, paying it off ASAP. Save/invest your money.
Once you are about 2-4 years from retirement, have your house built. In 25 years a lot will change. We will have more efficient this and that, better building methods for such and such. Plus, your wants may change. Maybe you need handicap access because of that "old" hip injury that you get in 5 years. Stuff like that. Maybe you have a different wife/girlfriend who wants a different lifestyle. It could happen, never know.
When you are building from scratch a retirement home, keep in mind EMT services and handicap access in general. You want PLENTY of room for EMT to get a stretcher in the house, maneuver around the house, etc. Think handicap access, single level living setup (although not necessarily used immediately). Guarantee when you get older, having a bedroom on main floor with bathroom right next to the bedroom will be niiiiice.
I guess to conclude with what I said originally: that far out, I would hold off on building anything substantial. Want to enjoy your property now? Buy an RV, go vacation on your land. You can even start working on gardens, using soil building crops. Now would be a good time to plant any trees that you may want that grow in that area. They are a long term investment.