For the amount of equipment and supplies needed to build at all, plus secure storage, the barn/shop is required. Unless the plan is to drive a large truck in with the materials and tools stored in it, back and forth every time.
Building sites are notoriously picked over when you are not there with thieves taking anything they can access. The wiring phase is their favorite.
With that in mind I would go for concrete tilt up panels. They don't have to be thick, as they go up they support the next, and once shelled in with a roof they are a lot more resistant to bad weather and attack from vandals/thieves when you are absent. It's masonry held to a minimum with some supporting frame work and the interior put up with steel studs like commercial. That will be less depreciable over time and hold up better to use, while allowing the option of making it as high end as you want.
Stick built can be blown away by 100mph winds, SC is known to get hurricanes and hurricanes make tornadoes moving inland. Build it to withstand that and it will also lower your insurance costs, too.
In India homes are done on sight with the ground itself being the form. No doubt American ingenuity can complicate that to the $100 a square foot level. I mention it because when younger my wife and I looked into earth bermed but found no one was knowledgeable about it. We later bought an A frame and it's working out ok - except it's in a sloped hill that drops 8 feet from the front to the back, meaning the basement is half buried. And we have water and insulation problems galore because of it. I do not recommend any concrete wall being exposed to the winter air then running into your living space - it's a major heat bleed sucking your money out of the home as it tries to warm up a 25 degree ice storm outside.
Concrete tiltup is inherently thermal breaking, it's only on the outside. You insulate on the inside and by pouring thin you reduce the thermal mass and it's carryover affect. The most durable houses on the face of earth are stone - minimize the disadvantages using thin panels and you still get the benefits.
Please choose the color carefully - if you ever succumb to the urge to paint it call me and I will bring my baseball bat.