Three years ago I was almost exactly where you are now, OP. We looked at land all over the North Georgia area and finally ended up buying 110 acres between Chatsworth and Ft. Mountain on highway 52 in Murray county. It is where I'm going to go to retire and die.
Two years ago we built a ~2000 sq. ft. house and a 30'x40' detached garage/workshop on the property and are slowly making other improvements. I still have about 4 - 5 years left before I can retire and flee the craphole that Douglasville is fast becoming, but in the interim we spend a lot of weekends and vacations up there. We've really grown to love the area and the people.
During the looking/buying/building process I developed a great relationship with a couple of Realtor-type guys who kind of specialize in undeveloped land in the Murray, Gilmer, Fannin area as well as a fantastic Contractor who was very easy to work with and did a fantastic job of building what I basically drew up on the back of a napkin. One thing to consider, depending on your plans and what you're looking to do, is that the zoning/land use/building permit processes in Murray county are EXTREMELY laid back compared to Gilmer and Fannin Counties.
Feel free to PM me if you'd like the names and contact info of the people who help us or to discuss what we learned during the process.