That's my neck of the woods...I live in Banks county and I'm in local politics, so I get some juicy inside bits on the towns around us.
If you enjoy fishing and mountains, without a doubt, Rabun is the place to go. Clayton is a quaint little town - sort of like Asheville without the hippies. Real estate is absolutely stupid up here. My wife is a realtor that specializes in out of town/ state buyers - finding a house or land in Habersham, Stephens or Rabun is just about impossible at the moment. Hell I think we only have 50 to 80 homes listed for sale in the whole county at the moment.
As far as hunting and fishing is concerned - you have the whole Chattahoochee National Forest up here. There's a WMA rifle range right outside Clayton and one in Alto.
Clayton has just about everything you could want...Walmart, Home Depot, nice restaurants, etc. The rest of Rabun county is wild. Head too far down Warwoman road and you'll see teenage kids with no shoes, digging in ditches with old coffee cups. And for the love of God, never bring up Deliverance...liable to get yourself shot. People up there are touchy about the way they were portrayed
. The kid that played the retard in that movie still works at the home depot, I think.
Habersham is the other hot spot, if you enjoy a bit more cosmopolitan living. Clarkesville is the town to be in. It's a well run town with a bit of an asheville vibe as well. The Suque river flows past town and has some excellent trout fishing from what I've heard. Avoid Demorest. It's a student town that's being run by Boss Hogg types. Cornelia is where every illegal in the area wants to live.
Banks county is a dump. Period. Not a single redeeming factor. Standoffish people. No nice rivers. Flat for the most part. Very agri-type.
Stephens county is nice. Cleveland is a great little town. More or less same as Clayton and Clarkesville. Land is a bit cheaper in that area. Fishing isn't great, but it's not that much of a drive to Habersham or Rabun for recreation.
Let me know if you have any questions. Shameless plug...if you need a realtor that knows these hills and hollers, let me know. I'm sure she can get you a good deal.