A starter kit is a great place to start. I have recommended those to just about everyone that shows some interest, but doesn't want to spend a fortune. Biggest single expense will be finding a brew pot. A big 3 gallon pot is $$$. Aluminum will work, stainless looks cool. Check Craig's List for pots. A used (or new) turkey fryer burner and pot works great. If you get a used one, clean the crap out of it. Oil will be your enemy when making beer.
Some more advice (as if you asked!)
A lot of new brewers tend to freak out over a everything. I see questions all the time like "I missed my mash temp by 1 degree! Is my beer ruined?" and "My yeast is blowing off! Is my beer ruined?" and "I put in 1 ounce too much canned pumpkin! Is my beer ruined?" etc, etc, etc...
Just relax. Take your time, don’t be in a hurry. It isn't going to hurt anything if you boil for 5 extra minutes or you pitch your yeast 30 mins after you wanted. Due diligence is important for sanitation, but you don't have to be ultra paranoid about it.
For your first batch or two, use store bought spring water. Good, clean and won't mess you up. Water is the largest component of beer, right? Starting with good water gets you ahead of the game.
There are beer brewing web sites out there that will help give you some ideas of how to start out and they do have good advice. I would warn you though, don't fall into the beermeister hero worship crap. That kind of stuff is rampant on one of the more popular beer brewing sites. Sometimes its hard for people new to the hobby to filter out the BS.