Mostly building the tools to build the receiver. If you get a pre-made one all you should have to worry about is fitting the trunnions and rivetting them in. Most of the noise then would just be pounding; mine is due to the fact that I have to jerry-rig all my tools. I might be forgetting something, but a rivet jig for the trigger guard, etc would also be handy. I'm working on making a cheaper version of the rivet jig now out of wood and sheet metal and my dremel, we'll see if it works. I'd go to one of the build information sites on google and look at what goes into rivetting everything together just in case I forgot something; but I think all you would need is to make a die to form your rivet heads, modified bolt-cutters, and a rivet jig. You would also need to drill the holes for your trunnions most likely, and that can be a pain. I found a drill press to use. But if you center punch well enough and then get a hand drill, it might work. You'd probably want to buy a vice though, I got a heavy duty one from harbor freight for $30 or something like that.
But... If your only motiation is to have a rifle and save a little money, you probably won't come out that far on top; if at all. I thought this would have only taken me about a month to build all the tools and everything, I'm already at 2 or three months; I'm like you, I only have a little bit of time every now and then to work on it.... 90% of that time is still being spent trying to make the tools I need. Still, its a lot more involved than i thought it was. You could do it, but count on it taking longer than you'd expect.
So far I've gotten the receiver mostly done with exception to heat treat, center support, and rails (basically I just cut all the holes and shapes and upper rails, and moved on to tooling); need a torch and a spot welder for that. Right now I'm working on that rivet jig, after that I have to make the die to dimple the mag supports. After that, i think its just a matter of pressing/pounding in the rivets.
Don't forget about the finishing too. I am going to try and jerry-rig a blueing tank to use in my back yard. Your receiver will be ready made, but the rivets won't be finished. You could just paint them or get some cold blue, but i'm not sure how it would look.
I'm also still counting on the cool-the-barrel-heat-the-trunnion method working to fit in my barrel... I hope that works, otherwise it's gonna be more delays.