I just replaced the nut and saddles on both my electric and acoustic guitar. I used TUSQ for both. For the nuts (TUSQ XL), I measured the string spacing on my original, and found that I could use one of their pre-notched versions for both the electric and acoustic.
Removing material from the base of a nut or saddle is tricky, as you have discovered. It can be difficult to keep it square across the bottom. When replacing the saddle for my acoustic, I didn't remove any material from the base of the saddle. It came perfectly square, and I didn't want to risk screwing that up. So, I used the original saddle as a guide (I measured the radius of the original saddle to make sure it correctly matched the neck radius) to draw a proper radius on the new TUSQ saddle. I then filed/sanded it down from the top, to the proper height (actually, a little lower than the original). I finished it off by sanding it up to 12000 grit, and then hitting it with some polishing compound.
For the nut, I did carefully remove some material from the base of the TUSQ nut (because it was so tall), but I did it by putting self sticking sand paper on the steel base of my jointer, and using the guide fence to insure it remained square. Then, I finished adjusting the string height by using the proper nut files to fine tune the depth of the notches. Attacking the nut from the top allows you to individually adjust the height of each string, so you can get the radius correct.
I honestly believe the tone and sustain of the acoustic guitar improved somewhat by using the TUSQ material, and, I swear it stays in tune better.
StewMac carries a variety of nuts and saddles, including pre-notched bone.
Good luck, and have fun with it!