I took the barrel nut off my nitrided mid-lenth using a DPMS multi-tool wrench. I used the crenelated wrench for standard barrel nuts and it came off with just a little straining.
There were no hints of any kind of thread locker. In fact it looked like it was assembled using a black grease that is very similar to the aeroshell grease I use on my own builds (the grease most of the snobs recommend).
It's possible you got one that was torqued on a little too tight. That happens sometimes. I've heard stories of other factory rifles with barrel nuts torqued waaaay over 100 ft lbs. I can't imagine why the teeth on your wrench would break though. Are you sure nothing else is getting in the way. This might be a dumb question (I don't know you so don't get offended if it is), but did you remove the gas tube?