The barrel extension pin fits into a slot in the upper receiver barrel socket (read there is some slop, and chances are, the pin was pinned over to the one side when the barrel was tighten). Even if the slot was beyound tight on the pin, the pin will slighty indent into the side (read if you where to pull the barrel to correct, you would still need to correct the problem with a file, and use shims to hold the barrel in the correct index as you retightened the barrel nut).
The only tension that the barrel nuts puts on the barrel/extension to the upper receiver is surfaces tension. Trust me, it going to take a few healthy blows to slip the barrel even a few degrees, and as long as the barrel nut was tightened correctly from the start (read greater than 35, but less than 80 lbs), the barrel nut will still be tight.
Bottom line is the choice of either drifting the barrel in the socket using a leather mallet, or pulling the barrel and playing the lets find the correct index under live fire, then re-adjusting with a few more barrel pulls, is yours. But, let me put it this way, you take the rifle to any AR smith, and your going to find that the handle on his leather mallet still warm as he comes out of the back room and hand the rifle back to you after correcting the problem.