A few years back, I had an SP1 upper that I actually had to relief the left side of the notch in the upper receiver (the way you described) to rotate the barrel and FSB on a Colt "CMP CHROME BORE" FSB to center rear sight. I peened the threads on the opposite side to prevent it from rotating back. That was a Colt factory upper set up. It happens. It just takes the upper being a little off and the index pin or FSB to be a little off. Add it up and it becomes a problem.
In the recent case of my AR15 Sport pencil barrel, I assumed it was the barrel because the DSArms M4 barrel was perfectly centered in the upper. As it turned out, when I pulled the AR15 Sport barrel, the index pin and FSB were dead nuts on. even though it was a minty 604 upper, there was a almost imperceptible amount of wiggle. Apparently that wiggle was enough to cause a different to the angle of the FSB depending on whether barrel or receiver was clamped. When I clamped receiver instead of barrel, it moved the FSB at least 1/16 or a little more to the left, centering it nicely.