I took out one of my M4 builds for the first time the other day.  I fired 90 rds and it ran perfectly.  The issue I was having was that it was shooting to the right quite a bit.  I was shooting at 50 yards.  I got it zeroed, but the rear sight (detachable carry handle) is cranked over pretty far to the left.  It isn't maxed out, but it's pretty far over IMO.  It will actually go another 20 clicks before it is maxed out.  The front sight is nice and straight, not canted at all.  I'm actually wondering if the carry handle is the issue.   It seems like it isn't 100% centered on the receiver, and it hangs off the right side just a bit.  I guess I should try a different rear sight before doing anything else.  In the meantime I would appreciate any thoughts on this.  

Parts used:
Colt Match Target lower
RRA flat-top upper
RRA M4 barrel  (chrome lined, heavy under handguards)
Colt m4 handguards
Bushmaster? carry handle  (Cardinal forge)

If you are sure the FSB is straight, then it most likely is in the carry handle. Take it off and remount it. If you have a buis, try it and see if it's also off.
It doesn't take being off much. If needed, you could probably shim the carry handle on one side.

Could also be an over-torqued barrel nut.

I would think if the barrel nut was over torqued, the front sight would be canted. That would have been my first thought but the OP said it was straight.

I think I figured out my problem.  Compared the problem gun with another rifle of mine with flat-top and carry handle.  It appears the rail is out of spec on the one upper.  Neither carry handle will fit properly, they are offset to the right.  Guess my options are to live with it or buy a new upper receiver.   Pics to show what I mean:
Good:

Bad:

Contact RRA. The most they can say is no.