What jig did you use on this lower, did you use a drill press, mill, router, did you drill the safety selector hole all the way through the receiver in one operation, or did you turn it over and drill side independently?
If that selector hole is actually off, I am wondering if your pin holes are off by the same amount.
When doing my lowers I drill all of my holes 1/4 inch deep before I start the milling process and when I do them, I do one side, turn it over and do the other side, that way I don't have to worry about deflection, even using a hand drill at slow speed you can drill each side 1/4 of an inch with out worries, and then mill the FCG pocket out.
That said, I am still going to say, completely assemble the lower with the safety detent, trigger group and hammer to see how everything lines up, cause I swear it looks like that selector is to deep, and you have to have the rest of those components in place for everything to line up.