Taper pins can be installed from either side, if they were machined from that side. So the answer to your question is no. There does seem to be a standard to drill them from the right side, so they come out to the right, like the receiver take down pins, but there is no specific reason for them to be that way. At least one company is known for installing them from the other side.
If someone removed taper pins and tried installing them from the wrong side, they wouldn't get very far because the pins are tapered, and the holes on the "wrong side" are too small to allow the pins to go in more than a fraction of an inch.
If you have a canted FSB, someone simply installed and pinned it at the wrong angle, or it was done properly, but the whole barrel assembly may be cocked in the upper receiver. That can happen if the slot in the receiver, that accepts the barrel alignment pin, is too large, or if the alignment pin is snapped or loose, and it allowed the barrel to shift to one side when the barrel nut was torqued.