How many different loads have you shot through it and what were they? I reload for several Enfields and have found that even minor changes in the load, but especially different bullets, can cause a horizontal variation that I don't see in many other rifles. Your case does sound extreme, however, so I would not rule out the possibility of a warped or ill-fitting stock. I would check near the muzzle, first, and see if you notice the barrel making contact with one side. Then, take the upper handguards off and look down the barrel channel for the same issue. You can also do the paper test to see if/where the barrel is floating or touching. I would expect it to touch underneath the muzzle end, but check the entire length. Also, as was suggested above, you can shoot it with the top handguards off, but it's hard to say if that will give you a definite answer, either. Check for any pressure marks in the handguards and forend and take note of them.
Oh, and are you sure that the king screw (main action screw) is tight? Enfields are very pressure dependent in how the stock interacts with the receiver and the barrel, so the least amount of play/pressure in the wrong spot can throw them off. EDIT TO ADD: I would avoid sanding anything or removing any material until you've pursued all other options. It's all too easy to go "too far" with these rifles.