I have a vortex razor HD 5-20 (Gen 1) mounted on my M82 in a Larue LT-104 QD mount.  This optic has that dreaded 35mm tube.  I already had the mount from previous guns so just purchased the 35mm inserts.  It seems to hold a zero during range sessions.  Once I pack it up into the factory Pelican case and take it out again for the next range session it seems to have shifted a few MILs so I have to re-zero.  I noticed Larue makes a 50BMG version of their QD mount.  Do I need to spring for the 50BMG version which has 3 levers instead of the 2 on the LT-104?  Trying to figure out their 50BMG version "necessary" as I'm only a casual shooter not backing this thing through hell and back?

In all honesty I guess I don't even need the QD feature since it's basically dedicated to the M82 at this point.  I have no funds for a different optic.  What say you?

Get proper rings/mount for the 35mm tube, using the adapters is what is causing your shift. You aren't getting the adapters back in the same places in exactly the same way, just do away with the need for the adapters with  correctly sized rings/mount.

I’d think the fact that LT makes a mount specifically meant for .50’s should be a clue as to the best course of action, but I don’t think your ring inserts are helping much either. Also, it seems like most people have their LTs adjusted pretty loose. I’ve always had mine on so tight that I can barely get them off without tools, and I’d probably go double on that advice with a .50.

It's not ring inserts, they're replacement ring halves. That portion isnt affecting any zero shift - if they have been installed correctly with the bottom portion torqued before the top portion.

The Larue lever mounts should return to zero when coming on and off as long as you push the scope/mount forward each time and you are tightening the levers nice and tight. If the levers aren't tight enough, the mount can shift front to back a bit until the recoil lug stops it. This will lead to a little bit of zero shift, how much will depend on how straight your rail is. If your rail is perfectly straight, you're probably dealing with under .1moa of shift. If the rail isn't perfectly straight, you may see as much as 4MOA of shift. This would usually show up during a course of fire if the mount is shifting under recoil.

My test for regular rifles with QD mounts is to stand the rifle on the butt stock and put all my weight down on the optic. If the mount will shift, I tighten the QD system. (Test is done after mounting with the mount pushed forward in the slot.) With a .50, you may need a tougher test.