Your eye is in a different place when the group moves. Different in relation to the optic (does the same for irons, or a scope and probably a red dot).
People who shoot competition at longer ranges can usually tell you that when the move from standing to sitting or sitting to prone, how many clicks up/down or left/right they have to adjust to keep their groups centered.
Consistency is key. That's why nose to the charging handle is so important. I set my scopes/red dots up so I keep my nose to the charging handle. Means I sometimes have to buy a second scope mount when I buy a different scope, but it works for me. If I shoot it with the scope on it, my nose is on charging handle. If I take the scope off to use the back up sights, my nose is on the charging handle.