I have been hitting the bolt release for the first round for over twenty years and I have never noticed any accuracy difference in releasing the bolt release to strip a round as opposed to the bolt carrier cycling via the gas system. If it were an issue of releasing the bolt stop affecting accuracy on that first shot, I would be concerned about safety. If it is only the first round every time stripping a round out of the mag that causes an accuracy problem, I would be concerned about bolt lockup.
First thing I would do is check a series of magazines to determine if it is a magazine issue. I would check to see if the follower is bad, hanging up the bolt hold open causing the rifle to not lock up properly. I would also check the bolt release to make sure it isn't snagging the bolt carrier on a full mag causing a bad lockup problem. I'd check for sheared off bolt lugs and bolt cam pin hole wear/cracks as well.
If your barrel is clean on the first few rounds, it will not print where the rifle is zeroed until the barrel is warmed up and just a little dirty. A cold barrel will print differently than a warm barrel.
I would bet any amount of money this is a case of you flinching and being a little anxious when you take your first shot. After three or four rounds, the brain will adjust to the chaos involved in rifle fire and you will calm down. I have seen this MANY times and after many years of rifle shooting, if I am not calm when I place my first shot or if my mind is elsewhere, like stressing about work, it will affect my shooting. Ruling your shooting out as the possible cause so fast will not wear well will experienced rifleman. EVERY rifle I own is a better shooter than I am.