Groups can open up at longer ranges. That is why i test my loads at 100 and 300 to make sure it holds together. But ES/SD numbers are what they are. How do they get worse at 300 yards?

If the load doesn't work at 300 then you haven't found the load yet. Keep working.

The weather changed.

The number one problem I see is someone that selects a load in a region where a slight charge change causes the group size to open up.

A plot of vertical dispersion with charge weight should look similar to a sine wave.  Low charges along the trough cause relatively small changes in group size or dispersion, then as the charges increase, one charge produces good results, but a grain difference either side opens up, and then at or near the maximum charge a peak forms where accurate loads live across a couple of grains difference in charge weights.

A little short knee between the peak and the valley will work if it's sufficiently wide, but it's most likely not.

Your load isn't tuned if it's not accurate over a broad range of charge weights.

Another issue is trying to develop loads without a wind flag.  It's not a crutch, it's a tool, and even the most rudimentary flag made from a piece of surveyor's tape on a stick provides important information.  Plus it saves barrel life and components.  If your groups opened absent vertical stringing, then the problem is wind.  A very light breeze across the range will move the bullet laterally enough to destroy a group, and a left to right breeze on a bullet spinning clockwise also depresses it vertically down due to precession; this effect is observable at 100 yards.

Another issue is rate of fire and barrel heat.

I have one light barreled .223 Rem rifle that shoots 3 shots in short order well enough, then heads to 1 or 2 o'clock in a string that stops by when the barrel is cooled.

Take a look here for a discussion of Creighton Audette's method -

https://precisionrifle.wordpress.com/2012/07/13/creighton-audette-ladder-testing/

The page shows a plot of charge weight against charge weight.  A similar plot of vertical dispersion vs charge weight can be made that will identify stable regions.

Audettes set up tall targets with a vertical line and scale, then started the string of shots at the bottom aiming point.  As the charge weight increases, the groups climb the target.

Accurate charges insensitive to small increments of gunpowder fall on top of each other, exactly what we want in loads that remain accurate over weather changes.

The barrel of your rifle can reach 160F to 190F at relatively slow rate of fire.  Groups may open.

If you are serious about precision you should measure the rate of growth of the throat as it erodes.  A .223 Rem rifle used in across the course matches will erode about 0.001 inches every 100 rounds.  Next thing you know, an adjustment of seating depth might be necessary.