Your getting barrel tension on the top of the barrel from heat to the wood.
Check to make sure that the rear hand gaurd does not touch the receiver, the barrel under the wood, nor the sides of the stock. Check the front hand guard against the gascyclinder, they should not touch.
Also check the middle band and the stock front band. Make sure that both are solid, and if needed, locktite the middle band to the barrel, and glass the front stock band to the stock.
Simple put, as the wood heats up, it streches and somewhere down the line, it is pushing the barrel down (best guess would be the front hand gaurd against the cylinder). The only tension that you want on the barrel is the middle band/front stock band that pull the barrel down at a constant rate. As for glassing in the receiver, the tell tale sign would be flyers as the receiver moves in the stock and changes the barrel tension from the middle band (read shots would not be more constantly downward pattern as the rifle heats up).