Just remember, before you spend any $ on custom gun stuff (besides a trigger, because I agree, rem factory triggers suck)....learn to work your gun.
If you are shooting at the extent of your rifle's capability, even if that is only MOA or whatever precision.... you will be a good shooter with a rifle that can put every shot through a ragged hole.
Learn how to work your rifle, your scope, and deal with changes in range, temp and such.
Until you learn to work a rifle, a scope, and how all of those things go together....a high end precision rifle (and all it's costs) are not going to do you much good.
The way I look at it is this: you got yourself a good rifle for work that doesn't need extreme precision at extreme distances. That's not a bad thing to have as long as you understand it's limitations. Once you figure out it's limitations and it's positives (like not having an extremely long barrel......long barrels are great at some things, but suck at others), then think about what you want in your next gun. What you want it to do, what you want to do with it.
But from my experience shooting a variety of guns in a variety of conditions, until I had time shooting, I didn't truly know what mattered to me in order to get the results I wanted.
Worst case, you shoot the original barrel out and you rework your action into something you want based upon your experiences and your desires.
Best case, you figure out what you want sooner than you burn that rifle's barrel out. Now you have a hunting gun you understand very well and you can build the rifle you really want. It will be about as expensive to build a whole new rifle as it will to eek the performance out of the action you have. And you have 2 rifles instead of just 1.