I inherited a rifle when my grandfather passed away. It is a post 64 winchester model 70 which he had a 26 inch douglas med-heavy (3/4 muzzle dia) barrel installed and chambered in 6mm-284. He had it done when I was still a kid in the early to mid 80's.
I have about 50 rounds of his handloads left out of the 100 he had loaded before he passed. He was using a 75gr hornady flat bast HP bullet and H4831. While the accuracy is not bad, I feel like it could shoot a lot better. It was probably about 5-6 years ago when I last shot it, so I can not remember the exact group size. But I do remember thinking it should be better than this. I got it out the other day and measured the twist rate and found it is 1-12. I also bore scoped it since I read this caliber is known to be a "barrel burner". Looks like it has had hardly any rounds through it as there is no throat erosion or heat cracking.

I kinda figure the bullets (being varmint/hunting type) might be the reason for the accuracy I saw. So I am wanting to try some better quality HPBT bullets. From what I see, 1-12 is on the slower side of what twist rates are offered for 6mm. Since I am new to 6mm, I was looking for suggestions on what bullet weight range to try out with the caliber/cartridge.
Should I stick with 70-80 gr bullets or lighter? I see sierra makes a match king 70gr. Or does anyone think 90-100gr might work.
I do not want to waste time and money trying too heavy of a bullet that will not stabilize, but would like to try the heaviest I can.

So does anyone have any experience with this cartridge that could give me some recommendations.

Off the top of my head, 87gr. Lemme check with JBM/Berger.

I'm seeing 75gr VMAX as very marginal at SG 1.1. The 65gr VMAX meets my personal minimum of SG 1.3.

Looks like the limit is about 0.9" long bullets.

Berger has a 68 grain Flat Base target bullet that should work, recommended twist is 1 in 13". Everything heavier than that they recommend 1 in 10" twist rate or faster.

https://bergerbullets.com/product/6-mm-68-grain-fb-target/

I'm seeing SG 1.4.

I've been using 3200fps, and a quick search shows that to be 300-400fps slower than typical. 100fps is worth aproximately SG 0.1.

SG is a new term to me. I had to google it to know what you were talking about. I guess in layman terms, it weather a bullet is/will be stable. This will have to be a subject for more reading when I get some free time.

The specific bullet he was using is Hornady #2420. Looking online, it looks like they have been discontinued. I have 875 of these (8 sealed boxes and 1 box of 75). So it kinda sucks they do not shoot exceptionally well. His load was 57gr of H4831 and a COL of 2.88. Quickload estimates this at around 3500 fps. I need to chrono some since I now have one.

The bergers posted above are just a little bit more expensive than than the 70gr SMK. I might pick up a box of each and test them.
As far as other brands, should I pretty much stick with 70gr and lighter bullets?

Yes.

https://www.jbmballistics.com/ballistics/lengths/lengths.shtml

That seems like a rifle from another era.

Bolt action; heavy barrel; long barrel; monster case capacity; slow twist; super high velocity,...

6.5X284 was one of the preferred long range shooting cartridge (1,000 yards).  It was a barrel burner (1,500 rounds) but racked up good scores while the barrel lasted.  The thing is, that would have a fast twist to shoot the heavy bullets.

Was he into varmint hunting... or benchrest shooting, perhaps?

I found the receipt from douglas barrels in some of his stuff after he passed. It is dated from 1988. Paid COD, if anyone remembers that.
He was into varmint hunting and deer hunting. Not long range that I know of. By the time I started getting into shooting, I was already living 6 hours away and he was getting too old to shoot anymore. So I did not get to talk with him very much about them. I now wish I would have.

He had this rifle but I am actually not 100% sure of its intended use. I do not think I ever really talked to him about it much before he passed.
He had a rem 700 in .17 rem he used on groundhogs, as I remember going with him a few times when I was a kid. And last he had a old mossberg 800B he had re barreled to 284 win that he used as his deer rifle. I have both of those rifles as well. The 17 rem has a headspace or chamber issue I need to have a gunsmith look at. The barrel is also a douglas he had installed in the mid 90's. It is very tight to close the bolt on FL resized brass, and it goobers up the case neck a little. I might have to have it set back a hair and rechambered.
The mossberg is also a douglas barrel, but it is a 22 inch and a light/sporter contour. The rifle is very light. Couple that with the shorter barrel and the cartridge, it kicks worse that a 3 inch magnum 12 ga slug.

Buy some Cerrosafe and make a chamber cast.  Brownell's sells it.  It is easy to do.  This will tell you what you have.  It will help you decide if it is what you want or guide you in what you want to do.

My description was not very good. When I say goober up the neck, I meant the mouth. If you start with a sized and trimmed piece of brass (no bullet or powder) and put it in the chamber, close the bolt, then remove it and inspect it. The mouth looks like you took the case, stood it up on a piece of concrete with the mouth down, the pushed down with your finger. Not really sure what is going on in there. Couple that with FL sized brass is still very tight to close the bolt, makes me think something is going on. It is not a benchrest rifle, the barrel on it is a sporter contour. It is also not a tight neck chamber.
He did have the neck turing kit for his forster case trimmer, but he used that to turn the necks on the 6mm-284 brass. He formed it himself from 284 win brass.
I have thought about getting some cerro safe, just never got around to it.

Another excellent little (and inexpensive) tool is Sinclair's Chamber Length Gage.

It will allow you to measure how long your chamber has been cut from base of case to end of the neck.  Your brass HAS TO BE SMALLER than this dimension or you have a dangerous, high pressure situation.

I have these little goodies for all my rifles.  $6 buys you real measure of safety.

Link to Sinclair Chamber Length Gage

So I was going through some of the boxes of bullets I have, and in the one open box of the hornady bullets I find a small bag of a random mix of bullets, 25 in total. Six 70gr, two 85gr, nine 87gr, and eight 95gr. Such a random mix of bullets. No idea when or where he got them, maybe one of his friends who knows how many years ago. I'm guessing to test them out. Who knows if he had more at one time.
I used a combo of a reloading manual and quickload and worked up some loads with the H4831 I have for the different bullets. I will try and shoot them this weekend IF the weather holds out. Maybe this might help me get a little idea of what different weight bullets do in this gun. Curious to see if the 95 gr bullets keyhole or not.

Ok, had a chance to try out the random lot of bullets I had.
Not sure if it makes any difference, but I was using new brass that he had formed from 284, but had not shot/fireformed. Also I loaded them a little conservatively, as I did not know what to expect. I used H4831, that was what my grandfather had a large can of (which I now have) of almost 8 pounds of it.

All the random bullets were HPBT, of unknown make.
Shot from the bench, front rest, rear bag, 100 yards.
I started out with the 87 gr bullets which I had 9 of. I shot a 5 shot group and a 4 shot group.They kept a group around 3/4-7/8. Not too bad, but there was one flier on both groups that opened the group to 1-1/2. The fliers most likely were me. Chrono put them at just shy of 3200fps.
Next I shot the 70 gr bullets which I had 6 of. I got a group around 1 inch. Speed was around 3500fps.
I then shot the 95 gr bullets which I had 8 of. I shot 1, saw no impact on paper. I moved to a circle higher up on the paper thinking it was way low. Still no impact. I fire one more at the dead center and still no impact anywhere I could see. I walk down range and inspect. I found the bullet holes in the plywood about a foot high and a foot to the left, and all 3 about 6 inches apart. The holes were not keyholes, but they were not perfectly round either. So obviously they were not stable at all. Speed was around 3000fps. I shot the rest of those into the berm to finish fire forming the cases.
The 70 and 87 might show promise if I decide to get a box of them to try, and actually work up a good load.

I decide since I have it out, I get the box of 50 rounds I had for it that my grandfather had loaded. Who knows when he loaded them (I was probably still a kid). From what is on the box, it is the 75 gr hornady that I have 875 of. 57gr of IMR 483. I shoot two groups of 5 rounds. I got 1/2 in groups! Velocity averaged at 3760 fps!! That is cooking!!
After cleaning and putting everything away, I run that load through quickload and it puts it at a dangerous load. It estimates chamber pressure of 64,000psi. However I read online that being a wildcat there is no "set" or "max" chamber pressure since it has not been standardized. The primers look normal, no flattening, no pressure signs, bolt was not sticky, no ejector marks, etc..

Only thing I can think, is several years ago when I last shot this rifle, the box of ammo I shot must have either been a different bullet, powder, or charge weight. While I can not remember exactly what the group sizes were, it was nowhere near what I got today. I think I am going to pull a couple of the bullets from the 40 rounds I have left and see if it is in fact these 75 gr hornady bullets I have so many of, and double check the charge weight.

I'm going to measure the case capacity today if I get a chance so I can enter that into quickload. I know internal capacities can greatly affect chamber pressures.
But like you said, I will probably stick with his load since it has no pressure signs.
If I decide to try another bullet I will work up a load.

I pulled the bullets on 3 of the rounds I have left that he loaded (that I shot yesterday). They are the 75 gr hornady bullets. Powder charge on all 3 was 57.2 gr. I then reassembled them to shoot.
I measured the case capacity of 5 of the ones I fire formed, and also 5 of the ones he had loaded and I shot, just to see if there was a difference. They both averaged out to about the same, or 66.64 gr of water. The cases are winchester (headstamp WW-Super).
I entered it into quickload and not much difference. It still says it is a unsafe load, but the velocity it estimated was pretty much dead on with what I was getting. Since there are no pressure signs I see no reason to deviate from his load. Who knows how much time and effort he spent working up this load while I was still just a kid.

When I saw the twist rate, my first thoughts were that heavier bullets would yield poor accuracy.  When rifles using the 6.5mm cartridges were first marketed by US manufacturers, they had problems with poor accuracy that was tracked back to slow twist rates for the bullet weights they were loading.    Suggestions that you use lighter weight bullets as noted above are correct.