Would a 26 inch 1:12 twist be fast enough to stabilize 175s?

Yes

Most likely.  FPNI strikes again.

Any type that might work better than others?

My rifle shoots Sierra TMK’s very well.

I used 185 Juggernauts in my 12 twist @sea level with no issues or apparent BC degradation.

Maybe.

My FN SPR A1 24" 1:12 would shoot bugholes with 175 SMKs and 178 ELD-M.  That rifle made ringing the 10" gong at 1K look easy.  At closer ranges, I once shot a quarter at 200 yards.  My buddy gave me some crap about 'how many shots did that take?' so I had him put a quarter downrange.  I hit it 5 for 5.  I still have the quarter as a souvenir of a great range day.

On the other side of the coin (pun--LOL), my Rem 700 26" 1:12 will shoot fairly tight groups with 168 SMKs and Hornady 168 BTHPM....but if I load the mag with anything heavier, it looks like I shot the target with a Mini 30.

Factory 175 FGMM shoots about 2 MOA and factory 168 FGMM shoots under 1 MOA.  My best 178 ELD loads are shooting 1.5-2 MOA while my best 168 BTHPM loads are .5-.75 MOA.

I've tried 3 different powders--4064, Varget and 4895-- 10 charge weights with each powder, fully prepped LC LR and Federal Match brass and I've played with the seating depth.  COALs from 2.800" all the way to 2.950" that I have to single load because they don't fit the mags. Nothing worked.

I know it's not me, or a bedding problem, or a scope issue because I have shot back-to-back groups--175/178 gr groups are always embarrassingly huge and the 168s are nice and tight.

My Remington just doesn't like the heavier bullets.  On the bright side, this gives me the excuse I needed to rebarrel it to 6.5 to take advantage of the 1760 yard station my club just opened

Gotta be a reason AI switched from 12 to 10 for a lot  of its .308s.  If I had a choice, I'd go with 10.  Although 12 would probably get job done with a 26" barrel.

JBM's length library shows the SMK at 1.240" long which matches Berger's Twist calculator default. The calculator says it has a stability factor over 1.5.

Bullets designed for trans-sonic stability are more critical, but yes.

What about bullet length?

I was reading shorter bullets don't need as fast of a twist rate?

The community refers to bullet weight when discussing required twist rate because that is the most common bullet attribute when in reality it isn't the weight of the bullet that requires a tighter twist rate, it is the length.  It just so happens that length and weight for a given caliber are directly proportional, so weight wins out over length in the discussion.

Which calibers give you grief? Furthest I’ve got to is 1200 yards and the 6.5 ELDM has proven worthy. I’m at 6500ft elevation.

Has anyone tried match burners with any success?

I think 1/12 twist only starts to become an issue with the plus 200 grain bullets.  I say this based on my experience with a 1/12 twist rifle in 300 WM.  It shoots up to 195 grain bullets with very good accuracy, but I haven't been able to develop a 208 or 220 grain load with a high quality bullet that achieves similar sub moa accuracy.