Looking for the best way to run subsonic ammo unsuppressed in an AR pistol 8" 300 Blackout with pistol length non adjustable gas system.

Hand loading 220grain to ~950fps.

I played around with different buffer weights and bcgs and multiple setups worked. since multiple setups work, is it recommended to go with a heavier carrier and lighter buffer or vice versa?

Full auto bcg and 3oz buffer - some short stroking

Semi auto bcg and 3oz buffer - less short stroking than full auto bcg but not perfect

Semi auto bcg and 1.8oz buffer (1 steel weight + aluminum spacer) - perfect

Lightweight bcg (8.8oz) and 3oz buffer
- perfect

Full auto bcg and 1oz buffer (empty buffer) - perfect

Sounds like you figured it out. Why do you want to run subs unsuppressed?

No suppressor yet.

I figured out ways to make it work, but my main question is - is it better to have a heavier bcg and lightweight buffer or lightweight bcg and standard weight buffer?

You are ending up with the same total reciprocating weight. When you get your suppressor, you most likely will have to change it, and the buffer is much easier to change than the bcg.
Glad you got it sorted out, and good luck with it.

I run a very similar load.
I use a standard bcg and a 4oz buffer in a 9" pistol length 300 non adjs gas system and I have had great results suppressed and unsuppressed
It feels a little "slow" unsuppressed though, like it could short cycle, but manages to move back enough to eject and pull another round

I bought a PSA 8” .300bo upper with a PSA premium BCG which I think was a FA carrier.

Would not cycle subs unsuppressed.

Swapped in a lightweight (8 oz) BCG & std. 3 oz buffer, runs subs just fine.

Powder can have an impact.  Accurate 1680 usually will run subs unsuppressed with a carbine buffer and M16 bolt carrier.  Sometimes they won’t lock back on an empty mag but still extract and eject adequately.  Since I rarely shoot subs unsuppressed (maybe to check a new load for stability) that’s good enough and with the suppressor I get full function to lock back on empty.

If I shot subs unsuppressed a lot I’d probably build a light buffer out of my KAK buffer-tinker-toy kit. It would suck to forget and run supers with that set up though.

7" 300BO Palmetto State Complete Upper and lower kit with 'Full Auto' BCG, standard carbine buffer, and spring.
208 Grain Hornady Black factory loads.
Amend 2 5.56 Magazine and Magpul 300BO Magazine
Unsuppressed - stove pipes ejected brass every time.
Suppressed - will eject brass no deflector strikes about 3:00, but will not cycle next round out of the magazine into the chamber. It will either go fully into battery on an empty chamber or will catch the round halfway into battery between the feed ramp and magazine.
Suppressed - Removed 1 weight from buffer - no change
Suppressed - Removed 2 weights from buffer - no change
Suppressed - Removed 3 weights from buffer (no weight in buffer) - will run semi-auto for about 3 rounds and will either go into battery on an empty chamber or catch the round halfway. Also the bolt will not lock open after the last round.

I have an Odin Works Adjustable (2.1 - 4.2oz) AR15 Lite Buffer on the way for Monday delivery.

What says the hive:
Buffer still needs to be lightened up?
Needs empty buffer with lighter or shorter buffer spring?
Heavier (H1) Buffer?
Adjustable Gas Block?

Ahh, the old "tuning" rabbit hole. I love doing it, but man it ain't cheap and it's not quick. I'll just give you my end-state: I follow Sprinco's advice to do as much with the spring as you can, to cut reciprocating weight - 10.5" BA DRP barrel, SupArms Gas Block at 28 clicks out, Sprinco "Orange" spring, Carbine Buffer, Rugged Razor, Magtech 200's. 3 to 4 ejection, 100% reliability. I just eat the extra when shooting Supers - don't feel like adjusting the block.

I'm using a standard carbine buffer + Nemo spring on a 9" 300BLK. Zero issues.

https://nemoarms.com/product/300-blackout-custom-carbine-buffer-spring/

You realize everything is going to change when you get your suppressor? Why don't you want an adjustable gas block? It's far more practical to readjust a gas block than to have to buy an entire collection of different weight buffers, springs, etc and swap them all around every time you want to change something.