I'm throwing around the idea of building a Glock mag compatible 9mm AR pistol for a house gun. My main concern is reliability. LRBHO is not entirely necessary. I'm not educated enough yet to figure out the best barrel length for this build but I am thinking 8"-10". What components should I be looking at for consistent function and reliability? Any help is much appreciated!

I think the key things to getting a reliable 9mm is getting a good buffer/spring combo and some require a little ejector tweaking. Just start with a 8oz or so buffer and go from there. Barrel length doesn't really matter as the pistol cartridge buts the powder off so fast short barrels don't effect it.

I went Colt Style mag. I went 8.5 CMMG barrel, with a Vltor A5H4 buffer(eliminates any spacer due to longer length, plus heavier weight at 6.83 ozs). In a standard carbine stock, with a rifle spring.

Now maybe I got completely lucky, but mine has been 100% from the get go, zero tuning. But I do not shoot suppressed either, so keep that in mind if you will be.

FWIW, I went with a PSA 9mm buffer, standard carbine spring and a home made spacer.  The best advice I was given was to start with that and tweak if necessary.

Just my $.02 but if reliability is your #1 concern, I'd stick with a Colt mag based system. The design (and the parts available for it) have been around a lot longer so its just a more proven system.

The other key IMO is don't buy parts based on price. Buy based on proven track record, even if that means spending more $. it's fine to try and save money if you are willing to tweak parts for a range toy but its not a good idea to do that for something you may need to use to save you or your family's life and with that in mind, there is a strong argument to make for just buying a factory built Colt 6951 and be done with it.

But if you want to build something up yourself, stick with well known parts with a strong history of quality and functionality.

For instance, I went with a Hahn magwell adaptor and Metalform mags which is probably the most expensive combo out there but it works with no tweaking, tuning or anything else. Rarely do you hear about anyone having issues with either of those items and I don't know that I've ever heard of someone having issues when using both together? (Although this is ARFCOM so now that I write that, some one will come out and claim they did )

I also went with a ramped bolt (RRA) and rounded M16 style hammer (LMT) and a Colt 9mm buffer with Sprinco "red" x-power spring and NFA spacer.

Get one from Mad Machinist.   Pistol caliber carbines is what he makes.

I've had feeding problems due to overall length of ammo, bullet type (FMJ vs JHP) and the barrel throat.

Longer rounds have fed most reliably for me.  Shorter rounds like Critical Defense 115gr Flex Tip gave me the most problems.

Bullet shape was another area where I would run into problems.  FMJ have worked best for my setups.  JHP that have the same shape as FMJ work fairly well.  Some of the newer JHP rounds like Federal's 124gr HST with a wider cavity in the bullet caused feeding problems in a couple of my barrels until I did mild throat job on them.  The Glock small frame lowers I've seen don't seem to use a feed ramp like the Colt lowers.  My two older DDLES frames have a feed ramp, but it appears the bolt strips the round directly from the magazine into the barrel.  The newer QC10 which looks like a copy of the older DDLES lowers don't have the feed ramp.  My third lower is an NFA Joebob Glock lower and it doesn't have a feed ramp.

Barrel throat shape might also affect feed reliability.  My first Glock AR had a Yankee Hill 16" barrel and I had to do a little throat work on it get it to feed JHP reliably.  My next build had an ADCO 7" barrel and it fed everything with no problems.  The barrel throat on it was more open than the YHM.  My last build was a 4.5" Faxon and it had the basic shape as the YHM barrel, so out came the trusty Dremel for a little judicious grinding to get it feed JHP reliably.  I've attached a couple of lousy pictures to show the ADCO and throated Faxon barrel to show the difference.

I like the Glock small frames lowers, but sometimes they take some work to get them to run right.  Hopefully, what ever you put together will work right from the gitgo.

Pete

ADCO

Faxon

He fixed my feed issues on my PSA barrel. Looks like the same throat issues lots of people seem to be having

For what it's worth,

When I built mine reliability and home defense was what I was looking for. I started with an adco barrel,  bottom loading hahn block,  cmmg bolt,  rra 9mm buffer,  and brownells mags.

I've had nothing but issues with mine not feeding into the chamber and the bullet getting compressed into the case.  I also would break firing pins after about 50-100 rounds. Recently I talked to cmmg and they recommended to switch to the dedicated/top loading hahn block.
I was told by cmmg that their customers have reported having feeding issues with the bottom loading block. When a top loading hahn block was installed it fixed everything.

After installing the dedicated hahn block, I was still having feeding issues when I hand cycled several mags. After a good cleaning and switching the lube (slide-glide lite to otis clp) it hand cycled perfectly.  I just put about 300 rounds down range through it yesterday without issues. Hopefully it will stay that way...

I built mine based on the Glock mag system just because I have a bunch of 9mm Glocks.  However, like Airgunner said, don't buy parts based on price.  I have yet to see anyone put together a good 9mm AR for less than what a decent 5.56 AR would cost.  However my main concern was fun at the range since you can shoot 9mm alot cheaper than 5.56.  

I have a QC10 lower and barrel and a 9mm bolt that I machined (ok, dremeled) myself.  I am no machinist or even a gunsmith but when putting mine together, I made sure that I got first rate parts, and made sure that they were put together right.  Mine has been 100% since the first round.  This is even with the Franklin Armory binary trigger (which certainly could throw a monkey in the wrench) but never a hiccup.

Personally I probably wouldn't want to use it for home protection (let's face it, it is still a 9mm plus I have a plethora of other firearms for that) but if push came to shove, I feel that I could trust it without reservation.