I have a build that I've shot quite a bit using lighter (55 gr and 62 gr) .223 loads and have had no issues with those. Yesterday I tried some of the new AAC 77gr 5.56 (with their house brand bullet) and got numerous bolt over brass malfunctions with that ammo. Tip of the round above the chamber, base is under the front of the bolt, bullet has been set back into the case and the case gets crushed/dented by the bolt. This happened 7 or 8 times out of 40 rounds fired.

Tried it out of 3 different magazines and had the same failure. My son and my friend both shot some out of their rifles with no issues, so I'm not looking at the ammo as the problem.

I tried a few single rounds loaded into a mag and the bolt was locking open, so it's not short stroking.

The build is a 16" Rosco Bloodline, mid length gas with a Toolcraft BCG and an Armaspec Stealth Recoil Spring with their H1 buffer. Gas block is not adjustable. Based on the fact that the gun runs fine with lower powered ammo, I'm thinking I'm overgassed with this ammo and the bolt is moving too fast?

I would agree.  I had a rifle that did that.  Put 9mm buffer in and problem went away.

Super overgassed.  Bolt is outrunning the magazine.

You can try a heavier buffer or stiffer spring for the cheapest fix.
I'd still make sure your other ammo functions with the changes also though.

This is with AAC's factory 77gr. load, rifle is unsuppressed. Appreciate the responses. I ordered an H2 buffer, will give that a shot.

One poster recently responded that AAC's 77 grain load was running 2875 fps from his 18" barrel. THAT IS A HOT LOAD!

I'm not sure it's safe.

My match grade ammo with 75/77 grain bullets run around 2650 fps from my 20" barrels. YMMV.

It's the ammo, not the gun.

I've seen several knowledgeable and experiences shooters report that some lots of that AAC 77gr are WAY too hot, and inducing this kind of malfunction.

I have seen the mixed reports on the ammo, and it's worth bearing in mind. I will send some through the chrono next time and see if  it's running hot.