I have a 10.5 SBR I was shooting at the range the other day with a 1-4x at 50 yards.

I was trying to zero the new optic and was using AE 5.56 55gr, which I have not shot out of this gun before. I could not get it to group for the life of me. It was very inconsistent, with a group of about 5 inches. The only thing I have done to my gun lately is put on an AAC Flashhider and a new optic.

I checked the optic to make sure it was tight and did a visual check on the rest of the gun.

Then I switched to Perfecta .223 55gr, which worked just fine. I was shooting groups of about an inch/inch and a half, constantly.

Both were 55gr, the difference being brand and 556 vs 223. I assumed the AE was higher quality....

Is this kind of thing normal? Is it just that particular barrel/gun that did not like the AE ammo? I even tried another box of AE to make sure it was not a bad batch, same problem. I have shot other 223 out of it in the past and it is all plenty accurate, but not this 556. Why would my gun not shoot 556 as well as 223?

All the Perfecta I shot did perfectly well, and it always has in the past.

It's all perfectly normal. Every barrel is a little different and it behooves the serious shooter to test several types to see what works best for their gun.

5"+ groups at 50m shouldn't be normal with any ammo other than birdshot. Usually even the shittiest ammo I've ever benched like Russian steel case would do about 4ish inches at 100. What you're describing indicates something isn't right, your groups should look a lot closer to what you're describing with Perfecta, but while AE isn't Black Hills or FGMM it is decent ammo, and I'd put it above perfecta subjectively for myself.

Is this a factory or homebuild?
Have you tried any high quality ammo?
Is it a good barrel?
If so, what twist is it (also, are the rounds keyholing)?
Is everything on the barrel torqued properly (barrel nut, flash hider)?
Is everything on the scope/scope mount torqued properly?
Was the rifle properly rested/benched?
Have you verified the same results with another known quality shooter?

I'm 99% sure the answer to your problem is in one of the above queries. If you give us more info we can likely quickly narrow it down. For now, personally, I'd make sure everything is torqued down properly, buy a box of FGMM or Black Hills SMKs (69SMK if you're shooting a 1/9 barrel, 77SMK if a 1/7), go to the range with a good rifle rest, and print out 4 of the best 5 shot groups you can muster. That would eliminate lots of variables, and possibly solve the issue altogether.

You did torque the FH properly correct?    

15-20 Ft lbs according to this..... 

All the 223 I have ever shot through it has not problem making inch groups at 50 yards.
Is this a factory or homebuild? Built
Have you tried any high quality ammo? Usually just cheap stuff but it always does fine except this AE
Is it a good barrel? YHM, so not really
If so, what twist is it (also, are the rounds keyholing)? no keyholing
Is everything on the barrel torqued properly (barrel nut, flash hider)? Yes
Is everything on the scope/scope mount torqued properly? Yes
Was the rifle properly rested/benched? It was rested on a range bag
Have you verified the same results with another known quality shooter? No

Every time I have shot it with various 223 it does great. The AE 556 just shot like shit for some reason.

I will try and make it to the range some time this week and test some other ammo as well.

Yes

If the flashhider was not torqued correctly, Why would it only affect the accuracy of the AE 556, but the other 223 shoots perfectly fine?

Given the above, then yes maybe your rifle just *really* hates that ammo for some reason. Usually .223 pressure ammo tends to favor accuracy over velocity while 5.56 pressure tends to prefer velocity over accuracy, but usually the difference is fairly subtle. I'd definitely buy a box of good match ammo to verify, or just not buy that AE 5.56 ammo anymore if it's the only ammo giving it fits anyway lol if you like lighter cheaper ball ammo I'd see how it likes Wolf Gold as it's about the best deal on m193-ish ammo out there right now. If your rifle likes it you're (no pun intended) golden.

It does seem overly large groups for that distance for some reason.  But my recce is a 2-3 MOA gun with most cheap ammo.   But it won't group AE 55 grain for crap.  5"+ at 100 yards.  So.....  It's not overly surprising to me to see a certain load just really suck.   But it is disappointing.  I would say it's not a hardware issue if you're grouping fine with other ammo.

Occasionally, I come across some very shitty ammo. Bad runs and bad lots of ammo turn up sometimes.

More likely, your particular gun hates that AE 556.

I have not had luck getting good groups out of my 10.5" most likely due to the crappy "shoot it like a pistol" stock.