I am contemplating hunting with an AR and an AR pistol makes a lot of sense rule-wise in my state. I currently have a 10.5" 5.56 AR that shoots my defensive ammo really well (Federal 64gr Power Shok 223). The gun is not really set up for hunting yet as it had a fixed front sight post and an Aimpoint PRO. I have been considering getting a dedicated 300 BO upper 10.5" with a bare rail and mounting a spare 2-7x33 Leupold I have on it.

Should I expect much better terminal performance from the BO or are they about the same on deer?

with 110gr barnes rounds, a 10" 300blk should provide about the same energy as a 16" M193 and makes a bigger hole.

This. I personally find even a 16in 5.56 to be marginal on deer. A 10.5" 300blk with 110gr TAC-TX is going to be miles better so long as you know your ranges.

300BLK Federal Fusions will slay some deer also (I use a 10.5").  They expand down to pretty low velocities.

these:
https://www.targetsportsusa.com/federal-fusion-msr-300-aac-blackout-ammo-150-grain-soft-point-f300bmsr2-p-76580.aspx

Where you hit a deer makes all the difference but for a typical chest shot anything 30 cal beats anything 22 cal.  The 22 cal relies on its high velocity to cause damage the 30 cal even in 300 BLK will have better penetration and more energy.

You guys are really convincing me... I'm an experienced deer Hunter, been hunting since I was a kid mostly with high power rifles but got into 44 and 357 rifles about 10 years ago and now love hunting with "just right" powered rounds for deer under 100 yards.

Anything have a bad experience with 223 out of a 10.5?

My first deer came from a 64gr federal fusion out of a 10.3 5.56 with irons. I still use it for under 75yards.

But I am also contemplating either 300blk (with 110gr tsx), or 308 but it would require more money to get setup.