Since the velocity of the bolt/carrier is a function of the gas pressure in the bore, not the size/weight/velocity of the bullet, I have no problem using my lower with other calibers up to and including .458 SOCOM. As long as the upper is gassed properly, the action will operate just like a .223- the gas in the bolt carrier making thing go doesn't care what caliber generated it.
The extra recoil you feel at the shoulder is generated before the bolt even unlocks, so it has nothing to do with any possible extra wear on your hammer or trigger pins and if your upper fits the lower properly, the recoil energy is transmitted straight back into your shoulder. If it's not strong enough to injure your shoulder, it won't hurt your receiver. An upper that fits the lower poorly could put extra force on the takedown pins.
Where you have to watch out is the 9mms. The action has now gone from gas operation to straight unlocked blowback and things can get violent in there. The AR-15 design was never intended for that and there's just not enough room in there for a suitably heavy bolt.
Just my opinion, YMMV, etc.