I think it depends entirely on the individual. I have heard for 25 years that you can’t carry a 5" 1911 concealed. "It's too big." and "It's too heavy" and "You have to get an OM size".
I have been carrying a 5" 1911 concealed regularly for about 22 years. If it wasn't concealed, I would have found out about it by now.
I have a new XD 45acp that I picked up a few days ago. I have carried it for one day in a piece of crap holster (the one that came with it) and it concealed just fine under a sport coat. It will do better in a real holster.
You are going to have some trouble if you are 5'5" and 130lbs or in a cheap holster, but you don't have to be 6'6" and 300 either. I am 5'10" and about 190.
A good holster (I like Alessi, Sparks, and DelFatti if I can wait that long, but others work just as well) on a good belt under a thick shirt or sport coat should do.
Contrary to marketing strategy, it is butt length that helps with conceal ability, not barrel length. Barrel length can play more into it with IWB, but even IWB I carry a 5” in an Alessi GWH.
Unless you are very small in stature, I would think that if you will commit to the investment in a good holster and belt and the right cover garment, there is no reason you cannot conceal a 4” XD.
It depends on how committed you are!