I'd be curious to know exactly what the length of the shoulders were AND the OAL of the cases cases were when loaded irrespective of what the chamber seems to be.
If your cases were 1.760 or under, then it shouldn't be an ammo problem. 1.760 case length is the SAAMI length stated as the MAXIMUM a case should be when loaded.
For example, your chamber might "seem" to measure 1.766 the way you're doing it, so cutting the cases to .003 under (to 1.763) would leave the brass out of spec and too long for the chamber. Since most factory ammo case length measures (for me) at 1.750, then that would explain why you don't have any trouble shooting factory ammo.
Now, you say that you built the rifle yourself, and we don't know how well you understand how barrel to bolt headspacing works, but if your cases were loaded at a case length of 1.760 or under and still don't shoot, then I'd revisit and check your barrel/bolt headspacing.
At least check your chamber with go/no-go gauges. You might find that the "go" gauge won't chamber which would be proof that you barrel/bolt headspacing is too short.
I might be confusing the issue because up to now I've been reference only overall case length, but the same issues apply to shoulder spacing except for the specific measurements you're going for.
Your reference to ".003 under the chamber" does indicate that you are referring to shoulder length, but look to OAL length as well as shoulder length, and if your shoulder length actually IS the problem, then I would look even closer at your original barrel/bolt installation for correct headspacing there..
For reference, I have measured brand new unfired brass, and it typically measures 1.750, and that's the "factory ammo" that most of us have no problem running through our AR's.
Since I load for several different AR's, that's the trim length I've settled upon because 1.750 is "right on" for duplicating brand new case length of factory ammo.
For case headspace, I typically size the shoulder to 1.455 using the Hornady case gauge if you would like a reference to consider.
Also double check and reconsider your powder type and charge weight. It could be as simple as not running a proper charge in them.
I'm just throwing out a bunch of things to consider as you troubleshoot the problem. Hope I made enough sense because sometimes I even confuse myself.