You have to consider the Napoleanic tactics of the day, where lines of troops stood about 100 yards away from each other. If you take an 1847 Walker, 1860 Army, or similar military pistol of the era with period correct sights and dimensions, and shoot at a 100 yards you'll find it shoots remarkably close to point of aim. The sights were designed to enable to shooter to engage a line of troops at 80-120 yards and they do exactly that.
For close quarter combat with an infantry pistol like the 1860 Army, or in the case of a horse pistol like a Walker or a 1st, 2nd or 3rd model Dragoon,where it was designed and intended to be carried in a pommel holster on horse back, the assumption was the pistol would be pointed and shot at the close ranges involved, rather than aimed, so the 100 yard regulation of the sights was immaterial.