I have owned nearly every single slug gun imaginable over the years, and I found the H&R to be the most accurate sluggun of all those I've owned.
I didn't buy this gun because it's what I wanted, but I bought it for my fiance to hunt with. I bought the 20 gauge Ultra Slug Hunter because I needed something with minimum recoil. However, after shooting it to sight it in it became my "go to" slug gun. I put all my other guns aside and used this one exclusively for deer season.
It sounds to me like your problem may be in your mounts and/or scope setup. There is no reason it should ever shoot that bad. I would venture to say the problem is in your optics setup.
As for slugs, I have found that Lightfields, Remington BuckHammers, and Hornady SSTs are some of the best performers. It does seem that each rifled slug bbl is a little different, however, so what works for one guy might not work as well for the next guy.
I am 11/11 on deer since buying that 20 gauge and my fiance is 2/2. I have complete confidence in this weapon and, if I were you, I wouldn't give up on it just yet.
Here is a pic of one of the deer that met its demise to it. If you look closely, you can see the gun in the background.
Clint