The accuracy of rifle is not limited by the barrel diameter.
I've seen some regular "hunting" profile barrels that would put dime sized groups on the paper at 100 to 150 yds.
Try different ammo and see what gives you good groups.
I don't know what type of varmint you are hunting but most of them don't require a "special" bullet. This means a standard 55 grn. bullet will take care of varmints like foxes, ground hogs, praire dogs, feral cats, etc. just fine.
Any one willing to use an AR type rifle as a defensive/offensive weapon against people shouldn't have any issues using it to shoot an animal 35 lbs. or even smaller.
I have a recently built AR (although built for varmint hunting to I used a bull barrel upper) I put together specifically to varmint hunt with.
My CAR15 with the ceiner kit is more than accurate enough to small game hunt with (head shooting squirrels).
Don't "stereotype" your rifle without trying it was some different brands/bullet weights.
Good luck.