I tend to wear gloves during a significant portion of the year....
I find I can manipulate everything I need with a rifle wearing a well loved pair of Black Diamond Patrol gloves. They're almost 100% leather, so the key is buying a touch on the small side. Once they pack out a bit, they fit perfect for medium dexterity activities. I can shoot, reload, rack, and manipulate my pistols with them too, I just can't fill pistol mags. I keep my AR food on stripper clips, so loading those mags is easy with the gloves.
They may be more than you're looking to spend though.