I see the only downside to a bolt gun in .223 and what you want is that the holes get hard to see through a spotting scope, depending on how far and how good your spotting scope is. Or how far off the paper you are. LOL. But like someone said, use shoot n see I guess.
Personally, I have longed for a bolt gun in a caliber that doesn't beat you up. It's kind of like my perfect picture of what rifle shooting is all about. I like shooting groups. And distance. And now with my eyes getting older, I like scopes. And sometimes an AR just makes you want to pull the trigger too fast. It's almost hard to NOT shoot fast with an AR. Which sometimes just gets dumb. With the prices of ammo. Don't get me wrong, they are much more capable of dealing with threats in a dangerous situation, as you have more ammo and firepower on tap. But for fun shooting, I think a bolt gun is the way to go.
And yeah, as far as recoil goes, pfffffft, .308 can get stiff and that's not overly fun for me either. Although you did say you like recoil. What if you go with something inbetween. I have always thought that a 6.8 bolt gun would be cool. Or you could do something more traditional like a .243. Or there is that competition round, what is it, the 6mm PPC something or other. All that ammo would be harder to get though, besides the .243.
If you do get one, let us know what you got!!!