That's not the best cleaning method.
What you're doing is leaving fouling from the cleaning brush. The brush is rubbing off in the bore, and the dry patch won't remove it.
What you're seeing later is fouling left by the brush.
Try this:
Don't dip the brush in the solvent, that contaminates it. Use an eye dropper or a Accu-Bore plastic pipette from Brownell's to apply solvent to brushes and patches.
Wet the brush with solvent and run it all the way through the bore. Don't reverse the brush in the bore.
Make 10 to 15 passes through, keeping the brush wet and wiping the rod off each time it comes out.
Wet two patches with solvent and run them all the way through and out, one at a time.
Let the bore soak for at least 30 minutes.
READ THE SOLVENT LABEL, SOME CAN DAMAGE A BORE IF LEFT IN TOO LONG.
After soaking 30 minutes, wet a clean patch and run it straight through.
If you see any blue or green stains or any black stains on the patch, let the bore soak another 30 minutes.
When you run a wet patch straight through and it comes out without any stains the bore is clean.
Run a couple of dry patches to dry the bore, and then run a patch with some lube to prevent rust.
After the lube patch, run one dry patch to remove the excess lube.
If you're going to be storing the rifle for a few weeks or longer, 2 or 3 days after cleaning, come back and run one wet patch through and let it soak, then run another wet patch after 30 minutes. If that patch has any staining, soak again.
This will remove any fouling that might be brought out of the rifling after a day to two.
If you take a dry patch and "pump" it up and down a gun barrel, the patch will have gray stains on it. This is metal staining not fouling. If you rub cloth on metal it'll get a gray stain from the friction on the metal.