The buffer will never strike the retaining pin when the rifle is assembled. The bolt carrier, even in the full forward postion, keeps that from happening. If the buffer did strike the pin, it would show damage at the very least, and break off in a short time. Note that as you close the upper on your rifle, the rear of the bolt carrier pushes back the buffer.
There is a groove or notch on the bottom of the bolt carrier that clears the buffer retaining pin when the bolt carrier cycles. Where that groove or notch exits the rear of the bolt carrier it often leaves a high spot or burr. This burr repeatedly hitting the buffer is what does the damage your photo shows. The buffer rotates around, and the damage is all around the rim.
Feel the notch on the bottom rear of your bolt carrier... is there a burr? If so, it will chew up the buffer. Use a grinding wheel on a dremel, and dress up that area on the back of the bolt carrier just a little, then use the abrasive rubber polishing wheel and smooth it further. You may want to touch up the spot with a little cold blue. Anyway, that is what chews up the buffers.
BTW, this is VERY common, and I have fixed it on dozens of rifles (including 3 of mine) as per above.