The sliding weights are there to provide a dead-blow effect, which acts to negate bolt carrier bounce, which is essential in full-auto fire. If you are shooting semi-auto only, it shouldn't matter for a hill of beans.
To me, it makes sense that you could get a smoother action with a lighter buffer, as it means you have less mass cycling back and crashing into the back of the receiver extension. Ever wonder why an HK 91 kicks so hard? It's because of that massive bolt carrier, slamming into the back of the receiver and giving you two recoil jolts for every shot fired.
Mind you, it will likely vary somewhat with the ammo you use. Stuff that gives a high port pressure might accelerate that lightweight recoil assembly so fast it throws it into the back of the extension much faster than it should.
Recall that the original Colt AR-15 rifles did not have buffers in them, using instead the very light Edgewater spring guide. Until ball powder came along, with its high port pressures, it wasn't an issue. I say if it works, and you like it, there is no reason not to run your buffer without weights in a semi-auto.