in your experience does leather or kydex cause more wear on the finish of a gun?

Kydex

Define "more"

Kydex will cause fast wear on the high spots then stop.

Leather will cause wear slowly over the entire surface contact area over time.

The maker is a variable too, some kydex holsters made with undersized drones (blue guns) or improper blocking/channeling or adjustable retention will cause wear atypical wear.  Same for leather.

If you don't want wear on your firearm then your best bet is to leave it in the safe.

I've had more/bigger area wear from leather.

The amount of dirt in the holster has a lot to do with it.