Yup, that's exactly how I did it. Just take your time and go slow.
The CZ grip panels only hold on with one screw in an odd spot really. So don't torque on the screws as hard as you would if you had wooden/plastic panels.
Once they're on there though, they aren't going anywhere. It's like a perfect fit, just a hair thicker than flat wood/plastic panels, and they're the perfect texture. Just tacky enough, but you can still shift your grip with the front panel gone.
Ross