Is there an easy way I can remove the forward swing swivel? Looks like a rivet, but maybe there's some simple way to handle this.

Yes. All you need is a 1/8" (I think) punch and a hammer. Notice that one side of the rivet is dimpled...simply place the punch tip in the middle of the dimple and tap with the hammer until it comes out. The rivet is cheap so don't worry if you damage it. Also, you may want to tape around the front sight housing and barrel to avoid marring the finish if you slip with either the punch or the hammer

I used a rubber mallet (In case I missed the punch).  Took three good wacks and it was out.  If I can do it, anyone can [:D]

Thanks guys... got it out no problem using a rubber mallet.