I started prepping our guest bathroom for paint today.

I started by peeling off some loose caulking at the base of the tub, because someone at some point in time did not take the packaging tape of the shower/bathtub combo. So after who knows how long, the tape looses it's stickiness and causes the caulk to weaken. 

My problem is, is that when I started peeling off the heat old caulk, the top 2 coats of paint peeled off really easy. Like it wasn't even grabbing the wall. It only does this for so far, and then it's strong to the wall.

So far, I can tell the first coat of paint is red, then white, and the top layer being bright fucking lime green. The white paint is the layer that is peeling off with the green stuck on top of it. I do not believe the white layer is a primer, as that is the color behind the toilet  (which I have removed for replacement pending a plumber coming to take a look at my flange).

So I'm not sure what is causing the problem. 

How can I fix this without sanding the wall of all paint? 

Before all this occurred, my plan was to Kilz the shit out of the lime green. Should this be the route I go and blend in the missing layers of paint and retexuring the wall to blend?

Spakle or joint compound, re-texture, prime then paint. I hate textured walls, we don't do it here but down south I guess it's pretty common. It just makes it harder to blend a repair and not notice.

It's orange peel, so it's a bit easier to blend in vs something like knock down.

I've already spackled around the old thermostat, sanded and retextured the wall for the Nest thermostat. It looks better without the cover plate the hide everything underneath the old thermostat and 20 years worth of built-up paint layers.

I've had this happen.

I scraped up as much as would come off, which was a lot.  It came off easily and in large sheets.

I sanded what was left to blend the edges and prep the surface that failed (break the shine).  I then skim coated and sanded the wall to smooth the edges.  That was coated with primer and later, a good quality exterior paint.  Actually, I used exterior paint for the walls and the trim in all my bathrooms because of the moisture they endure.