I go and talk with everyone that is on or crossing my ramp, whether they have a badge or not. I have a lot of traffic at my fbo some are transients some are local to the area stopping at our restaurant for breakfast or a burger. As an owner would you feel the fbo is responsible for your plane on their ramp? I bet you would. And you would probably be pissed if some nose picker that didn't belong was sniffing around your plane. Yall probably shouldn't be snippy with someone who is just trying to keep control of access doors to people's assets. your plane is out there but who else's is? And unbadged personnel at airports with security are supposed to be escorted, guess who will be in the hot seat for that, probably not you when you point to the door you accessed the ramp from.
Edit
yes private owners and operators are badged, it is in the airport operations manual, part 139 I believe