Caution is always in order, but actually it's not just inappropriate what the owner did, it IS illegal and breach of contract DEPENDING on what the lease sets out.
If you rent a single-family home, can the owner of it just come in whenever they want? The short answer is no.
When you rent a property, as in a lease for 15 acres and a home, you have a leasehold interest. You are entitled to full enjoyment of the property, you are for all intents the owner for the duration of your lease. That's basic property law.
So, yes, caution is always in order, because you don't want to anger your landlord unnecessarily, but assuming I'm right that he hasn't restricted the use of the property through the lease and leased the entirety of the property, it would be breach of contract for him to restrict the property in this way arbitrarily after the terms of the lease have been agreed to.
What is the remedy for his breach? Not a whole lot probably, but that's a different story. I suspect the worst case is a pissed-off landlord, since he'd have no remedy to evict you...
(Again, armchair analysis not having seen the lease...)