Since it sounds like they guy wants to bend you over and ram it in, i would tell him he has two choices. Reduce the agreed upon price in half, or give you 90-95% of the money he got from the sale of the trees. Anything else, I would threaten to sue him with misrepresentation of property for sale. Do you have a buyers' realtor? the sellers realtor has no obligation to look out for your interests. get a buyers realtor, they (both realtos) will split the comimision, and he well will represent you in the transaction.
Edit- I would sue him if you have ernest money down and an agreement for the condition of the property, minus the 3-4 trees. I would get all the money he gets from the sale of the trees, plus the legal fees you incur, plus whatever is a good amount representing your time wasted, loss of emotional attachment to house, and a significant punishment to this shyster!
Make him pay!
ps, what was the acres/ house size and sale price? If you don't mind me asking.