If you guys had an agreement and you said something to the effect of "I'll take it" then changed your mind, then yeah you technically backed out of the deal. That's all the guy is saying.
Now, if you had a change of heart or didn't have the money or whatever and discussed it with him AND he let you off the hook, then he'd have no room to bitch. Recently one of my friends on this board had a rifle for sale. I told him I wanted it, but a couple hours later I was doing some math with the checkbook and figured I really didn't have the coin to spend. So I called him and told him I wasn't sure whether I could come up with the cash. He said no problem. Now, if he had said he expected me to go through with the deal I would have gotten the money together somehow. I didn't mess up his sale with any other potential buyers, so it was cool.
That's rare for me to tell someone I want something and then back out of it. But it's all about the communication. In my case I spoke too soon and I knew it. I should have said "I want it, but need to figure out if I have the cash." Lesson learned.
You did right by telling the guy well in advance of him making a trip to meet you. Now I could see the guy getting upset if you strung him along for a few days and maybe cost him a real sale with another member; I'd be pissed about that too.
Bottom line, if you guys had an agreement and you broke the agreement without him letting you off the hook, I think you did wrong. Doesn't mean you're a bad guy or a criminal. We still love you.