I've never considered it haughty for either medical docs or PhD's to use the title "Dr." rather than Mr. (or Mrs.) outside of work settings. Both groups of people put in a lot of work to get those, so why not use it?
Although I get ticked when people who have them make a big deal if you don't use it. Not like I can tell you got your PhD/MD unless you carry it around your neck, damnit. Lighten up.
The real question for me, should I as a TA be slightly annoyed when some of my students insist on calling me "Doctor" (always the same few who do it over and over again). It's like, damnit kids, just because I'm standing in front of you doesn't mean I have a PhD. I'm just the TA. Some of them seem a little irked when I correct them, but I haven't earned the title. Constantly correcting the same few kids though, one day I'm just gonna say "Yes, I am a doctor, but you must call me Professor!" and laugh manically.
Hey, here's a rather interesting article on the usage:
en.wikipedia.org/wiki/Doctor_%28title%29. Apparently some doctors in the UK prefer "Mister" as a historical nod.