Don't rely on it staying in the holster. Getting into an altercation and then staring at someone while you lift your shirt to show a holstered weapon could easily fill the requirements for a "threatening manner".
I thought about this just last night- I left my apartment to go check on some laundry in the building's laundry room and there was a guy standing outside. He could have been listening/waiting at either my door or at another door three feet away. I made eye contact, walked past him slowly and then turned to head down the stairs. As I did, I pulled my shirt forward to make my gun print even more than it already was. When I turned away from him I saw him turn his head to look back at me, so I know he saw it. Could that be called brandishing? Technically it was a threat, just a subtle one. Anyway, I saw my old lady neighbor coming up the outside steps and held the door for her while I watched him get let into the next door neighbor's apartment. So could this guy come back and say he felt threatened? I sure think so. I think the law is going to look at any move towards a weapon as brandishing, even if it's just to go OC.