If you hunt out of a blind, wear black. The inside of most blinds are black for a reason - to cut out all light. You don't want to be a light blob in the blind.
Make sure the back of the blind is COMPLETELY closed. This is more important than you could imagine. Deer have an extremely sharp sense for movement, and will pick your moving silhouette out EVERY time.
As stated above, you have to make sure the deer is looking away, or has it's head down or behind a tree or bush. If they catch movement, they will do exactly as you have described and try to pinpoint you. She may not have known you were a threat, but she definitely caught movement, and had all the time in the world to try and confirm what it was she saw.
At this point, you're very lucky if they calm down and go about their routine again. If you continued your draw, you would've confirmed what she saw and she'd have bolted. As you can imagine, this isn't good for future prospective shots from that same blind location.