I've exposed myself to it on a few occassions, but the amount you're talking
about I wouldn't worry about.
I'd be more worried if I worked all day with it, or was a professional installer and
get exposed to the dust on a daily basis. What you got is nothing to worry about.
I worry more about all the God damned lawn chemicals, fertilizers, weed killers, etc...
that I have to handle every year.