Looking into a laser is something that should never be done and could damage the eye. Having said that, lasers are generally divided into 3 classes. Class III lasers are 5 milliwatts or less and are the consumer lasers that are generally considered eyesafe, such as pointers and gunsights. Something to remember is that when you look at a laser through optical magnification, you increase the effective ability of the laser to damage your eye by the power of the optics. A laser rangefinder that is considered eye dangerous to 100 yards has to be considered dangerous to 1000 yard if viewed through a 10X riflescope for example.
The closer the laser and the longer you view it are both risk factors. In other words, don't take a pointer and look into it and turn it on.
Under those circumstances you described, there was realisticly no risk unless the laser was higher powered, something you don't want to take even a slim chance with. I have seen, for example, laser tubes removed from tank rangefinders and sold in electronics hobby shops. Or, if it was industrial surplus, etc.
Depending on local laws, this could be considered an assault and could be taken very seriously, the idea being you cannot know whether it is a weapon sight.