Scooter, the robot behind Alta Vista, follows links, and only follows links.
If the "directory browsing" option is enabled on a server, and someone publishes
the URL for a directory, then the robots gets back a page of HTML which lists
every file as a link, but that is not intentional. And yes, this has led to
embarrassing situations, but again, it's not intentional.
In the absence of strong conventions about directory names or file extensions
it is hard for a robot to exclude anything a-priori. I wish it was easier...
To keep a document private, list it in /robots.txt, password-protect it, change
the protection on the file, or simpler: do not leave it in your Web hierarchy.
Can you imagine what happens when someone uses / as web root, exposing for
example the password file? It has happened!
Remember that what a robot does, anyone with a browser can do: find this private
file and then post to usenet for example, robots have no magic powers!
The bottom line is that the usual danger is not aggressive robots, but
clueless Web masters.
--Louis