>> Now, the /robots.txt won't do a bit of good here. Why? Because (a)
>> robots don't have to support the robots.txt file, and (b) because the
>> goal is to keep said data _private_ from everyone, not just
>> robots. The problem is that users feel that hiding data is a good
>> solution to security. Robots just publicly announce that security of
>> that form is bogus. The issue people have with robots I think is
>> bogus; what they should be addressing is that there needs to be a
>> better form of protection on the Web, or at least a more intuitive
>> method of setting access control lists than the funky .htaccess file
>> stuff (or at least a better UI!).
> What if you're using ROBOTS.TXT to exclude CGI's which don't appear in /cgi-bin?
> What if the CGI's--or any data types unknown to the robot--are indistinguishable from
> directory pathnames or acceptable data types except if (a) it is excluded with
> something like ROBOTS.TXT or (b) the robot spends considerable time and resources to
> analyze it?
I'm not arguing here about the usefullness of robots.txt to exclude
things that shouldn't be searched, i.e. to tell the robot of useless
things, etc. For example, I recently discovered that many robots, like
Lycos and Inktomi, have played many a game of "Hunt the Wumpus"
because the links in the game are URLs to cgi-bin scripts. However,
robots.txt should not be something used to enforce security.