While I think that the /robots.txt is very nice, I don't think it's a
worthwhile, or even workable, solution to the Idiot's Security
Problem.
The Idiot's Security Problem: this is when an idiot I puts some
private data P on the Web but attempts to keep them private, by
having either a subtle link somewhere or none at all. Later, a robot R
finds data P and puts it in some database D.
Now, the /robots.txt won't do a bit of good here. Why? Because (a)
robots don't have to support the robots.txt file, and (b) because the
goal is to keep said data _private_ from everyone, not just
robots. The problem is that users feel that hiding data is a good
solution to security. Robots just publicly announce that security of
that form is bogus. The issue people have with robots I think is
bogus; what they should be addressing is that there needs to be a
better form of protection on the Web, or at least a more intuitive
method of setting access control lists than the funky .htaccess file
stuff (or at least a better UI!).
-Erik