robots.txt: allow directive
Leslie Cuff (lez@fastfwd.com)
Fri, 14 Jun 1996 08:39:59 -0230
I'm new to this discussion, and I added it because I wanted to lobby shamelessly
for an extention to the robots.txt configuration directives: I don't want
to export the names of the directories that I want to remain hidden.
I think there should be an (optional) deny: all
allow: directory
allow: directory2
directive.
That way, some yobbo can't figure out what I'm paranoid about simply by
reading a plain text file (necessarily) world readable off my root document
structure.
les cuff
nf.ca