Re: Suggestion to help robots and sites coexist a little better
Nick Arnett (narnett@verity.com)
Wed, 17 Jul 1996 09:25:49 -0700
>Why not propose a form of a robots.txt or some other file that can
>include information for robots at the root of each directory? Such a file
>could be put in user directories and include the necessary information on
>each resource available thus preventing the need for the robot to download
>all the information therein for indexing. A simpler approach might even
>just include instructions to allow/prevent the robot to access specific
>pages in that tree.
Rather than reproducing the discussion, I'd like to suggest that we should
focus on the specific proposal that was developed by people from Lycos,
Excite and InfoSeek. This is also where Verity is headed unless good
arguments are made for a different approach.
http://www.w3.org/pub/WWW/Search/9605-Indexing-Workshop/ReportOutcomes/Spide
ring.txt
Nick