>Rather than reproducing the discussion, I'd like to suggest that we should
>focus on the specific proposal that was developed by people from Lycos,
>Excite and InfoSeek. This is also where Verity is headed unless good
>arguments are made for a different approach.
>
>http://www.w3.org/pub/WWW/Search/9605-Indexing-Workshop/ReportOutcomes/Spide
>ring.txt
before anyone focuses on this, please will robot owners agree a string
to add to the USER_AGENT so that server admins can do something with it.
This should be a quick and painless task; it just need doing.
Some suggestions for strings to use..
(robot)
(automated)
(follows-robots.txt)
(server-friendly)
I'm no expert on the syntax.. reformat as appropriate, pick your favorite
and let us know. Thanks.
regards,
rob
--
Rob Hartill (robh@imdb.com)
The Internet Movie Database (IMDb) http://www.imdb.com/
...more movie info than you can poke a stick at.