Re: RFC, draft 1

Martijn Koster (m.koster@webcrawler.com)
Tue, 24 Dec 1996 08:28:43 -0800

At 11:32 PM 12/10/96, Hallvard B Furuseth wrote:
>> Robots need to be aware that the amount of resources spent on dealing
>> with the /robots.txt is a function of the file contents, which is not
>> under the control of the robot. To prevent denial-of-service attacks,
>> robots are therefore encouraged to place limits on the resources
>> spent on processing of /robots.txt.
>
>Something must be said about these limits.
>
>- Some minimum which one should expect the robot to handle. (I would
> say "MUST handle", but of course robots do as they please.)
>
>- What should the robot do when it reaches a limit?

Agreed.

In case you're wondering why I haven't progressed on the Draft,
it turns out that draft-url-syntax-fielding is going to change
the BNF, and some of the wording, on URL's (wonderful :-/).
Seeing as I use both, I'm waiting for that to stabilise before
updating again....

> A related point: it might be useful to allow robots to tell www sites
> that they did not like their robots.txt. E.g.
> Errors-To: /cgi-bin/robot-message

Interesting idea...

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html