Re: Broadness of Robots.txt (Re: Washington again !!!)

Captain Napalm (spc@armigeron.com)
Wed, 20 Nov 1996 18:41:19 -0500 (EST)

It was thus said that the Great Erik Selberg once stated:
>
> [paraphrase of mail to Denis McKeon directly; thought I was sending to the
> list]
>
> One other thing which may / may not be part of robots.txt is a concept
> of an access rate. One of the reasons behind robots.txt is to prevent
> robots from overwhelming a site. One idea Denis mentioned:
>
This has already been proposed. Check

http://www.armigeron.com/people/spc/robots2.html

for details on some suggested additions.

-spc (Actually, couldn't this also be handled with .htaccess?)

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html