Re: The Internet Archive robot

Gareth R White (Gareth_R_White@sbphrd.com)
23 Sep 96 9:53:55 EDT

I was thinking about this recently, perhaps a new SRE directive would help, to
indicate at what time of the day / week / month etc a robot can access the
site, and how many pages / bytes they can retrieve per access.

To: robots @ webcrawler.com @ INET
cc: robots
(bcc: Gareth R White)
From: jsigmon @ www.hsc.wvu.edu (Jeremy Sigmon) @ INET
Date: 20-Sep-96 16:04:24
Subject: Re: The Internet Archive robot

> Ahh, but when I tape a show it doesn't have the possibility of adversely
affecting my neighbor's picture quality. ;-)
> Cheers,
> Eric

Very good point. If you grabbed 100 or so pages from a site during its
slow period then it wouldn't be so bad. But you would need the site to
provide connection statistics to give you that.

Maybe you should be able to request a "load level" from the server
and if it is low enough then grab the pages.