Re: The Internet Archive robot

Jeremy Sigmon (jsigmon@www.hsc.wvu.edu)
Tue, 24 Sep 1996 09:29:59 -0400 (EDT)

>
> On Mon, 23 Sep 1996, Jeremy Sigmon wrote:
>
> > For now a few cron jobs could switch between a few robots.txt files during
> > appropriate times of the day.
>
> How often is a robots.txt supposed to be checked? I generally only check
> it at the first request of the day.
>

Sorry, I wasn't thinking ahead much. The little robotlings I have played
with didn't run continuously and requested it each time. So it worked
for me.

======================================================================
Jeremy Sigmon B.S. ChE |
Web Developer of the Robert C. Byrd Health | Use
Sciences Center of West Virginia University | FreeBSD
WWW.HSC.WVU.EDU | Now
Graduate Student in Computer Science |
Office : 293-1060 |