Re: The Internet Archive robot

Martijn Koster (m.koster@webcrawler.com)
Mon, 23 Sep 1996 08:51:01 -0700


At 10:53 AM 9/23/96, Jeremy Sigmon wrote:

>For now a few cron jobs could switch between a few robots.txt files during
>appropriate times of the day.

I don't know how often you expect robots to update their cached version
of /robots.txt, but often used values are a day or a week. So the above
won't work...

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html