Re: How long to cache robots.txt for?

Greg Fenton (gregf@opentext.com)
Wed, 10 Jul 1996 10:45:51 -0400

Aaron Nabil wrote:
>
> My next robot implementation is going to cache robots.txt for a fixed
> period, say 1 week. Does this sound reasonable?
>

I have come across one site where the system administrator is extremely
anti-robot/anti-web (it is an FTP site). This site does have a
/robots.txt URL. He allows a robot ONE access of this URL and then
locks the robot out of his site entirely. And then he complains
bitterly if ANY other access is made of his site...ever.

The gripe is over the caching of the /robots.txt file. My robot
received his /robots.txt file, cached it for 24 hours and then went
to get it 24 hrs later. We got slammed via e-mail. The FTP admin
insists that since he sends a "Retry-After:" header of Dec. 31, 1999
my cache should honour that date.

It involves some more code being flung into the robots, so I'm
currently just ignoring that site altogether.

How do folks feel about this Retry-After method of locking out robots
(completely, we can't even get the /robots.txt file again and
we get flamed when any attempts are made)?

gregf.

-- 
Greg Fenton, Software Engineer           http://index.opentext.net/
    || Open Text Corporation           mailto:gregf@opentext.com
    || 180 Columbia Street West         Phone:(519) 888-7111 x261
    || Waterloo, Ontario N2L 3L3          Fax:(519) 888-0677