Re: How long to cache robots.txt

Daniel T. Martin (MARTIND@carleton.edu)
Thu, 18 Jul 1996 09:36:11 -0500 (CDT)

While this has not to my knowledge affected the performance of my server, it
occurs to me that there may be a bug in the way InfoSeek's robot is caching it's
copies of robots.txt; I have robots.txt mapped to a little script that logs the
time and any other headers in a separate log file - this seems to show that in
some cases infoseek isn't caching for more than 10 minutes!

Although a daily update may not be out of order, an hourly (or much more
frequently) update seems a bit ridiculous!

Remote-Host: 204.162.98.28:44990
VMS-Connect-Time: 18-JUL-1996 03:07:39.80
From: mna@infoseek.com
User-Agent: InfoSeek Sidewinder
Date: Thu, 18 Jul 1996 08:07:33 GMT
***************
Remote-Host: 204.162.96.14:59473
VMS-Connect-Time: 18-JUL-1996 04:56:42.47
From: mna@infoseek.com
User-Agent: InfoSeek Sidewinder
Date: Thu, 18 Jul 1996 09:56:37 GMT
***************
Remote-Host: 204.162.96.14:58835
VMS-Connect-Time: 18-JUL-1996 05:09:47.77
From: mna@infoseek.com
User-Agent: InfoSeek Sidewinder
Date: Thu, 18 Jul 1996 10:09:42 GMT
***************
Remote-Host: 204.162.98.28:35919
VMS-Connect-Time: 18-JUL-1996 05:13:31.50
From: mna@infoseek.com
User-Agent: InfoSeek Sidewinder
Date: Thu, 18 Jul 1996 10:13:27 GMT
***************
Remote-Host: 204.162.98.28:47008
VMS-Connect-Time: 18-JUL-1996 05:22:32.05
From: mna@infoseek.com
User-Agent: InfoSeek Sidewinder
Date: Thu, 18 Jul 1996 10:22:27 GMT
***************
Remote-Host: 204.162.96.14:49386
VMS-Connect-Time: 18-JUL-1996 08:52:57.44
From: mna@infoseek.com
User-Agent: InfoSeek Sidewinder
Date: Thu, 18 Jul 1996 13:52:53 GMT

--Boundary (ID nHkzJ3TvJtUMTDO+vj2P/A)--