An explanation:
Infoseek's spider caches the robots.txt for the duration of a
particular process. Normally, a process will process several thousand
urls and then exit. A single site's urls are normally contrained to be
running in a single process. However, if, as it appears to be in this
case, a particular site spans multiple processes we will ask for the
robots.txt more frequently.
This implementation is fairly conservative about caching of
robots.txt. We developed this policy after receiving complaints from
both ends of the spectrum regarding caching of robots.txt. Some
webmasters believed it should always, always, always be retrieved
before *every* access because they want their changes to robots.txt to
be seen instantly. The other extreme want spider writers to cache it
for up to a month. IMHO, neither extreme is correct. Our policy may
not be the best technical solution but as a "service company" we are
often obliged to take middle of the road solutions that hopefully will
bother the least number of people.
Mike
====================================================================
Mike Agostino mna@infoseek.com
InfoSeek Corporation
VP, Pizza Configuration
"I feel myself getting the urge to build an igloo" -- The Dead Milkmen