Re: How long to cache robots.txt for?

Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Next message: Michael Schlindwein: "Re: nastygram from xxx.lanl.gov"
Previous message: Martin Kiff: "Re: Alta Vista getting stale?"
In reply to: Jaakko Hyvatti: "Re: How long to cache robots.txt for?"
Next in thread: Greg Fenton: "Re: How long to cache robots.txt for?"

Martin Kiff (mgk@webfeet.co.uk)
Wed, 10 Jul 1996 21:13:37 +0200

> - still the cached copy should adapt to the changes by the webmaster
> reasonably fast, like once a day could be maximum I think

Can you consider /robots.txt to be a file and not the results of a
script varying with the load on the server :-)