Re: How frequently should I check /robots.txt?

Martijn Koster (mak@surfski.webcrawler.com)
Mon, 27 Nov 1995 23:22:54 -0800


In message <Pine.LNX.3.91.951127231751.1718A-100000@rusty.waterworks.com>, gil
cosson writes:

> How about adding an entry to the robots.txt file that specifies how
> frequently the robots.txt file should be checked?

Hmm.. and then how often do you check if the checking frequency has
changed? :-)

Seriously though I don't think there'd be a lot of benefit; as an admin
you tend not to know when you'll make the next change.