Re: robots.txt changes how often?

Darrin Chandler (dchandler@abilnet.com)
Wed, 14 Feb 1996 12:14:16 -0700

At 09:32 2/14/96 EST, you wrote:
> in my robot, i don't want to check the robots.txt everytime i visit
> that host because i think that robots.txt doesn't change very often -
> ususally.
>
> is this true?
>

Depends on what you mean by "visit". Checking once per day would be very
little overhead, but would make your program fairly responsive to changes.
Also, use the "If-modified-since" request header so that you'll get a 304
result if robots.txt hasn't changed.

Darrin
______________________________________________

_/| _| _| _|
_/_| _| _| _| _| _|_|_|
_/ _| _| _| _|
_/_|_| _|_|_| _| _| _| _| _| _|
_/ _| _| _| _| _| _| _| _| _|
_/ _| _|_|_| _| _| _| _|_| _|_|_|
_|
_|_|_|

Darrin Chandler, Duke of URL
Ability Software & Productions
Email: dchandler@abilnet.com
WWW: http://www.abilnet.com/
______________________________________________