robots.txt changes how often?

Tangy Verdell (TVerdell@dca.com)
Wed, 14 Feb 96 09:32:58 EST

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Nick Arnett: "Re: Money Spider WWW Robot for Windows"
Previous message: John McGrath - Money Spider Ltd.: "Money Spider WWW Robot for Windows"
Next in thread: Darrin Chandler: "Re: robots.txt changes how often?"
Maybe reply: Darrin Chandler: "Re: robots.txt changes how often?"
Reply: Jaakko Hyvatti: "Re: robots.txt changes how often?"
Maybe reply: Martin Kiff: "Re: robots.txt changes how often?"

in my robot, i don't want to check the robots.txt everytime i visit
that host because i think that robots.txt doesn't change very often -
ususally.

is this true?

Next message: Nick Arnett: "Re: Money Spider WWW Robot for Windows"
Previous message: John McGrath - Money Spider Ltd.: "Money Spider WWW Robot for Windows"
Next in thread: Darrin Chandler: "Re: robots.txt changes how often?"
Maybe reply: Darrin Chandler: "Re: robots.txt changes how often?"
Reply: Jaakko Hyvatti: "Re: robots.txt changes how often?"
Maybe reply: Martin Kiff: "Re: robots.txt changes how often?"