> in my robot, i don't want to check the robots.txt everytime i visit
> that host because i think that robots.txt doesn't change very often -
> ususally.
I can imagine the /robots.txt _files_ don't change very frequently but
what would happen if you met a script?
I'm wondering whether a script would/could deflect robot activity towards
the 'quieter' hours on the server. I was sure that there was a 'server too
busy' error code, one of the '500's, but I can only see a '503 Service
unavailable' in the draft spec I have....
What would robots do if presented with that when reading /robots.txt, back
off and come back later? back off completely? ignore and carry on searching?
Regards,
Martin Kiff
National Physical Laboratory, UK
http://www.npl.co.uk/