> 15 year old kids looking for Pam Anderson gifs don't seem to be causing
> trouble at that site.
Unless they're using a Robot to do a traverse and find all pages
with the words "Pam" or "Pammela" and "Anderson". :)
Chris,
chris@jm-crowther.co.uk
www.dungeon.com/~jmcrowther/chris.html
ChegHchu djajVam djajKak!
a bit daft.
> The gripe is over the caching of the /robots.txt file. My robot
> received his /robots.txt file, cached it for 24 hours and then went
> to get it 24 hrs later. We got slammed via e-mail. The FTP admin
> insists that since he sends a "Retry-After:" header of Dec. 31, 1999
> my cache should honour that date.
Well, as long as your robot was programed to do it the first place,
it no great hardship. If someone says robots shouldn't come back
their site until xxxxx then you shouldn't. Just so yuo dont get
flamed to death :)
> It involves some more code being flung into the robots, so I'm
> currently just ignoring that site altogether.
Put the code in on the next revision?
> How do folks feel about this Retry-After method of locking out robots
> (completely, we can't even get the /robots.txt file again and
> we get flamed when any attempts are made)?
Like I said - I think blocking access to the robots.txt file is just
a little daft.
wanabe robot programer - don't shout at me......to much <g>
Rudewind-Rustling B.F.,B.Am.Ta.
Life is like the Ankh - only you can't walk over it....
--------------------------------------------------------
rudewind.rustling@jm-crowther.co.uk
www.dungeon.com/~jmcrowther/chris.html