Just for the record, I did not write the software in question. I agree
strongly that robots should adhere to the REP. In fact, I was very much
in sympathy with the complaints of xxx.lanl.gov until I visited that site.
Chris Cowther wrote:
>
> > 15 year old kids looking for Pam Anderson gifs don't seem to be causing
> > trouble at that site.
>
> Unless they're using a Robot to do a traverse and find all pages
> with the words "Pam" or "Pammela" and "Anderson". :)
Or if they're looking for words such as "porn", "erotic", "bondage", etc.
Yes, xxx.lanl.gov is justified in complaining about robots ignoring
robots.txt. Still, I find it awfully hard to sympathize with their
skull-and-crossbones take-no-prisoners approach, and I find their
complaints a little hard to take seriously when they have embedded all
those provocative words in an HTML comment that are guaranteed to trigger
hits.
Also. remember that this is a US government site, using our tax money
to threaten US citizens (and non-citizens) with obliterating reprisals.
It seems to me that this thread has demonstrated that there are both
robot writers and website administrators who refuse to act responsibly
and who cloak this refusal in some very self-righteous talk about rights
and abuse. (yeah, this could generate flames from BOTH sides!).
-Gordon Bainbridge