>
> Okay, I've read a lot of crap in this thread about the reaction
> of the xxx.lanl.gov admin to the clown who sent a pile of HEAD
> requests to that site and ignoring the robots.txt file there.
>
> Some of the URLs this clown accessed started long CPU intensive
> processes that involved multiple machines in xxx's e-print archive
> system. There are limited resources for this system and they were
> being wasted by this clown's irresponsible actions. This has a knock on
> effect that people using the service properly are presented with
> "system overload" messages, and their research comes to a screaching halt.
Sounds like very poor database design. I routinely search multi-tens of
megs of stuff in seconds for my Usenet archives. And my design is
*PRIMITIVE*. Scarcely more than a perl based grep on a pentium based linux
box. And he could probably prevent the *entire* problem by using FORMs to
start any CPU intensive tasks. I don't know of *ANY* robot that trys to
submit forms.
> If you don't want to be mailbombed - respect robots.txt
> If you don't want to be sued - respect robots.txt
> If you don't want to be billed for damages - respect robots.txt
Go look up the term 'denial of service attack'. The admin at xxx.lanl.gov
is playing a very stupid game that may result in *him* getting burned. The
robot owner may be negligent, but is almost certainly not legally culpable
for failing to obey a *voluntary* standard (if you think differently -
think about the potential liability of site owners who create infinite URL
spaces but *DON'T* let robots know about it with robots.txt). But the
admin at xxx.lanl.gov would not have that mitigation - his attack cannot
be construed as accidental but is clearly deliberate. And he had better
not ever try it against someone with more resources than he has. He could
find himself return mailbombed (ping stormed,tracerouted,...) in the ratio
of a T3 to a T1. Or prosecuted criminally for a deliberate denial of
service attack. Government immunity won't protect him from breaking the
law outside of official policy. I am sure that there is no authorized
policy allowing him to mount denial of service attacks.
Don't mistake me - I think that robot owners *should* respect robots.txt.
Not simply for the safety of the servers being indexed but for their own
protection again infinite URL spaces. It is in their own interest to
respect robots.txt. But the admin at xxx.lanl.gov is not behaving
reasonably. Rather than reason with robot owners or take precautions on
his end to prevent the problem (lets see, no new searches from any IP
address unless the previous search has already finished? Hmmm?) he chooses
to throw the net equivalent of a juvenile temper tantrum by perpetrating
his own net abuse in the form of mailbombing. One of these days he will
get spanked for it (either legally or via 'I've got more resources to
play the denial of service game than you do') and won't understand why.
Being an asshole on the net rarely pays. You NEED other people's goodwill.
-- Benjamin Franz "Anyone else see the irony involved in xxx.lanl.gov attacking the robots because he was too stupid to realize they could do his job _for_ him for free if he just put the documents online in a static text tree?"