Almost hating to put any more fuel on this fire, but I just have to ask a
simple pratical, not philosophical, question wrt 2) below. I know its a minor
pain to have to do it, but doesn't blocking out the IP of the offending robot
at a firewall take care of shutting down offending robots? It seems to work
for us.
________________Ron Wolf, Netsource, ron.w.wolf@tsphere.com
There are basically 2 opinions here:
1) robot owners should follow guidelines and respect site ownsers choice
of what should be hit automatically and what shouldn't.
2) robot owners can do whatever they damn well feel like and it's the
job of a site owner to build a fortress of safety systems to prevent
the robots from causing damage or denial of service.
I can't believe there are so many people here who think (2) is
reasonable. I say to those people, go build me this fortress so
that I can forget about your stupid robots.
Dumb robots are a big problem. It's not just 'xxx.lanl.gov'.. all the
robots that visited that site also rampaged through other sites.. it
just so happens that early warning systems there set off the alarm bells.
Few site admins check their logs for signs of attack, so this problem
goes undetected... they have their robots.txt in place but still get
hit without realising it.
If I hadn't run "top" the other day I wouldn't have noticed the robot
that was firing off 20 requests a second on my site causing a denial
of service there. The people who think (2) is reasonable should tell
me now, how the hell I protect my site against such a mind-numbingly
stupid attack from an unattended robot. Is it at all reasonable that
I had to call the company's Internet providers to ask them to trace the
owners to get them to stop?
So what magical bit of software do supporters of (2) think I should
have been running?, and how would it have prevented this dumb robot
from sending 20 requests per second? even though the robot eventually
ignored 'forbidden' messages too.
Look, if you have a robot, follow the rules and switch the damn thing
off if it is causing trouble. Don't pass the buck to server admins and blame
them for not outsmarting your buggy software.
rob
------------------ RFC822 Header Follows ------------------
Received: by tsphere.com with SMTP;11 Jul 1996 00:24:43 -0800
Received: from webcrawler.com (surfski.webcrawler.com) by tsphere1.tsphere.com
(5.x/SMI-SVR4)
id AA12746; Thu, 11 Jul 1996 00:22:30 -0700
Received: by webcrawler.com (NX5.67f2/NX3.0M)
id AA29970; Wed, 10 Jul 96 21:13:36 -0700
Message-Id: <199607110414.WAA28180>
Subject: dumb robots and xxx
To: robots@webcrawler.com
Date: Wed, 10 Jul 1996 22:14:16 -0600 (MDT)
From: robh@imdb.com (Rob Hartill)
Organization: Internet Movie Database
X-Mailer: ELM [version 2.4 PL24 ME8a]
Content-Type: text
Sender: owner-robots@webcrawler.com
Precedence: bulk
Reply-To: robots@webcrawler.com