Re: make people use ROBOTS.txt?

Kim Davies (kim@staff.iinet.net.au)
Sun, 24 Nov 1996 07:00:41 +0800


Robert..

Quoting HipCrime:
| Hi Kim ...
|
| People ignore signs all the time. Fences and locks are what
| the law in every country recognizes as a way to control access.

I'm not entirely sure what country you live in, but "their door
was unlocked and ajar" isn't a defense when you go into someones
house and smash it up with a baseball bat, at least where I live.

| The InterNET already has such things. They're called file access
| priviledges, passwords, and firewalls. Let's not reinvent the
| wheel. If you have information that you don't want to make
| public, don't put it in a file with universal read access on
| the WORLD wide web.

It's not a case of not making it "public", it's a case of not making
it available to your rapidfire denial-of-service type attacks.

A street has a speed limit. The fact you can get on it means
you can go whatever speed you like when you're HipCrime!
I'm sure you would argue that police should patrol every street
to enforce this otherwise it's just not a speed limit.
Perhaps set up a booth at every street? You can't get through
without the password?

As with sites such as xxx.lanl.gov, I run a site which generates
CPU-intensive documents on the fly. Getting multiple requests from
your robots could quite easily kill the machine. How are you to know
that it is simply okay for you to bring this server to it's knees?
Surely the administrator who set up robots.txt has a better clue
as to whats bad and whats not that you?

On Friday, I had someone spam every one of our mailing lists with
forged approved headers, resulting in many tens of thousands of
sendmails spawning up in a couple of minutes - bringing this
mail server to a halt and spreading 2 copies of this spam to
every single one of our clients.

If it were Activeagent who did it by disobeying our robots.txt what
is your defense?

"Mailing lists where designed to be spammed!"
"Mail servers are designed for denial of service attacks!"
"I own the Internet, I can do what I damn well like!"
"People offer stuff on the Internet out of their good will,
hence I have a right to terrorise them!"

| Displaying information on the WWWeb is not the way to keep
| it private. In fact, displaying info on the web is the way
| to make it PUBLIC, but then, you probably knew that.

A footbridge is pretty public. It may have signs saying "no cars",
or "pedestrians only".

Of course, for you, I'm sure you would consider it an ideal
opportunity to drive your car over it, running over the innocent
pedestrians using it for it's intended purpose, and testing out
how quickly you can break it.

kim

--
% kim davies <kim@iinet.net.au> tel: (09) 322 7770 fax: (09) 322 6660
  iiNet Technologies, 105 St Georges Terrace, Perth WA 6000
  url = http://www.iinet.net.au/~kim
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html