> Terry Coatta (coatta@opentext.com) wrote:
> > To me, this means two things:
> > (1) Web site administrators need tools which allow them to block access =
> > to clients whom they consider bothersome. These clients may be browsers, =
> [...]
> > (2) The robots.txt protocol should be designed with the assumption that =
> > the people using it are *a priori* interested in having their software =
> > "behave".
> [...]
>
> So basically, you say that we should talk about two different things.
> The first would be a tool for bad guys (easy-to-use server-level
> restrictions), and the second for the good guys (something like
> /robots.txt). I find such a view quite agreeable.
So it seems to me what we should be treating /robots.txt not as a
security protocol (i.e. robots aren't allowed here) but as a
"guide" protocol, (i.e. good stuff you want is here).
So, we should probably let the sysadmins talk about how to better
guide robots to places they require, or methods of how to give robots
info they want. I'll make a motion that we move the "how to get rid of
bad robots and prevent rapid-fire attacks" off to another list;
perhaps the c.i.w.servers.* lists? :)
-Erik
-- Erik Selberg "I get by with a little help selberg@cs.washington.edu from my friends." http://www.cs.washington.edu/homes/selberg _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html