Re: make people use ROBOTS.txt?

Hrvoje Niksic (hniksic@srce.hr)
25 Nov 1996 03:43:32 +0100

Terry Coatta (coatta@opentext.com) wrote:
> To me, this means two things:
> (1) Web site administrators need tools which allow them to block access =
> to clients whom they consider bothersome. These clients may be browsers, =
[...]
> (2) The robots.txt protocol should be designed with the assumption that =
> the people using it are *a priori* interested in having their software =
> "behave".
[...]

So basically, you say that we should talk about two different things.
The first would be a tool for bad guys (easy-to-use server-level
restrictions), and the second for the good guys (something like
/robots.txt). I find such a view quite agreeable.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Hocemo 101-icu!
--------------------------------+--------------------------------
"Psychos _do not_ explode when sunlight hits them."
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html