There are some who tend to disagree. I don't see why RES would be a
waste of time, or a "strain", as you call it later. I see it as a
guidance mechanism.
> The way I see it, robots.txt is a convenience for the robot, not a method to
> restrict its access. Speaking as a user, I really don't care what burden I place on
> a distant server as long as I acquire the information I want. Even if "Obey Robot
> Exclusion" was an option on the agent, I wouldn't use it. And why should I? What
> is the penalty for not obeying? Perhaps this list needs to refocus on
The penalty is that the server admin can (and will) shut down requests
from your clients completely, complain to your ISP and do all kinds of
nasty stuff.
Don't mess around with people just because you think "HipCrime" is
cool.
> Perhaps I need to reemphasize the point that most programmers/users don't care about
> the strain placed on webservers, nor do they care about making life easier for a
> sysadmin. If I bring your server to its knees, I really don't care -- you should
There are some of us that do care. There are some of us living very
close to angry sysasmins. :-)
-- Hrvoje Niksic <hniksic@srce.hr> | Hocemo 101-icu! --------------------------------+-------------------------------- * Q: What is an experienced Emacs user? * A: A person who wishes that the terminal had pedals. _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html