> (My 'hipcrime' analogy is that it is equivalent to a driver who does not
> dip his headlights when driving towards you. Annoys a lot of people, may
> easily be dangerous, but is done so the he can see that little bit
> better or drive that little bit faster. There may or may not be a law
> stating that you have to dip but there is a code - you don't do it
> because you don't want it done to you)
My analogy is that the robots.txt standard is like USENET
FAQs. They're great for people who know about them and abide by
netiquette to look up answers there first, but for the throngs of
people out there, they still all ask the same questions over and over
and over and over....
The simple truth is that you _need_ some method of enforcement of your
robots.txt for people who are ignorant. You need to ensure folks are
aware of it, and you need a strong incentive for people to use
it. Right now, the only real incentives out there are that if you
don't follow it, Paul and Rob will send you hate mail after the
fact. If the robot author is able to obtain what he wants, then it
seems perfectly reasonable to me that folks will choose not to support
the standard (assuming they even care to look to see if there is one).
-Erik
-- Erik Selberg "I get by with a little help selberg@cs.washington.edu from my friends." http://www.cs.washington.edu/homes/selberg _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html