>What I am proposing is that we re-evaluate the reasoning behind
>robots.txt. The proposals I have seen in this list seem to rely on the
>assumption that robots.txt is enforcible when it is quite clearly not.
Perhaps the important thing is to have a recognised standard for a place
where the wishes of the site owner are expressed. What areas are out-of-
bounds, maximum hit rates, restrictions on use of links collected from
the pages etc.
This is the longer term view - if there is not a well defined standard
in this area then there will be a free-for-all, if there is then perhaps
pressure can be brought to bear on people who abuse the services.
I would say don't water down /robots.txt because it is not enforceable
at the moment.
(My 'hipcrime' analogy is that it is equivalent to a driver who does not
dip his headlights when driving towards you. Annoys a lot of people, may
easily be dangerous, but is done so the he can see that little bit
better or drive that little bit faster. There may or may not be a law
stating that you have to dip but there is a code - you don't do it
because you don't want it done to you)
Regards,
Martin Kiff
mgk@webfeet.co.uk
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html