> and `Allow:'. We need official WWW standards to refer to the Robot
> Exclusion Standard so that new robot writers will find it, and that
> means it must be "official" somehow. So let's get adopted by IETF or W3
> Consortium and go through their procedures for evolving and adopting
I agree with you .... I found out that not too many people know about the
role of robots.txt (they have actually never heard of it), and when they
write their wanderers they just forgot about some of the issues involved
in writing robots. A concrete example is the WebExplorer article posted
on JavaWorld (http://www.javaworld.com) in the November issue. That
program failed to parse robots.txt, I e-mailed the author and he said
that he never heard of robots.txt. My point is that not too many people
(especially those who administer Web servers and write search engines) know
about the guidelines for writing robots...etc.
I am not sure what is the best way to spread the word!
BTW, WebExplorer is written in Java, and it just access documents on the
same server SO FAST - something WebMasters would hate! :)
Regards,
Qusay H. Mahmoud