> It is not an official standard backed by a standards body, or owned by
> any commercial organisation. It is not enforced by anybody, and there
> no guarantee that all current and future robots will use it. Consider
> it a common facility the majority of robot authors offer the WWW
> community to protect WWW server against unwanted accesses by their
> robots.
I believe this is a mistake, and even more important to fix than regexps
and `Allow:'. We need official WWW standards to refer to the Robot
Exclusion Standard so that new robot writers will find it, and that
means it must be "official" somehow. So let's get adopted by IETF or W3
Consortium and go through their procedures for evolving and adopting
standards. If we wait until we are satisfied with version 2, it might
remain a good, well thought of, but half-secret standard, until we have
slowly dragged it through the bureaucracy.
Or if I'm outdated and this is already happeing, say so clearly at the
top of norobots.txt. Currently it contains no indication that it
is/will be anything more than an "unofficial standard".
Regards,
Hallvard