Re: RFC, draft 1

Klaus Johannes Rusch (e8726057@student.tuwien.ac.at)
Thu, 21 Nov 1996 22:50:58 CET

In <v02140b0baeba5bb635bd@[192.216.46.39]>, m.koster@webcrawler.com (Martijn Koster) writes:
> The spec requires User-agent precedes all other lines,
> so the above is invalid, and it's semantics not defined.
>
> "lines inbetween" are not allowed, as they separate records;
> The above record matches webcrawler, has no Allow/Disallow,
> so WebCralwer is allowed to go anywhere.

Martijn,

wouldn't it be reasonable to require robots being liberal in what they
understand? I guess there are quite a number of sites out there with a
robots.txt having blank lines and comment lines everywhere, and the robots
probably handle most of them fine.

Or, even generally allow comments and blank lines anywhere?

Anyway, my original question got lost between syntactic problems, so
here it is again:

Is

User-agent: infoseek
User-agent: webcrawler
Disallow: /tmp

equivalent to

User-agent: infoseek
Disallow: /tmp

User-agent: webcrawler
Disallow: /tmp

Klaus Johannes Rusch

--
e8726057@student.tuwien.ac.at, KlausRusch@atmedia.net
http://www.atmedia.net/KlausRusch/
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html