Re: RFC, draft 1

Martijn Koster (m.koster@webcrawler.com)
Fri, 22 Nov 1996 08:14:32 -0800


At 2:50 PM 11/21/96, Klaus Johannes Rusch wrote:

>wouldn't it be reasonable to require robots being liberal in what they
>understand? I guess there are quite a number of sites out there with a
>robots.txt having blank lines and comment lines everywhere,

That is one unproven assumption.

>and the robots
>probably handle most of them fine.

And that is another.

Blank lines have been the separator from day one. It is the way we
group records. It confirms this visually. Sure, accepting liberally
is great, but it must stop somewhere.

>Or, even generally allow comments and blank lines anywhere?

Then you're talking about a comletely different set of semantics,
whith their own confusing aspects. I don't see any benefit to that,
let alone benfits which would justify breaking bacwards compatibility.

>Anyway, my original question got lost between syntactic problems, so
>here it is again:
>
>Is
>
>User-agent: infoseek
>User-agent: webcrawler
>Disallow: /tmp
>
>equivalent to
>
>User-agent: infoseek
>Disallow: /tmp
>
>User-agent: webcrawler
>Disallow: /tmp

Yes.

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html