Re: RFC, draft 1

Klaus Johannes Rusch (e8726057@student.tuwien.ac.at)
Wed, 20 Nov 1996 00:07:49 CET

In <v02140b02aeb2aa2bd2fe@[192.216.46.39]>, m.koster@webcrawler.com (Martijn Koster) writes:
> The record starts with one or more User-agent lines, specifying
> which robots the record applies to, followed by "Disallow" and
> "Allow" instructions to that robot. For example:
>
> User-agent: webcrawler
> User-agent: infoseek
> Allow: /tmp/ok.html
> Disallow: /tmp
> Disallow: /user/foo

The scope of User-agent is still somewhat ambiguous, does it end at the next
User-agent line, or at the next User-agent line after an Allow/Disallow line?

In other words, should

# 1. two agents with no lines between
User-agent: webcrawler
User-agent: infoseek
Disallow: /tmp

# 2. two separate definitions
User-agent: webcrawler
Disallow: /tmp
User-agent: infoseek
Disallow: /tmp

# 3. two agents with some lines inbetween
User-agent: webcrawler

# comments and blank lines

User-agent: infoseek
Disallow: /tmp

be equivalent?

Klaus Johannes Rusch

--
e8726057@student.tuwien.ac.at, KlausRusch@atmedia.net
http://www.atmedia.net/KlausRusch/
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html