Re: robots.txt syntax
Captain Napalm (spc@armigeron.com)
Mon, 14 Oct 1996 09:56:42 -0400 (EDT)
>From some obscure corner of the Matrix, John D. Pritchard was seen transmitting:
>
>
> napalm,
>
> that's way too complicated...
>
> if disallow rules take precedence, everything's simple. if both rule types
> are matching, then the request is disallowed. i don't think this impedes
> the expressiveness of the resulting rule language. the rule language then
> has disallow as an absolute disallow rather than a fuzzy, implementation
> dependent disallow. :-)
>
Well, the reason I had explicit allows higher than disallow was for the
following case:
Allow: /index.html
Disallow: *
Such that the only file allowd would be /index.html and nothing else.
After that, disallow rules take precedence. Something that hasn't been
clarified (by me or anyone else) is the following:
Allow: /index.html
Does that mean ONLY /index.html is allowed? In general, if it isn't
explicitely allowed, generally disallowed, is it allowed?
-spc (And nothing in my scheme would disallow explicit disallows ... )