Re: An extended version of the Robots...

Martijn Koster (m.koster@webcrawler.com)
Fri, 15 Nov 1996 17:02:46 -0800


At 1:09 PM 11/15/96, Hallvard B Furuseth wrote:

>>> Show;R;I: /foo.*bar # regexp; ignore case
>>> Show;3-: /baz.* # Only for robot versions 3 and later
>>
>> Complex...
>
>Why, no.
>
>;R is no more complex than parsing the Allow/Disallow: arguments to find
>out whether they are a regexps, explicit or general matches. And it's
>safer because explicit matches can't accidentally be parsed as regexps
>if we insert an URL with an unusual character.
>
>;I is no more complex than an Ignore-Case: field if we want that
>functionality.
>
>;3- *is* maybe more complex than a Robot-Version: field, if that is
>introduced somehow. Like I said, I'm not sure I like that one.

I didn't mean conceptually complex, or complex to implewment,
just syntactically complex for people who aren't used to looking at code.
"Disallow" or "DisallowRegexp" are more obvious and easier on the eyes.

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html