>The same incentive as they have in implementing any of RES, which might
>not be very much.
Quite.
>Again, I think someone should provide a Perl library
There is: libwww-perl5. I've not used that myself.
>and a C library which implements the entires RES;
As discuessed before, I'm happy to share the WebCrawler /robots.txt
parsing code (ANSI C) with anyone who asks? :-)
>that should help considerably.
>A better approach might be to use headers with *options*. Example:
>
> Hide: /Foobar # explicit
> Hide;R: /Foo.*barbaz # regexp
> Show;R;I: /foo.*bar # regexp; ignore case
> Show;3-: /baz.* # Only for robot versions 3 and later
Complex...
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html