> It was thus said that the Great Captain Napalm once stated:
> >It was thus said that the Great Hallvard B Furuseth once stated:
[...]
> >> No. Then a robots.txt's default rule set (User-agent: *) would have to
> >> assume the robot does not understand version 2. This makes version 2
> >> pretty useless.
> >
> > You have to assume that reguardless of what the 2.0.0 spec looks like.
>
> No no no -- it's true that webmasters should write their robots.txt so
> it works for old robots, but they may also add rules that makes new
> robots behave *better* than old ones. That would be impossible if new
> rules interfere with old ones, so the default rule set cannot contain
> new rules.
There appears to be a problem with throwing *all* unknown robots into the
same default rule set regardless of whether they understand the 2.0 spec
or not. Has anyone suggested a special code for default version 2
robots? For example:
User-agent: other2_0
Robot-version: 2.0
Allow: *index.html
Disallow: /images*
...
User-agent: *
Disallow: /images
...
A version 2 compliant robot would recognize "other2_0" and use that
rule set, but a version 1 robot would ignore that rule set and use the
bottom one.
-- LLLL LLLLL LLLLLL Arthur Matheny LIB 612 LL LL LL LL LL Academic Computing University of South Florida LLLLLL LLLLL LL matheny@usf.edu Tampa, FL 33620 LL LL LL LL LL 813-974-1795 FAX: 813-974-1799 LL LL LL LL LL http://www.acomp.usf.edu/
_________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html