There is no problem if we don't change the meaning of `Disallow:', at
least not with that example. Then we can just say
User-agent: *
Robot-version: 2.0
Allow: *index.html
Disallow: /images
An 1.0 robot will ignore Robot-Version: and Allow:.
> A version 2 compliant robot would recognize "other2_0" and use that
> rule set, but a version 1 robot would ignore that rule set and use the
> bottom one.
That will work, of course. However, we'd have the same problem with
specific user-agents, so it must either be possible to specify the robot
version with other user agents as well:
User-agent: 2.0:thatspider
or User-Agent-2.0: thatspider
Or robots.txt authors must spend time to find out the robot version of
each User-Agent. I'd prefer the former.
Regards,
Hallvard
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html