>my own suggestion would be to complement the "Disallow" operator with an
>"Allow" operator... this may also make robots.txt easier to
>write if your allow list would be shorter than your disallow list.
And would prevent the /A /B /C disallow hacks people use.
>is robots.txt a meta information entity which should include only negative
>and not positive information? can meta information require meta
>information, eg, "expires"? would robots.txt need "expires" more if it
>included "Allow" than with only "Disallow"?
Not sure what you mean here. the /robots.txt is a document, and
as such can be served by HTTP with an Expires header.
>perhaps most importantly, the grammatical primacy of regexp-enhanced bot
>types in the current robot exclusion draft implies that the "Disallow"
>operator is binary,
you lost me there... :-)
>and so the grammatical logic of the exclusion draft is
>incomplete in the sense of only half as expressive as could be... without
>"Allow" in addition to "Disallow".
>so would Dis/Allow make robots.txt the "robots' clusion standard" entity? :)
>maybe robot cooperation or robot direction is better.
Yes, not half :-)
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html