Re: robot clusion; was Re: we should help spiders and not say NO!
John D. Pritchard (jdp@cs.columbia.edu)
Mon, 01 Jul 1996 11:06:00 -0400
> >my own suggestion would be to complement the "Disallow" operator with an
> >"Allow" operator... this may also make robots.txt easier to
> >write if your allow list would be shorter than your disallow list.
>
> And would prevent the /A /B /C disallow hacks people use.
>
> >is robots.txt a meta information entity which should include only negative
> >and not positive information? can meta information require meta
> >information, eg, "expires"? would robots.txt need "expires" more if it
> >included "Allow" than with only "Disallow"?
>
> Not sure what you mean here. the /robots.txt is a document, and
> as such can be served by HTTP with an Expires header.
just asking "why no Allow, anyway"? this is the only reason i could
imagine, that robots.txt was intended as meta information which wouldn't
require meta information... and that a Disallow grammar would not imply a
need for an expiration or other meta data.
-john