An extended verion of the robot exclusion standard

Captain Napalm (spc@armigeron.com)
Fri, 8 Nov 1996 14:22:35 -0500 (EST)

Last month I posted an article
(http://info.webcrawler.com/mailing-lists/robots/1460.html) about some
extentions to the robot exclusion standard (robots.txt). I've gone ahead
and converted that to an HTML document describing the extentions I think are
needed. It doens't have some of the extentions recently mentioned, as
that's still being hashed out.

Anyway, take a look at http://www.armigeron.com/people/spc/robots2.html
and give me feedback. Also, if anyone has implemented the extentions
mentioned, I'd be willing to use a few sites that I maintain as a testing
ground. Email me for more details on that.

Thanks.

-spc (Note - this ISN'T a request for unsolicited junk email trying to
sell me email spamming programs)