An updated extended standard for robots.txt

Captain Napalm (spc@armigeron.com)
Mon, 11 Nov 1996 18:29:35 -0500 (EST)

Okay, I've made some updates to the extended standard for robot exclusion.
It's still located at:

http://www.armigeron.com/people/spc/robots2.html

I should warn that I used the <strike></strike> tag, so if anyone has any
difficulties with this, I'll remove them.

The updates include clarifications of retrieval rates, an update on the
'Visit-time:' syntax (as well as allowing more than one), specified what
exactly should be used for regexs (basically, what one finds in /bin/sh on
Unix) and got rid of the Allow:/Disallow: precedence (changed it to order it
appears in file).

Thanks for the feedback. Things look like they're coming along.

-spc (chug chug chug)