Re: An extended verion of the robot exclusion standard
Captain Napalm (spc@armigeron.com)
Mon, 11 Nov 1996 14:29:59 -0500 (EST)
It was thus said that the Great Hrvoje Niksic once stated:
>
> Captain Napalm (spc@armigeron.com) wrote:
> > Anyway, take a look at http://www.armigeron.com/people/spc/robots2.html
> > and give me feedback. Also, if anyone has implemented the extentions
>
> <comment> - general comment line format
>
> Any text following a "#" up to the end of a line is to be
> ignored. The "#" character can appear at any portion of a
> line. Some examples:
>
> I don't like this -- I prefer the old definition that a blank must
> precede the comment. It makes sense in sh, and it makes sense in
> /robots.txt.
>
Okay, this I can change as well.
[ rest about regexs deleted - as discussion is still going on about that ]
-spc (I'll make the changes when some more consensus about regexs are
solidified)