Re: robots.txt extensions

Adam Jack (ajack@corp.micrognosis.com)
Thu, 11 Jan 1996 00:37:14 -0500

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Mark Schrimsher: "Re: Does anyone else consider..."
Previous message: : "Re: Does anyone else consider..."
Maybe in reply to: Adam Jack: "robots.txt extensions"
Next in thread: Martin Kiff: "Re: robots.txt extensions"

Jaakko Hyvatti wrote:
>
> I am sorry I sound quite negative.. Actually, the ideas might be
> pretty good. I do not mean to be rude :-)
>
Don't worry about that. I appreciate your information. Thanks.

Okay - so zero for 3... How about I just comment on the following :

monier@pa.dec.com wrote:
>
> sites with better /robots.txt [...]
> index, and read stuff that was never intended to be indexed.
>
Our admin knows + cares little for our content. Our content is
automatically transfered to our own sub-tree locations each hour.

We are in a position to determine what is worthy & what not --
but not in a position to modify /robots.txt.

I am sure our site is not alone in this...

Adam

--
+1-203-730-5437 | http://www.micrognosis.com/~ajack/index.html

Next message: Mark Schrimsher: "Re: Does anyone else consider..."
Previous message: : "Re: Does anyone else consider..."
Maybe in reply to: Adam Jack: "robots.txt extensions"
Next in thread: Martin Kiff: "Re: robots.txt extensions"