Re: robots.txt extensions

Skip Montanaro (skip@automatrix.com)
Tue, 16 Jan 1996 08:54:14 -0500

Steven L. Baur writes:

A little too good sometimes. The problem comes when one has an
archive of something like a manual that is under development. I
maintain two such archives, and had several robots going through the
pages while pieces were being changed actively, while other more
static pages were ignored.

Hmmm... seems like you need a Disallow: item to keep those pesky robots
away from your development tree.

I too think a positive hint would be useful. If a robot is well-behaved, it
will take some period of time to munch my entire site. I'd like to be able
to suggest where it should munch first.

Skip Montanaro | Looking for a place to promote your music venue, new CD,
skip@calendar.com | festival or next concert tour? Place a focused banner
(518)372-5583 | ad in Musi-Cal! http://www.calendar.com/concerts/