Re: robots.txt usage

Daniel Lo (daniel@servonet.com)
Tue, 18 Jun 1996 07:43:24 -0700


> Hi,
>
> I've written a robot+searchengine (www.il.ft.hse.nl/ilse (dutch sites only))
> and I just implemented the robots.txt 'stay away' stuff, but it caused some
> problems...

> Sysadmins probably would like their sites to be as fast as possible, but
> the data on those site will not be in Ilse, so people will blame Ilse for
> being incomplete...
>
> Have others seen the same ? if so, what should I do ?
> I can't go contacting all the webmasters/roots at these machines... And I
> don't want to disguard a possible /robots.txt...
>

I would definatly like my site to be as fast as possiable, but
I also want it indexed so that people could find my site. Whats
the point of having an un-listed site?

- daniel