Re: robots.txt , authors of robots , webmasters ....

Adam Jack (ajack@corp.micrognosis.com)
Thu, 18 Jan 1996 18:59:12 -0500

Benjamin Franz wrote:
>
> On Thu, 18 Jan 1996, Robert Raisch, The Internet Company wrote:
>
> > In other words, we should not index a web-site if there is no
> > robots.txt file to be retrieved that gives explicit permission
> > to do so.

>
> In other words - it completely destroys the usefullness of robots for
> resource discovery.
>

I wonder wether it would, instead, be the single best mechanism for
mass education.

What if a number of the major robots proclaimed that this was to be
the case as of XXXXX date. No longer would a site be accessed (for
either update or review) -- if it didn't have a robots.txt.

I doubt these robots would loose out -- since they, undoubtedly,
have not completed a full WWW index. They'd still have plenty of
information -- it would probably be increased quality also....

People who want their data accessed would conform. Editting a robots.txt
wxis no more difficult an activity than using submit-it.

Adam

--
+1-203-730-5437 | http://www.micrognosis.com/~ajack/index.html
ajack@corp.micrognosis.com -> ajack@netcom.com ->  ajack@?.???