About the question what a robots is (was Re: PS)

Michael Schlindwein (m_schlin@informatik.uni-kl.de)
Tue, 16 Jul 1996 14:35:04 +0200 (MET DST)


> >
> >Great idea, but again is begs the question, "What qualifies as a robot?" For
> >example, does WebCompass (which retrieves two levels deep to summarize
> >findings) count as a robot? What about other types of "user agent"
> >interaction?
> >
> >Getting everyone to agree to put (robot) in their User-Agent will pretty
> >much require (again) an active definition of what should and shouldn't. In
> >my opinion, this is even less a cut-and-dried issue than "Should it adhere
> >to the robot exclusion standard?"

Rob Hartill wrote:
>
> Anything that's not directly human controlled and that has not been given
> permission to visit the site unregulated.

Depending on what exactly you consider "directly human controlled" and
visiting a "site unregulated" I think I would agree with you.
But these expressions may also be interpreted in a way that makes this
criterion too coarse.

I currently work out several suggestions, to the problem of categorizing
programs that access Web pages, which then will hopefully be discussed on
the list.

I think, that as time has passed and the Web has grown more and more, not only
in size but above all also in popularity
(please have also in mind, that in Germany it is not for a long time yet that
many companies are present in the internet and that the acceptance from
people/consumers has just started; the opinions about the "end of the game"
are still oppositional),
the time is over to see only indexing and/or statistics gathering and/or
link maintenance programs and to have only these programs in mind when talking
about (automatic) Web access in general and about robots in special.

I believe, that reflecting on (automatic (BTW: How is this one defined? ;-))
Web access in general will open the minds and then bring out a differenced
view of such programs and a (perhaps) comprehensive set of properties, that
identify programs which should be called robots and hopefully the insight, that
there are levels between human (browser) access and robots (in the sense of an
indexing program which accesses an enormous amount of documents).

>
[...]
>
> --
> Rob Hartill (robh@imdb.com)
> The Internet Movie Database (IMDb) http://www.imdb.com/
> ...more movie info than you can poke a stick at.

I'll hurry up now ;-)

Mike