RE: robots.txt usage

David Levine (David@InterWorld.com)
Tue, 18 Jun 1996 13:46:41 -0400


This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible. Contact your
mail administrator for information about upgrading your reader to a version
that supports MIME.

------ =_NextPart_000_01BB5D1C.94CF9480
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

I would have to say that it will vary from page to page, based
on the content. For instance, I imagine that very few of the
people coming in to the Yahoo home page are referred there
by search engines... most probably come from specific links
to the page. On the other hand, an obscure technical paper
may get ALL of its traffic from search engines.

David Levine
Applications Engineer
InterWorld Technology Ventures, Inc.
david@interworld.com

>----------
>From: Brian Clark[SMTP:bclark@radzone.org]
>Sent: Tuesday, June 18, 1996 1:21 PM
>To: robots@webcrawler.com
>Subject: Re: robots.txt usage
>
>-- [ From: Brian Clark * EMC.Ver #2.5.02 ] --
>
>
>> (b) search engines are probably by far the most popular method of
>finding
>> pages; I wouldn't be surprised if they turn out to multiply the
>number
>> of hits on the average site by a factor five or so;
>
>
>If I can poke my head in here on this one, I've got to disagree with this
>assumption (this is a bit off the topic technically, but not
>philosophically
>.) All of the sites that we develop (and many that we retro-fit for others)
>including comprehensive tracking of incoming hyperlinks through the
>REFERRER
>environment variable ... this does a suprising job of tracking not only
>where people find links to you, but also how many people come through that.
>

------ =_NextPart_000_01BB5D1C.94CF9480--