> (b) search engines are probably by far the most popular method of finding
> pages; I wouldn't be surprised if they turn out to multiply the
number
> of hits on the average site by a factor five or so;
If I can poke my head in here on this one, I've got to disagree with this
assumption (this is a bit off the topic technically, but not philosophically
.) All of the sites that we develop (and many that we retro-fit for others)
including comprehensive tracking of incoming hyperlinks through the REFERRER
environment variable ... this does a suprising job of tracking not only
where people find links to you, but also how many people come through that.
Drawing from a cross section of sites (ranging from half-million hits a day
to a few thousand hits a month), search engines represent only a fraction of
the referrers to the websites (highest stat was 22%, lowest was 7%). This is
not to say that search engines aren't an important way for users to find
what they are looking for, but our experience shows that prominant linkage
from high traffic site (whether through purchased advertising, "cool site of
the day" or other mentions) generates hundreds of times the traffic of
effective search engines.
As a case study, we're currently running a net-event (Florida Film Festival
Cybercast at http://www.enzian.org/fff_96/cybercast.html) that has a couple
of prominant linkages. For comparison, the last 5 days have seen the
following number of visitors come from various sources:
Yahoo category searches: 18
Yahoo "Picks" area: 126
Lycos searches: 8
Excite searches: 2
Filmmaker Magazine reciprical link: 488
Real Audio feature link: 2957
Virtual Film Festival reciprical link: 948
Other sources: 428
In summary, while I believe that search engines play an important role in
helping users find what they are looking for, these databases have become so
large (for the most part) that placement is much more important (from the
perspective of a webmaster looking to build traffic) than just inclusion. I
think your other arguments hold more water - and I, too, believe that
excluding all robots is not what the exclusion standard was geared to do.
-------------------------------------------------------------------- Brian Clark - Production Coordinator - bclark@radzone.org ------------------------------------------------------------------ GlobalMedia Design http://www.radzone.org/gmd/ Radiation http://www.radzone.org/gmd-bin/radiation Email Autobot radbot@radzone.org Rover Search Service http://www.roverbot.com/ ------------------------------------------------------------------