On the subject of abuse/pro-activeness

Scott 'Webster' Wood (swood@thewild.com)
Tue, 9 Jul 1996 10:56:26 -0400 (EDT)

Just out of curiousity, I chewed apart my server logs to see what kind of
accesses I have received from various robots on the net. Basically, I
searched for requests from the specific domains I know to be search
locations and narrowed the list as best I could. (if anyone is surfing
from these sites, the numbers ARE reflected in my totals)

I stripped out those entries that were less than 100 requests (yahoo
and lycos were just under 100) and ended up with the following for
the past 6 months. At two points in time, requests were made to
all four of these and others that were <100 (including lycos and yahoo)
to index certain portions of my site.

I have also since found out that there is another company doing some
collecting for digital, and thus the altavista numbers may not be complete.
(really!) These totals do not count situations where individuals are using
the 'named' search programs on their site, as these totals are from the
actually addresses where the requests are made from. Anyone know if these
companies' software uses the company's site to handle customer requests?

altavista 707
magellan 538
excite 242
infoseek 153

Totals reflect mid February to the end of June, takes into account all httpd
requests, and is from a log of 242068 requests.

Scott