Re: McKinley Spider hit us hard

Michael Van Biesbrouck (mlvanbie@undergrad.math.uwaterloo.ca)
Tue, 5 Dec 1995 15:56:33 -0500 (EST)

> A spider from magellan.mckinley.com hit us hard today and did a
> deep recursive search of our web tree. Not very friendly, but their
> spider did check /robots.txt which indicates that they may have
> successfully implemented the robot exclusion protocol.
>
>
> Christopher Penrose
> penrose@ucsd.edu
> http://www-crca.ucsd.edu/TajMahal/after.html
>
> here is their internic info if anyone else wants to complain to them:

The spider in question is Wobot/1.00; the correct person to bother
with complaints is cedeno@mckinley.com.

They visited a site that I watch over on 21 Nov and did nothing after
reading /robots.txt. The robots.txt is somewhat long, but not very
restrictive.

However, it seems to have gone ballastic today on another machine. As a
result I will be complaining. In this case it came from
radar.mckinley.com.

I sugest that other people check their logs and complain if necessary.

-- 
"You're obviously on drugs,      Michael Van Biesbrouck
but not the right ones."         ACM East Central Winning Team
-- bwross about mlvanbie         http://csclub.uwaterloo.ca/u/mlvanbie/