I'd read the thread of some people's befuddlement on how McKinley's robot
hit their site...well it just hit my site...<a
href="http://www.intercom.com">Intercom Online</a>
galileo.mckinley.com [12/May/1996:23:32:06 /robots.txt - Wobot/1.00
galileo.mckinley.com [12/May/1996:23:32:07 / - Wobot/1.00
galileo.mckinley.com [12/May/1996:23:42:27 / - Wobot/1.00
galileo.mckinley.com [12/May/1996:23:49:14 / - Wobot/1.00
I program for McKinley, and helped write the spider. What was
happening here was that "/" happened to be a redirect to index.htm at
this server. We weren't storing that information, so every time it
received a new reference to "/" at this server, the spider tried to
retrieve it again. This has been fixed. Hopefully, the time-delay
between site visits helped reduce any inconvenience.
Thanks,
Rafhael Cedeno
cedeno@mckinley.com