I believe it is only requesting robots.txt every time it starts a new
heierarchy, but I'm not sure. At MIT, this robot often hits our site
tens of thousands of times a day, walking a filesystem. While we
don't object to this, it seems unlikely that they mean to be indexing
this. And, typically it only requests robots.txt a few hundred, or
occasionally thousand times a day, in any case, certainly not once per
request. Still, a bit overzealous.
...Matthew
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html