actually, it's requesting robots.txt every single time it requests a html file.
> tens of thousands of times a day, walking a filesystem. While we
> don't object to this, it seems unlikely that they mean to be indexing
> this. And, typically it only requests robots.txt a few hundred, or
> occasionally thousand times a day, in any case, certainly not once per
> request. Still, a bit overzealous.
well, this one make 5,500 requests to our web site last week, half of those
request being requests for robots.txt
how dumb is that ? :)
Otis
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html