If there really are clearly separate classes of activity worthy of distinct
exclusion rules, then perhaps the robots.txt file should be tarted
up to cater to them. In the meantime, I think that a robots.txt file
that says: "Robots not welcome here" should be honoured by all
non-humans, unless specific permission has been obtained.
It's always easy to assume that what your program does is an
exception, or not that big a deal, but that's still jumping to a
conclusion about someone else's motives in excluding robots,
and indeed what they consider to be a robot. If I had such a
blanket exclusion file up, and an automatic process ignored it
without getting my permission first, I'd consider its author to
have bad manners no matter what the program did.
-- Frank Wales [frank@limitless.co.uk]