Yes... speaking from experience, we had a bug in our big robot's robots.txt
processing one day and [Murphy leaping into the breach] of course it was
xxx.lanl.gov that we flailed away at a bit. Provoking a storm of mail-bombs,
nastiness, and threats that was quite extraordinary.
Of course they then locked us out of all of their site, which meant that
we could no longer retrieve their /robots.txt, which at that time we took to
mean that there was none, thus we went and tried to do a bit more crawling;
more mail-bombs, nastiness, threats, etc....
Suffice it to say that for reasons that doubtless seem good to them, the
kind, gentle, wise, mature folks who run xxx.lanl.gov do not want the
attentions of anyone on this list, to the extent that they have invested quite
a lot of ingenuity and time and money (all well-spent and highly relevant to
the core mission of their enterprise, I'm sure) in making life unpleasant for
anyone who transgresses. They do carefully note that they are "not
responsible" for any of the consequences of this policy. -Tim