I've thought this for quite a while. If you look at anyone's server
stats you find that they're pretty quiet between Midnight- 7 am (local
time). Maybe there should be a line in robots.txt that says what
they're preferred time for robot access is (in GMT).
Like many other sites, we are happy for robots to look at us, but
would prefer that they came out of office hours. I don't believe there
is a way to express this now. Hence we don't have a robots.txt because
there is nothing constructive we can put it. Correct me if I'm
wrong...
Jeremy