Re: robots.txt changes how often?

Jeremy.Ellman (jeremy@mari.co.uk)
Mon, 19 Feb 1996 10:25:06 +0000 (GMT)

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: David Schnardthorst: "Robot for Sun"
Previous message: Paul Francis: "Ingrid ready for prelim alpha testing...."
In reply to: Martin Kiff: "Re: robots.txt changes how often?"

>
> I'm wondering whether a script would/could deflect robot activity towards
> the 'quieter' hours on the server. I was sure that there was a 'server too
> busy' error code, one of the '500's, but I can only see a '503 Service
> unavailable' in the draft spec I have....
>

I've thought this for quite a while. If you look at anyone's server
stats you find that they're pretty quiet between Midnight- 7 am (local
time). Maybe there should be a line in robots.txt that says what
they're preferred time for robot access is (in GMT).

Like many other sites, we are happy for robots to look at us, but
would prefer that they came out of office hours. I don't believe there
is a way to express this now. Hence we don't have a robots.txt because
there is nothing constructive we can put it. Correct me if I'm
wrong...

Jeremy

Next message: David Schnardthorst: "Robot for Sun"
Previous message: Paul Francis: "Ingrid ready for prelim alpha testing...."
In reply to: Martin Kiff: "Re: robots.txt changes how often?"