RE: Preferred access time
Martin.Soukup (martin.soukup@fulcrum.com)
Tue, 24 Sep 1996 12:42:07 -0400
>
>> Sorry if this question is moronic, however, i can't seem to find
>> anywhere a definitive spec for where to find the preferred access times
>> for servers, and a concrete value for backoff times for bots and
>> crawlers. Is this because there is no concrete spec for them? Thanks a
>> billion.
>>
>> M/
>
>how would one want to handle this?
>
> AccessFromToGMT HHMM HHMM
>
>in "/robots.txt"?
>
>or HTTP Error Code?
>
> XXX Server too busy for non humans
>
either of these would be great, in fact both would be better. But how
likely is it for this to become widely used? A large number of sites
aren't even using a robots.txt yet. If there is no way to query the
server for this information currently, i suppose the only way is to do
your own load test?
>M/