I ask because it appears that this is handled inconsistently; opentext and
lycos apparently treat this response as though it were a "not found"
response - altavista's scooter, however, treats the response as though it
were a file consisting of:
Disallow: *
That is, it assumes that the site is completely off-limits.
Based on my own experience, I'm inclined to say that it should be treated
as a "Not Found" response, as should other server errors.
-=-
Daniel Martin * your sig here *