> I know what the standard is when a site reports (code 404) that the
> /robots.txt URL is not found; what is the standard behavior when other
> responses are received (say if a site reports code 403 - Forbidden)?
What is the correct way of accessing robots.txt?
Is it always located at http://<host>:[<port>]/robots.txt ?
Or are you supposed to do something other than a GET /robots.txt HTML/1.0 ?
(I can't find a robots.txt at www.lanl.gov thru Netscape, for example.)
-- Fred