Re: robots.txt unavailability

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Istvan : "Re: nastygram from xxx.lanl.gov"
• Previous message: Steve Leibman: "htaccess"
• Maybe in reply to: Daniel T. Martin: "robots.txt unavailability"

>What is the correct way of accessing robots.txt?
>
>Is it always located at http://<host>:[<port>]/robots.txt ?

Ayup.

>Or are you supposed to do something other than a GET /robots.txt HTML/1.0 ?

Nope.

>(I can't find a robots.txt at www.lanl.gov thru Netscape, for example.)

Misconfiguration or sloopyness...

-- 
R Levitte, Levitte Programming; Spannvägen 38, I; S-161 43  Bromma; SWEDEN
Tel: +46-8-26 52 47, (via nation.se) +46-8-21 24 90;  No fax right now
PGP key fingerprint = A6 96 C0 34 3A 96 AA 6C  B0 D5 9A DF D2 E9 9C 65
Check http://www.lp.se/~levitte for my public key.

• Next message: Istvan : "Re: nastygram from xxx.lanl.gov"
• Previous message: Steve Leibman: "htaccess"
• Maybe in reply to: Daniel T. Martin: "robots.txt unavailability"