If Anyone is interested this is the robots.txt file from
xxx.lanl.gov.
# robots.txt for http://xxx.lanl.gov/
User-agent: *
Disallow: /e-print/
Disallow: /ps/
Disallow: /year/
Disallow: /pdf/
Disallow: /abs/
Disallow: /find/
Disallow: /absfind/
Disallow: /ftp/
Disallow: /refs/
Disallow: /cits/
Disallow: /list/
Disallow: /format/
Disallow: /PS_cache/
Disallow: /lh94/e-book/
Disallow: /lh94/photos/
Disallow: /Announce/Conference/
Disallow: /Announce/Jobs/
Disallow: /seek-and-destroy
Chris, ChegHchu djajVam djajKak!
wanabe robot programer - don't shout at me......to much <g>
-----------------------------------------------------------
chris@jm-crowther.co.uk
www.dungeon.com/~jmcrowther/chris.html