User-Agent: BSE-Slurp/0.6
From: gauthier@cs.berkeley.edu
read but did not heed my /robots.txt file. Does anybody know where
this one comes from? It appears in tons of agent logs, but I don't
see an entry for it in "known robots."
Here's my /robots.txt file, as read by lynx:
------
barley 104>lynx -source http://barley.uwaterloo.ca/robots.txt
# robots.txt for http://barley.uwaterloo.ca
# this is a search engine. go away.
User-agent: *
Disallow: /
------
Here it is again, using telnet:
------
barley 105>telnet barley.uwaterloo.ca 80
Trying 129.97.186.13...
Connected to barley.
Escape character is '^]'.
GET /robots.txt HTTP/1.0
# robots.txt for http://barley.uwaterloo.ca
# this is a search engine. go away.
User-agent: *
Disallow: /
Connection closed by foreign host.
barley 106>
------
Gordon V. Cormack CS Dept, University of Waterloo, Canada N2L 3G1
gvcormac@uwaterloo.ca http://cormack.uwaterloo.ca/~gvcormac