Re: defining "robot"

Rob Hartill (robh@imdb.com)
Mon, 25 Nov 1996 20:15:45 +0000 (GMT)

Martijn Koster wrote:
>
>At 2:44 PM 11/24/96, Erik Selberg wrote:
>
>>So does this mean you'll put MetaCrawler in your list of robots? Way
>>back when I think you deemed it "inappropriate for the list" or
>>something. :)
>
>Back then I didn't realise you pulled the links, I thought you just
>returned them.
>
>Of course, if all your sources are search-engines generated by robots
>that do /robots.txt, you should never get a link that violates it :-))

Not necessarily. Lots of the search engines will offer links to my
site that are "protected" by robots.txt. How they got there I don't
know or care just as long as the robot for that index doesn't ignore
robots.txt if it decides to validate the links.

Erik'll know, but I think he told me that MetaCrawler got its recent
forbidden URLs from webcrawler ! :-)

-- 
Rob Hartill.       Internet Movie Database Ltd.    http://www.imdb.com/  
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html