>So does this mean you'll put MetaCrawler in your list of robots? Way
>back when I think you deemed it "inappropriate for the list" or
>something. :)
Back then I didn't realise you pulled the links, I thought you just
returned them.
Of course, if all your sources are search-engines generated by robots
that do /robots.txt, you should never get a link that violates it :-))
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html