>As some members of the list have been quick to point out, our robot is
>(almost) no different than any other search-engine indexing robot, so all
>the exclusion rules and load-politeness etiquette that applies to them
>applies to us.
Just a quick check. Will you be using robots.txt to guide the robot
away from graphics or just html ?
I've always had my graphics blocked from robots via robots.txt. Will
your robot respect that or will it be allowed to grab any inline images
on a robot-allowed html page ?
btw, what USER-AGENT will you use to indetify the robot to the servers
being visited ?
thanks,
rob
-- Rob Hartill (robh@imdb.com) http://www.imdb.com/ ... why wait for a clear night to see the stars?.