Conceptbot obeys the standard for robot exclusion, and will only submit
requests for matrial with a Content-Type of text/html or text/plain in
order to reduce bandwidth. It is a breadth-first search spider, so hits
to an individual site should be few and far-between. We are working
hard to minimise impact on the net.
Please let us know if the spider is causing you any problems or hassles.
Conceptbot identifies itself with "conceptbot/0.3" in the User-Agent:
field of HTTP headers. The version information may change.
As soon as we have a large enough database to create an experimental
search engine, we'll publish the URL so that you can test it.
Again, please let us know if you encounter any problems with Conceptbot.
-- Dave Sifry david@sifry.com (408) 471-0667 (voice) (408) 471-0666 (fax)