robot-id: conceptbot robot-name: Conceptbot robot-cover-url: http://www.aptltd.com/~sifry/conceptbot/tech.html robot-details-url: http://www.aptltd.com/~sifry/conceptbot robot-owner-name: David L. Sifry robot-owner-url: http://www.aptltd.com/~sifry robot-owner-email: david@sifry.com robot-status: development robot-purpose: indexing robot-type: standalone robot-platform: unix robot-availability: data robot-exclusion: yes robot-exclusion-useragent: conceptbot robot-noindex: yes robot-host: router.sifry.com robot-from: yes robot-useragent: conceptbot/0.3 robot-language: perl5 robot-description:The Conceptbot spider is used to research concept-based search indexing techniques. It uses a breadth first seach to spread out the number of hits on a single site over time. The spider runs at irregular intervals and is still under construction. robot-history: This spider began as a research project at Sifry Consulting in April 1996. robot-environment: research modified-date: Mon, 9 Sep 1996 15:31:07 GMT modified-by: David L. Sifry