We have such a robot available for our search products. Take a look at our
site (http://www.verity.com/). We have quite a few customers using it for
this purpose. It is not a free-ranging robot; you tell it which servers
and which directory trees you want to index. In other words, it doesn't
jump from server to server, so you can focus it on your intranet or parts
of it.
The basic robot that comes with our Web server products will only index the
machine on which it's running. There's an upgrade to a version that will
index other machines (anywhere on the net). These products are shipping
for a number of Unix and Windows platforms.
In addition, although it's not shipping yet, Netscape has announced its
Catalog server, which is designed for this specific purpose. It is also
based on our search engine. There's information about it on their site,
(http://www.netscape.com/).
Nick Arnett
Internet Marketing Manager
Verity Inc.