We have a project, called Ingrid, to create a fully distributed,
self-configuring search infrastructure. It automatically creates
links between relevent servers, and the resulting topology can be
efficiently searched in real-time by robot-like or agent-like
processes.
Please see http://www.ingrid.org/
PF