A plug of my own...
We have a research project going that we hope in the not
too distant future will lead to this kind of functionality.
What it does is automatically set up an alternative web
topology (fully distributed), whereby the links are topical
rather than explicit pointers as in HTML. The resulting
topology is then searched from the local site robot-style
(at search time).
Our hope is that 1) this approach will scale well, 2)
by pushing the control of both indexing and searching to
the users, we can improve the quality of searches overall,
and 3) other interesting applications can be built on
the distributed infrastructure.
One of the features of our navigator/robot that would help
you is that you can search for things that are new since your
last search in the topical area of interest.
Some drawbacks to the scheme for now are 1) that we first
have to make it work, and 2) that a large number of people
have to locally install it.
We are just about release our alpha version. (Embarassingly,
I said about the same thing last February! But this time it
is true. We have finished the binaries we plan to release, and
are now cleaning up the documentation and just waiting for
after the Paris WWW conference, where we are exhibiting.)
Anyway, check us out at http://www.ingrid.org. We are in
general looking for people to play with our alpha code.
PF