Re: Inktomi & large scale spidering

Martin Hamilton (martin@mrrl.lut.ac.uk)
Sun, 26 Jan 1997 12:37:04 +0000


--==_Exmh_-2120710340P
Content-Type: text/plain; charset=us-ascii

andy@andy.net writes:

| I think we can all agree the aliweb model was not the greatest idea.
| If webmaster have to create the index manually it will not gain
| widespread support. If somebody were to invent a nice clean java
| app that would form the index and took virtually no time to setup
| perhaps you could get some support. However, there were a number
| of great perl scripts to form site indices for aliweb and very few
| sites used them. Would and applet be enough less configuration
| and hassle that people might actually use it??? Nobody knows. I
| guess there is only one way to find out, but I'm not going to rush
| out and write it.

I suspect the most effective approach would be to make this automatic
index generation a feature of the WWW server itself - i.e. something
which the server admin doesn't have to install & configure separately.
Let them add manually generated metadata, and let the indexer harvest
author-supplied metadata (e.g. embedded in HTML docs), but don't rely
on either of these. Oh, and ship the HTTP daemon with this feature
turned on by default ;-)

Cheerio,

Martin

--==_Exmh_-2120710340P
Content-Type: application/pgp-signature

-----BEGIN PGP MESSAGE-----
Version: 2.6.3i

iQCVAwUBMutP59ZdpXZXTSjhAQEbSwP6A1PfTXK9PAnawo0ne+b1yHFMu2qUL8a+
wU4vm4QylzqE0n2Z+PNX67YJnh3NZTOnz/k5BlexjjHqqwx39yqVyxGmBXax820N
/DJ9RBejQfpEgqLbOuM82XPSOTh9snG3xZYP6VjkJ9nsKiMWrEUTHUF22CX9fbBF
1FkRelWUC6c=
=wJME
-----END PGP MESSAGE-----

--==_Exmh_-2120710340P--
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html