Re: Inktomi & large scale spidering

Nick Arnett (narnett@Verity.COM)
Sun, 26 Jan 1997 10:26:31 -0800


At 12:03 PM 1/26/97 -0500, Otis Gospodnetic wrote:

>Netscape's Web servers already have tons of configurable options accessed via
>Web GUI, so adding something like a On/Off switch for this local index would
>be easy, and persuading Netscape's developers would be the only 'difficult'
>task.

Netscape (and a number of other) servers come with our search engine built
in. The indexes could easily be in a publicly accessible location under a
well-known name. The obstacle is that index formats are proprietary because
they contain structures that give the engine a lot of its speed and
accuracy. Thus, any attempt to have an open standard would have to allow
for multiple engines... and with index disk space overhead of 30-60 percent
(or more, with some older technology), webmasters are unlikely to be eager
to support more than one.

Nick

---------------------------------------
Verity Inc.
Connecting People with Information

Product Manager, Categorization and Visualization
408-542-2164; home office 408-369-1233; fax 408-541-1600
http://www.verity.com

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html