Re: databases for spiders
Nick Arnett (narnett@Verity.COM)
Thu, 14 Nov 1996 13:19:52 -0800
The Verity full-text indexes have a 255-field flat field database in
addition to the full word index, so we we store the URLs with the search
engine. This means that for every document indexed by the spider, there is
also a searchable field that contains every URL in that document.
Nick Arnett
Evangelist
Verity Inc.
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html