I am looking for websites, journals, etc.. that will provide
detailed information on the following items. I need the information
for a class presentation discussing "internet spiders". I am only concerned
with spiders used for information retrieval and indexing, or website/link
maintenance.
Specifically, I am interested in finding:
1) Information on what algorithms spiders use to avoid overloading
servers
2) Information on what indexing/similarity/etc algorithms are used
on the information that a spider gathers, and what kind of searches
(ie keyword, concept space, etc.) these algorithms support.
3) Actual code of any spider
The student in the class are all advanced programmers, so I don't want to
bore them with the definition of a spider, etc.. I want to get into
algorithmic detail.
I would appreciate any and all information.
please send mail to : pinkman@bpaosf.bpa.arizona.edu