>For example, if WebCrawler already indexed URLs: A B C D, does it go through
>its database, check if A has been updated, and the same for all other URLs,
>and if they have, re-fetch and re-index them, find if there are any new (never
>seen before) URLs in them, and so on ?
>
>That seems kind of limited to me....
It would be a bit, depending on what A-D are.
>Or does a robot have some permanent source of new URLs that it then fetches,
>indexes, and goes into web-crawling from there ?
Sure. There are user submissions, and news feeds for a start...
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html