Re: Crawling & DNS issues

Martin Beet (Martin.Beet@ncl.ac.uk)
Wed, 22 Jan 1997 09:19:41 +0000


Neil Cotty wrote:
>
> [...]
> How do web crawlers find new sites ?

directories
user registration
"what's new" lists
usenet postings (alt.internet, comp.internet.happenings,
comp.infosystems.www.announce)

more, anyone?

-- 
Regards, Martin
---------------
University of Newcastle Dept. of Computing Science | Tel:+44 191 2226157
Claremont Tower, Newcastle upon Tyne, NE1 7RU, UK  | Fax:+44 191 2228232
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html