> I would start from http://rs.internic.net/
> I think they have just that kind of info at their FTP site....
Yes I've seen that file. Is that where people point their robots at ? I
guess I'm looking for a way to automate this process by querying some
Internet service. That's why I was looking at the name server issue. The
Internic file is fine but it only looks after US specific domains and
not .au, .nz etc country specific domains. Other country specific NIC's
don't always offer this file. Besides I guess the file would get out of
date very rapidly.
How do web crawlers find new sites ? Is it purely from existing
references in HTML documents ? What if a site has no listing anywhere
else on the Internet, how would a crawler ever find this site if it
couldn't locate the new domain from a name server ?
Sorry to be a pain but this is a concept I'm finding hard to grasp! Gee
I luv this list serve it's the most interesting I've seen !!
Thanks again for the help,
Kind Regards,
Neil Cotty.
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html