Re: Q: size of the web in bytes, comprehensive list

Otis Gospodnetic (otisg@panther.middlebury.edu)
Mon, 20 Jan 1997 18:59:52 -0500 (EST)

> Yes, You can ftp it from
>
> ftp://rs.internic.net/netinfo/
>
> There is the file is about 50 MEGS!
>
> It appears to be all listed domain names at the Internic. Does anybody
> know of a Java application that will create a database of just the index
> page on each of these servers?

hm, just because there is a domain registered doesn't mean that is has a Web
server (or any server for that matter).

Oh, 50 meg file - some people have told me the same, look for a 50 meg file,
but I found only a 19-20 meg file there.
What am I missing ?

Thanks,

Otis

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html