The Internet Archive robot, which will identify itself as "ia_archiver" in
the "User-Agent:" HTTP header field, will begin archiving the Web over the
next few days.
In the short term, the archiver will focus only on images; the lists of
these have been derived from our existing HTML feeds, kindly donated to us
by several of the commercial search engines.
The archiver will obey the Standard for Robot Exclusion, and will take pains
to tread softly on the surface of the Net. If anyone feels we have failed
in either case, please let us know so they we can rectify the problem as
quickly as possible.
Internet Archive is gathering, storing, and providing access to public
materials on the Internet such as the World Wide Web, Netnews, and
downloadable software. The collection, reaching ten terabytes, will provide
historians, researchers, scholars, and others access to this vast collection
of data, and ensure the longevity of the information.
For more information, including how you can help the Archive, please visit
our web site (http://www.archive.org).
Mike Burner