> I think the only reasonable solution in a lot of ways is to make a
> spider which is attached to a server. This spider would then create
> the tarballs (and you could have one big file, as well as a week's
> worth of changes in another for incremental updating). The spider
> could also do other useful things, like make sure you got all your
> scripting done right, you didn't forget any links, etc. etc. etc.
>
> Rob --- would this be something the apache folks would be interested
> in?
Probably not directly, but anyone can write their own module for
others to share.
_______________________________________________________________________
Rob Hartill. Internet Movie Database Ltd. http://www.imdb.com/
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html