Internet Agents: Spiders, Wanderers, Brokers, and Bots
by Fah-Chun Cheong
New Riders
The WebWalker perl source can be found at:
http://deluge.stanford.:8000/book/WebWalker
it requires the libwww-perl:
http://www.ics.uci.edu/WebSoft/libwww-perl/
I'd give you my spider, NetNose, which I wrote mostly before I had
this book, but it is not as well organized and complete at the http
protocol level.
Christopher Penrose
penrose@ucsd.edu
http://www-crca.ucsd.edu/TajMahal/after.html