Hi - Tim Bray here, recently departed from a company called Open Text.
OT markets a spider for precisely such applications, loosely based on
the one my group wrote for the Open Text Index. It's quite cool. It
might be worth your while to talk to the product manager, who could refer
you to some people who are running *big* internal spiders (that we sold
'em) and about the general issues with respect to such products. His
name is Kevin Weatherston (kevin@opentext.com) but you won't be able to
talk to him without the go-ahead from the PR guy, Dave Paolini
(davep@opentext.com, 416 487 0543).
Cheers, Tim Bray