>Are robots for sale or can one "hire" someone who has one to do some
>work, or how does that whole thing work.
Verity offers a couple of variations of its Web robot, but they are
designed specifically to build Verity search indexes, not as
general-purpose robots. The only generally available robot-ish code that I
know about is the Harvest Gatherer code. Its primary purpose is to index
the server on which is it running, but it's a fairly small step to make it
do the same over the wire.
I think there's a widespread reluctance to push robots hard in the
commercial space, since marketing success would fairly quickly breed
failure -- having lots of robots doing redundant work would be a huge
inefficiency.
Nick