if you're looking for info, go to the established search engines, eg,
yahoo, alta vista. why do the work yourself? it's been done!
i think most robots these days should be metabots that feed on the output
of webcrawlers and spiders like yahoo. work smarter, not harder. and dont
fill up your disk with stuff that's already indexed by someone else!
my three ducats.
-john