robots retrieve documents, its indexers that index documents.
To check how the larger indexing sites have done their job just
do an experiment - use the same query (say "Ford") on each of the sites and
look at the first documents returned with a weighting of 1.00000 .
A simple way to do this is use the metacrawler at :
http://metacrawler.cs.washington.edu:8080