RE: How can IR Agents be evaluate ?

Nick Arnett (narnett@verity.com)
Sat, 17 Aug 1996 14:32:23 -0700


At 2:56 PM 8/10/96, Dan Quigley wrote:
>I have not heard or read of any evaluation "standard" for retrieval
>systems other than those typically used to benchmark database engines.
>I believe that any fair comparison of the current genre of self-indexed
>IR engines could only occur at fundamental data processing levels.

There is a comparison, TREC, sponsored by NIST periodically, in which
search engines are evaluated. However, companies tend to drop in and out
of it based on various factors (No. 1 being how busy they are with other
things).

However, it's important to bear in mind that any search evaluation is going
to be quite subjective, since in the end, whether or not a given query
should find a given document is a matter of opinion. Anything that deals
at this level with language is inherently fuzzy and subjective. Simple
speed comparisons are worse than useless, they're deceptive because a less
accurate search is nearly always faster, given equally good code.

We've been developing some in-house tools to make comparisons more
objective for our own QA purposes, but they are naturally limited by the
factors I've described.

Nick