Measure-based Metasearch
Every measure of performance essentially dictates a prior (or importance) over various ranks. For example, Precision-at-cutoff 10 dictates uniform values over the first 10 ranks and null values for all other ranks.
We used these measure-imposed ranking values as documents scored within each systems and then average them across systems, per document to obtain metasearch scores.
metasearch
Hedge Metasearch
Hedge is a well known algorithm for online allocation. We applied it for metasearch in the following way:
- within each system, each document gets a ranking value, based on a prior.
- initially all underlying systems are given uniform weights
- each document has a computed score as a average ranking values weighted by systems weights.
- top scored document is selected and judged; systems weights are updated
Last two steps are repeated in a loop as long as judging can be afforded. When stopped, the documents scored are used to compute the metasearch list.
The table on the left shows the metasearch performance with no judgments (“Hedge 0”), that is document scores being the average of ranking values. The results are on par with most efficient metasearch previously known.
Metasearch Problem. Combine multiple ranked lists (obtained running the same query over the same collection, but different search engines) into one master list.
