DO BATCH AND USER EVALUATİONS GİVE THE SAME RESULTS?

William Hersh, Andrew Turpin, Susan Price, Benjamin Chan, Dale Kraemer, Lynetta Sacherek, Daniel Olson

Hande AdıgüzelHayrettin Erdem

BATCH EXPERİMENTS VS TREC INTERACTİVE TRACK

Measuring recall and precision in the noninteractive laboratory setting

Interaction is the key element of successful retrieval system use, and relevance-based measures do not capture the complete picture of user performance .

The TREC Interactive Track instructs human users to tag the relevant documents for different topics.  

These results are used for independent relevance judgment. 2

CONTRİBUTİON OF THE PAPER

To find whether IR approaches achieving better performance in the batch environment could translate that effectiveness to real users.

3

EXPERİMENT STEPS1. Establishment of the best weighting

approach for batch searching experiments.

2. User experiments to determine if those measures give comparable results with human searchers.

3. Verification that the new TREC interactive track data gives comparable batch searching results for the chosen weighting schemes.

4

Finding an effective weighting scheme for experimental system using TREC 6-7

5

Interactive searching to assess weighting scheme with real users

6

Not statistically significant.

Interactive searching to assess weighting scheme with real users

7

All of the difference between the systems occurred in just one query, 414i.

VERİFYİNG WEİGHTİNG SCHEME WİTH TREC 8

This experiment is to verify that the improvements in batch evaluation detected with TREC-6 and TREC-7 data held with TREC-8 data.

8