25/10/20151gianluca demartini desktop search evaluation sergey chernov and gianluca demartini trec...

20/04/23 1Gianluca Demartini

Desktop Search Evaluation

Sergey Chernov and Gianluca Demartini

TREC 2006, 16th November 2006

Pre-Track Workshop

20/04/23 2Gianluca Demartini

Outline Why we need a Desktop Track?

What are the settings?

Does it solve THE Privacy Problem?

What we do next?

Microsoft

Copernicus Beagle

Google Yahoo

And roughly 20 more…

How we compare their performance?

Proposed Track Building the DataSet

Personal documents Include activity logs containing the history of each

file, query logs, email and clipboard usage, instant messenger history, ...

Activity logs and metadaat should substitute missing hyperlink structure on a desktop

Main problem

Privacy Issue

Track will not run in 2007

Can be proposed for 2008

Questions 1 how to build the collection? (desktops from

participants?) 2 how to protect privacy? 3 Data? (text docs, mails, pics, audio) 4 Tasks? 5 Topics? 6 Evaluation measures? binary or multi-graded

relevance? 7 Logged information? Logged applications?

“Permanent” Information to Log

Permanent Information (Applied to)

URL (HTML)

Author (All files)

Recipients (Email messages)

Metadata tags (MP3)

Has/is attachment (Emails and attachments)

Saved picture's URL and saving time (Graphic files)

“Timeline” Information to Log

• Timeline information (Applied to) • Time of being in focus ( All files)• Time of being opened ( All files)• Being edited ( All files)• History of moving/renaming ( All files)• Request type: bookmark, clicked link, typed URL ( HTML)• Adding/editing an entry in calendar and tasks (Outlook Journal)• Being printed (All files)• Search queries in Google/MSN Search/Yahoo!/etc. (Browser search field)• Clicked links (HTML)• Text selections from the clipboard Text pieces within a file and the filename

(Text files)• Bookmarking time (Browser bookmarks)• Instant Messenger status, contact's statuses, sent filenames and links (IM

History)• Running applications (Task queue)• IP address User's address and addresses user connects to• Email status Change between received/read (email client)

Data Gathering

Data is not publicly avalilable

Data format is known

Retrieval Systems can be run on the data by track coordinator and results are sent back (See Spam Track)

Collection Structure

Text Documents, EMails and Instant Messages – yes Images - ??? Audio – only metadata would be extracted Video - no What else?

Proposed Tasks

AdHoc Retrieval Task Find several documents

containing pieces of necessary information

Known-Item Retrieval Task find a single specific

document

Folder Retrieval Task Find the folders with

the relevant information

Topic Format

title Eleonet project deliverable June

metadata date:June topic:Eleonet project type:deliverable

task description I am combining a new deliverable for the Eleonet project.

narrative I am combining a new deliverable for the Eleonet project and

I am looking for the last deliverable of the same type. I remember that the main contribution to this document has been done in June 2006.

Relevance & Evaluation Measures trec_eval to a set of common metrics Binary relevance assessments or 3 levels? Ranking is important:

MAP Gain & Discount Metrics (DCG, nDCG, AWP, AGR, Q-

m) Uncomplete assessments:

Bpref (/Rpref)

Logged Applications

Acrobat Reader

MS Word

MS Excel

MS Powerpoint

MS Internet Explorer

MS Outlook

Mozilla Firefox

Mozilla Thunderbird

The same questions again

1 how to build the collection? (desktops from participants?)

2 how to protect privacy?

3 Data? (text docs, mails, pics, audio)

4 Tasks? 5 Topics? 6 Evaluation measures? 7 Logged information?

Logged applications?

Desktop Search Workshop Summary

• Strong interest – about 20 participants

• Main novelty – activity logs

• Privacy is still an issue

• We need a clear task definition (suggestion: “Find all documents related to a project”?)

• We are planning a workshop to discuss it further

• A mailing list is available – to subscribe visit https://info.l3s.uni-hannover.de/mailman/listinfo/personal-activity-search