the clef 2005 cross-language image retrieval track

The CLEF 2005 Cross-Language Image Retrieval Track

Organised by

Paul Clough, Henning Müller, Thomas Deselaers, Michael Grubinger, Thomas Lehmann, Jeffery Jensen and William Hersh

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005

Overview

• Image Retrieval and CLEF

• Motivations

• Tasks in 2005• Ad-hoc retrieval of historic photographs and medical

images

• Automatic annotation of medical images

• Interactive task

• Summary and future work


Image Retrieval and CLEF

• Cross-language image retrieval• Images often accompanied by text (used for retrieval)

• Began in 2003 as pilot experiment

• Aims of ImageCLEF• Investigate retrieval combining visual features and

associated text

• Promote the exchange of ideas

• Provide resources for IR evaluation


Motivations

• Image retrieval a good application for CLIR• Assume images are language-independent

• Many images have associated text (e.g. captions, metadata, Web page links)

• CLIR has potential benefits for image vendors and users

• Image retrieval can be performed using• Low-level visual features (e.g. texture, colour and shape)

• Abstracted features expressed using text

• Combining both visual and textual approaches


ImageCLEF 2005• 24 participants from 11 countries

• Specific domains and tasks• Retrieval of historic photographs (St Andrews)

• Retrieval and annotation of medical images (medImageCLEF and IRMA)

• Additional co-ordinators• William Hersh and Jeffrey Jensen (OHSU)

• Thomas Lehmann and Thomas Deselaers (Aachen)

• Michael Grubinger (Melbourne)

• Links with MUSCLE NoE including pre-CLEF workshop• http://muscle.prip.tuwien.ac.at/workshops.php


Ad-hoc retrieval from historic photographs

Paul Clough (University of Sheffield)

Michael Grubinger (Victoria University)


From: St Andrews Library historic photographic collectionhttp://specialcollections.st-and.ac.uk/photo/controller

イングランドにある灯台の写真

Изображения английских маяков

Fotos de faros ingleses

Pictures of English lighthouses

Kuvia englantilaisista majakoista

Bilder von englischen Leuchttürmen

انجليزيه لمنارات صور

St Andrewsimage collection

http://specialcollections.st-and.ac.uk/photo/controller


Topics• 28 search tasks (topics)

• Consist of title, narrative and example images

• Topics more general than 2004 and more “visual” • e.g. waves breaking on beach, dog in sitting position

• Topics translated by native speakers• 8 languages for title & narrative (e.g. German, Spanish, Chinese,

Japanese)• 25 languages for title (e.g. Russian, Bulgarian, Norwegian,

Hebrew, Croatian)

• 2004 topics and qrels used as training data


Relevance judgements

• Staff from Sheffield University were assessors

• Assessors judged topic pools• Top 50 images from all 349 runs

• Average of 1,376 images per pool

• 3 assessments per image (inc. topic creator)

• Ternary relevance judgements

• Qrels: images judged as relevant/partially relevant by topic creator and at least one other assessor


Submissions & Results (1)

11 groups (5 new*)CEA*NII*AlicanteCUHK*DCUGenevaIndonesia*MiracleNTUJaen*UNED

Dimension Type #Runs (%) Avg. MAP

Language English 119 (34%) 0.2084

Non-English 230 (66%) 0.2009

Run type Automatic 349 (100%) 0.2399

Feedback (QE) Yes 142 (41%) 0.2399

No 207 (59%) 0.2043

Modality Image 4 (1%) 0.1500

Text 318 (91%) 0.2121

Text + Image 27 (8%) 0.3086

Initial Query Image 4 (1%) 0.1418

Title 274 (79%) 0.2140

Narrative 6 (2%) 0.1313

Title + Narr 57 (16%) 0.2314

Title + Image 4 (1%) 0.4016

All 4 (1%) 0.3953


Submissions & Results (2)Language #Runs Max. MAP Group Initial

QueryFeedback Modality

English 70 0.4135 CUHK Title+img Yes Text+img

Chinese (trad.) 8 0.3993 NTU Title+narr Yes Text+img

Spanish (Lat. Am.)

36 0.3447 Alicante/Jaen

Title Yes Text

Dutch 15 0.3435 Alicante/Jaen

Title Yes Text

Visual 3 0.3425 NTU Visual Yes Image

German 29 0.3375 Alicante/Jaen

Title Yes Text

Spanish (Euro.) 28 0.3175 UNED Title Yes Text

Portuguese 12 0.3073 Miracle Title No Text

Greek 9 0.3024 DCU Title Yes Text

French 17 0.2864 Jaen Title+narr Yes Text

Japanese 16 0.2811 Alicante Title Yes Text

Russian 15 0.2798 DCU Title Yes Text

Italian 19 0.2468 Miracle Title Yes Text


Submissions & Results (3)Language #Runs Max. MAP Group Initial

QueryFeedback Modality

Chinese (simpl.) 21 0.2305 Alicante Title Yes Text

Indonesian 9 0.2290 Indonesia Title No Text+img

Turkish 5 0.2225 Miracle Title No Text

Swedish 7 0.2074 Jaen Title No Text

Norwegian 5 0.1610 Miracle Title No Text

Filipino 5 0.1486 Miracle Title No Text

Polish 5 0.1558 Miracle Title No Text

Romanian 5 0.1429 Miracle Title No Text

Bulgarian 2 0.1293 Miracle Title No Text

Czech 2 0.1219 Miracle Title No Text

Croatian 2 0.1187 Miracle Title No Text

Finnish 2 0.1114 Miracle Title No Text

Hungarian 2 0.0968 Miracle Title no Text


Summary

• Most groups focused on text retrieval• Fewer combined runs than 2004

But still gives highest average MAP

• Translation main focus for many groups13 languages have at least 2 groups

• More use of title & narrative than 2004• As Relevance feedback (QE) improves results

• Topics still dominated by semantics• But typical of searches in this domain


Ad-hoc medical retrieval task

Henning Müller (University Hospitals Geneva)

William Hersh, Jeffrey Jensen (OHSU)


Collection

• 50,000 medical images• 4 sub-collections with heterogeneous annotation

• Radiographs, photographs, Powerpoint slides and illustrations

• Mixed languages for annotations (French, German and English)

• In 2004 only 9,000 images available


Search topics• Topics based on 4 axes

• Modality (e.g. x-ray, CT, MRI)• Anatomic region shown in image (e.g. head, arm)• Pathology (disease) shown in image• Abnormal visual observation (e.g. enlarged heart)

• Different types of topic identified from survey• Visual (11) – visual approaches only expected to perform well• Mixed (11) – text and visual approaches expected to perform well• Semantic (3) – visual approaches not expected to perform well

• Topics consist of annotation in 3 languages and 1-3 query images


An example (topic # 20 - mixed)

Show me microscopic pathologies of cases with chronic myelogenous leukemia.

Zeige mir mikroskopische Pathologiebilder von chronischer Leukämie.

Montre-moi des images de la leucémie chronique myélogène.


Relevance assessments

• Medical doctors made relevance judgements• Only one per topic for money and time constraints

Some additional to verify consistency

• Relevant/partially relevant/non relevantFor ranking only relevant vs. non-relevant

• Image pools created from submissions• Top 40 images from 134 runs

• Average of 892 images per topic to assess


Submissions

• 13 groups submitted runs (24 registered)• Resources very interesting but lack of manpower

• 134 runs submitted

• Several categories for submissions• Manual vs. Automatic

• Data source usedVisual/textual/mixedAll languages could be used or a single one


Results (1)• Mainly automatic and mixed

submissions• some further to be

classified as manual

• Large variety of text/visual retrieval approaches• Ontology-based

• Simple tf/idf weighting

• Manual classification before visual retrieval

Query types Automatic Manual

Visual 28 3

Textual 14 1

Mixed 86 2


Results (2) – highest MAP

Query types Automatic Manual

Visual I2Rfus.txt

0.146

i2r-vk-avg.txt

0.092

Textual IPALI2R_Tn

0.208

OHSUmanual.txt

0.212

Mixed IPALI2R_TIan

0.282

OHSUmanvis.txt

0.160


Average results per topic type

0.0000

0.1000

0.2000

0.3000

0.4000

0.5000

0.6000

0.7000

Avera

ge

Visual

Mixe

d

Seman

tic

Weight

ed

P10Avg

Topic Type

MA

P

Auto-Mixed

Auto-Text

Auto-Vis

Man-Mixed

Man-Text

Man-Vis


Summary

• Text-only approaches perform better than image-only• But some visual systems have high early precision• Depends on the topics formulated

Visual systems very bad on semantic queries

• Best overall systems use combined approaches• GIFT as a baseline system used by many participants

and still best visual completely automatic

• Few manual runs


Automatic Annotation Task

Thomas Deselaers, Thomas Lehmann

(RWTH Aachen University)


Automatic annotation

• Goal• Compare state-of-the-art classifiers for medical image

annotation task

• Purely visual task

• Task• 9,000 training & 1,000 test medical images from Aachen

University Hospital

• 57 classes identifying modality, body orientation, body region and biological system (IRMA code)

e.g. 01: plain radiography, coronal, cranuim, musculosceletal system

• Classes in English and German and unevenly distributed


Example of IRMA code

• Example: 1121-127-720-500Example: 1121-127-720-500• rradiography, plain, analog, overviewadiography, plain, analog, overview• ccoronal, AP, supineoronal, AP, supine• aabdomen, middlebdomen, middle• uuropoetic systemropoetic system


Example Images

http://irma-project.org


Participants

• Groups• 26 registered

• 12 submitted runs

• Runs• In total 41 submitted

• CEA (France)

• CINDI (Montreal,CA)

• medGift (Geneva, CH)

• Infocomm (Singapore, SG)

• Miracle (Madrid, ES)

• Umontreal (Montreal, CA)

• Mt. Holyoke College (Mt. Hol., US)

• NCTU-DBLAB (TW)

• NTU (TW)

• RWTH Aachen CS (Aachen, DE)

• IRMA Group (Aachen, DE)

• U Liège (Liège, BE)


Results

...

...• Baseline error rate: 36.8%


Conclusions

• Continued global participation from variety of research communities

• Improvements in ad-hoc medical task• Realistic topics

• Larger medical image collection

• Introduction of medical annotation task

• Overall combining text and visual approaches works well for ad-hoc task


ImageCLEF2006 and beyond …


ImageCLEF 2006 …

• New ad-hoc• IAPR collection of 25,000 personal photographs

• Annotations in English, German and Spanish

• Medical ad-hoc• Same data; new topics

• Medical annotation• Larger collection; more fine-grained classification

• New interactive task• Using Flickr.com … more in iCLEF talk


… and beyond

• Image annotation task• Annotate general images with simple concepts

• Using the LTU 80,000 Web images (~350 categories)

• MUSCLE collaboration• Create visual queries for ad-hoc task (IAPR)

• Funding workshop in 2006

• All tasks involve cross-language in some way

the clef 2005 cross-language image retrieval track

Documents

usersimage retrieval

future workimage retrieval

clirassume images

title narr57

title image4

image vendors

text image27

usinglowlevel visual