matthew simpson, md mahmudur rahman, dina demner-fushman, sameer antani, george r. thoma
DESCRIPTION
Text- and Content-based Approaches to Image Retrieval for the ImageCLEF 2009 Medical Retrieval Track. Matthew Simpson, Md Mahmudur Rahman, Dina Demner-Fushman, Sameer Antani, George R. Thoma Lister Hill National Center for Biomedical Communications, - PowerPoint PPT PresentationTRANSCRIPT
Text- and Content-based Approaches to Image
Retrieval for the ImageCLEF2009 Medical Retrieval Track
Matthew Simpson, Md Mahmudur Rahman, Dina Demner-Fushman, Sameer Antani, George R. Thoma
Lister Hill National Center for Biomedical Communications, National Library of Medicine, NIH, Bethesda, MD, USA
CLEF 2009
Retrieval tasks and approaches
• ITI project long term goal– Find a way to combine image and text features so
that the whole is greater than the sum of its parts
• Ad-hoc image retrieval– Text-based– Image content-based– Automatic mixed– Relevance feedback mixed
• Case-based document retrieval– Text-based
Text-based approach
• Indexing:– Create image documents for ad-hoc image
retrieval– Create surrogate documents for case-based
retrieval– Index using Essie
• term normalization using the SPECIALIST Lexicon• query expansion based on UMLS synonymy• term weighting based on location in the document• Phrase-based search
Text documents
• Image document– Title and caption provided by organizers– Mention extracted from paper– MEDLINE citation (abstract +MeSH)– PICO frame of the caption + image modality
(structured caption summary)
• Surrogate document– MEDLINE citation – caption, mention, and structured caption summary of
each image contained in the article
Text retrieval
• PICO-based structured query and case representation– <topicID>19</topicID> <description>Crohn's disease CT</description>
– <modality essieExp="false">ct</modality> <modSyn>c.a.t.</modSyn><modSyn>cat</modSyn><modSyn>computerised axial tomography</modSyn>….
– <cond essieExp="true">Crohn's disease</cond><condPN>crohn disease</condPN><condSyn>Regional enteritis</condSyn> <condSyn>eleocolitis</condSyn><condSyn>Cicatrizing enterocolitis</condSyn><condSyn>granulomatous enteritis</condSyn><condSyn>INFLAMMATORY BOWEL DISEASE</condSyn><condSyn>regional enterocolitis</condSyn> …
CBIR - Image feature representation
• Concepts - color and texture patches from local image regions
• Low-level global features– Color (Color Layout Descriptor, MPEG-7)– Edge (histogram of local edge distribution and
direction)– Texture (grey level co-occurrence matrix)– Average grey level (256-dimensional vector of blocks
in image normalized to gray-level 64x64)– Lucene (LIRE)-based Color Edge Direction Descriptor
and Fuzzy Color Texture Histogram
Image similarity computation
• Category-specific– Determine image category (training set of
5000 images manually assigned to 32 mutually exclusive categories)
– Use category-specific weights in linear similarity matching
• Relevance feedback– Feature weights updated using images judged
relevant
Combining text and image
• Based on text search results,– Compute mean vector of top 5 retrieved images, use
as input to category-specific retrieval– Select 3-5 relevant images manually, use as input to
category-specific retrieval– Re-rank text retrieval results using visual retrieval
scores
• Provide feedback using all retrieval results, – expand query using image documents– Pad selected relevant images with new retrieval
results
Relevance Feedback
Results
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
MAP P@5 Recall
visual
category-specific
RF text
mixed
re-ranked
case-basedBRF RF RF+QE
Image-text search engine
Thank you!
Questions?