integrating user feedback log into relevance feedback by coupled svm for content-based image...
DESCRIPTION
Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval . 9-April, 2005 Steven C. H. Hoi * , Michael R. Lyu * , Rong Jin # * Department of Computer Science & Engineering The Chinese University of Hong Kong Shatin, N.T., Hong Kong SAR - PowerPoint PPT PresentationTRANSCRIPT
1
Integrating User Feedback Log into Relevance Feedback by Coupled SVM for
Content-Based Image Retrieval
9-April, 2005
Steven C. H. Hoi * , Michael R. Lyu *, Rong Jin #
*Department of Computer Science & EngineeringThe Chinese University of Hong Kong
Shatin, N.T., Hong Kong SAR
# Department of Computer Science and EngineeringMichigan State University
East Lansing, MI 48824, USA
The 1st IEEE EMMA Workshop in conjunction with 21st IEEE ICDE, Japan, April, 2005.
2
Outline• Introduction• Background
Log-based Relevance Feedback• Coupled Support Vector Machine
Support Vector Machine Formulation Alternating Optimization A Practical Algorithm
• Experimental Results• Conclusion
3
Introduction• Content-based Image Retrieval (CBIR)
An important component in visual information retrieval QBE: query-by-example based on low-level visual
features Semantic gap: low-level features, high-level concepts
QBE
4
Introduction• Relevance Feedback (RF)
A powerful tool to attack the semantic gap problem Interactive mechanism to solicit users’ feedbacks Boost the retrieval performance of CBIR greatly Many existing techniques already…
• Problems Regular relevance feedback needs too many rounds of i
nteractions for achieving satisfactory results.
5
Introduction• Motivation
Can user feedback log be used to improve the Can user feedback log be used to improve the regular relevance feedback?regular relevance feedback?
Relevance Feedback
User Feedback Log
?
Problem
6
Background• Log-based Relevance Feedback (LRF)
Relevance Matrix: R RF round / Log session: Nl images are marked Elements: relevant (1), irrelevant (-1), unknown (0)
Log Sessions
Image samples
1 -1 1 -1 -1 0 1 -1 -1 11
-1 1 -1 -1 -1 -1 -1 1 -1-10
7
Background• Learning Problem for LRF
Low-level image content:
User feedback log:
Multi-Modal Learning Problem
},,,{ 21 NxxxX
},,,{ 21 NrrrR
8
Coupled Support Vector Machine• Motivation
How to attack the learning problem on the two modalities?• Low-level Image content: X• User relevance feedback log: R
Support Vector Machines: superior classification performance
• A Straightforward Solution: Learn an SVM classifier on each modality respectively
• For image content X, we learn an optimal weighting vector w;• For log content R, we learn an optimal weighting vector u;
Combine their results together linearly
9
• A Straightforward Solution For the image content modality: wTx
For the user feedback log modality: uTr
Coupled Support Vector Machine
10
• Disadvantages of the straightforward solution Linear combination Modality Consistence
• Our better solution: Coupled SVM Learn the two modalities in a unified formulation Enforce the prediction on the two types of information
to be consistent.
Coupled Support Vector Machine
11
• Formulation: Coupled SVM
Coupled Support Vector Machine
12
• Optimization of Coupled SVM Hard to be solved directly Alternating Optimization (AO)
• AO: two-step optimization Fix Y’, try to find (u, b_u), and (w, b_w) Fix (u, b_u) and (w, b_w), try to find Y’
Coupled Support Vector Machine
13
• Alternating Optimization Fix Y’, the primal optimization is equivalent to solving
the two optimization subproblems:
Coupled Support Vector Machine
14
• Alternating Optimization (AO) By introducing non-negative Lagrange multipliers, the
above two subproblems can be solved
Coupled Support Vector Machine
15
• Alternating Optimization (AO) After solving (u, b_u) and (w, b_w), fixing them, the op
timal Y’ can be found to fit the data as follows:
Coupled Support Vector Machine
16
• Summary of AO procedure 1) Beginning with a small value of 2) Performing the two-step AO procedure 3) Repeating 2) by increasing until it achieves the setting
threshold
• Comments on the Coupled SVM Can be a general approach for multi-modal learning problems Need to investigate the convergence issue of Alternating
Optimization Need to study better methods for solving the optimization
problem Require to take some practical considerations when fitting for
specific problems.
Coupled Support Vector Machine
17
• A Practical Algorithm Practical considerations
• Cannot engage all unlabeled samples due to response requirement for relevance feedback
• Strategy for choosing unlabeled samples– Closest to the decision boundary of SVM: most informative
according to active learning– Closest to the labeled samples: to avoid too much effort in
learning the label information• Introducing a parameter to control the error for label
correction to avoid overlarge change in the labeled set
Coupled Support Vector Machine
18
• A Practical Algorithm (cont’d)
Coupled Support Vector Machine
19
• A Practical Algorithm (cont’d)Coupled Support Vector Machine
20
Experimental Results• Dataset
Images selected from COREL image CDs Two ground-truth datasets
• 20-Category: each category contains 100 images, totally 2,000• 50-Category: each category contains 100 images, totally 5,000
21
Experimental Results (cont’d)• Low-level Image Representation
Color Moment • 9-dimension
Edge Direction Histogram • 18-dimension• Canny detector, 18 bins of 20 degrees each
Wavelet-based texture • 9-dimension• Daubechies-4 wavelet, 3-level DWT• Entropies of 9 subimages are generated for the texture feature
22
Experimental Results (cont’d)• Collection of User Log Data
Log format• A log session (LS) corresponds a relevance feedback round• Each log session contains 20 images labeled by users
Log data• On 20-Category: 161 log sessions• On 50-Category: 184 log sessions
23
Experimental Results (cont’d)• CBIR GUI for collecting feedback data
24
Experimental Results (cont’d)• Performance Evaluation
Measurement Metric• Average Precision = # relevant images / # returned images
Experimental Setting• 100 queries• 20 initially labeled images• SVM: RBF kernel, parameters set via training data
Comparison Schemes• RF-SVM
– traditional relevance feedback by SVM• LRF-2SVM
– log-based relevance feedback by learning two SVMs respectively• LRF-CSVM
– log-based relevance feedback by Coupled SVM
25
Experimental Results (cont’d)• Performance Evaluation: on 20-Category Dataset
26
Experimental Results (cont’d)• Performance Evaluation: on 50-Category Dataset
27
Experimental Results (cont’d)
28
Experimental Results (cont’d)
29
Conclusion• A log-based relevance feedback scheme was studied by
integrating user feedback log into the content learning of low-level visual features in content-based image retrieval.
• A general multimodal learning technique, i.e. Coupled Support Vector Machine, was proposed for studying the data with multiple modalities.
• A practical algorithm by Coupled SVM was presented to attack the log-based relevance feedback problem in CBIR.
• Experimental results show our proposed scheme is effective for the log-based relevance feedback problem.
30
Q&A
31
References• Chu-Hong Hoi and Michael R. Lyu, A Novel Log-based Relevance Fe
edback Technique in Content-based Image Retrieval, in Proc. ACM Multimedia, New York, USA, 10-16 October, pp. 24-31, 2004
• S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proc. ACM Multimedia, pages 107--118, 2001.