supplementary slides. more experimental results mphsm already push out many irrelevant images query...
TRANSCRIPT
Supplementary Slides
More experimental results MPHSM already push out many irrelevant
images
Query image
QHDM result, 4 of 36 ground truth found ANMRR=0.6464
MPHSM result, 9 of 36 ground truth foundANMRR=0.4819
More about experimental results Still some irrelevant image found
No spatial information Cannot identify background colors Does not account for unmatched colors Initial query might not be accurate
BlackBackground
GreenBackground
More about experimental results Can be improved by
Relevance Feedback Makes relevant images to have higher ranks
Irrelevant normally can’t have higher similarity by RF
But relevant images does Give more information about the interested objects Inconsistent backgrounds can be averaged out
More on experimental results Irrelevant images got lower rank / out of top
20 after RF
Query image
Ground truth images
Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043
First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688
Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688
B D
E F G
A
E
B D
A
B D
CA
E
More about experimental results Still some irrelevant images found
Some colors are very common (Blue sky, black night, green grass, etc.)
Different semantics might have similar color distribution
No single feature can do perfect retrieval
Can be improved by several approaches Choose suitable features Combining features
Suggestions on further developments For DCD
Use unmatched colors Challenge 1: Did the unmatched colors representing object
of interest? Or just a obstacle? Challenge 2: How to define the similarity function?
Separate foreground/background Challenge 1: Can we identify it by only using DCD? Or in
RF? Challenge 2: Or we need to combine other shape/texture
descriptors? The DCD generation is not very accurate
GLA generates an optimal for quantizing the image, it might not be accurate dominant colors.
Can quantize up to 16 or more colors, and then approximate the least significant colors to obtain an 8 color DCD
Suggestions on further developments For general CBIR
No single descriptor gives perfect retrieval Choosing suitable features Combining features (color+shape, color+texture, etc.)
Automatically? Manually? How to set weights?
Visual description about a CBIR System flow of a CBIR system
Online Process
Offline Process
Image DB Stored Features
Feature extractionUser initial input Results outputSimilarity measure
Similarity= 50%
Similarity= 100%
= 50%
= 30%...…
Feature extraction
…
… …
Color based CBIR approaches Three major approach of CBIR based on colors
Area of matching – Count the area with matched colors (CSD, SCD, DCD)
Color distance – Use color distance to adjust the similarity (DCD-QHDM)
Spatial distribution – Matches colors having similar layout (CLD)
Optional parameters Spatial coherence
obtained by a simple connected component analysis. A smooth surface gives a higher spatial coherence value.
Color variances computed as variances of the pixel values within each
cluster. But this parameter is for a dedicated similarity measure algorithm. So it is not commonly used.
Spatial coherency adjustment Similarity measure
MPEG-7 suggests to use a modified Quadratic Histogram Distance Measure (QHDM) to measure the dissimilarity between descriptors
Spatial coherency adjustment
1 2 1 22 2 2
1 2 1 2 1 ,2 1 21 1 1 1
( , ) 2 N N N N
i j i j i ji j i j
D F F p p a p p
1 2(0.3) (0.7) SD s s D D
Results with Spatial Coherence MPHSM improves DCD for both datasets to be more
close to other non-compact descriptors While using Corel_1k dataset MPHSM outperforms CLD
slightly MPHSM benefits from spatial coherency adjustment as
well as QHDM
Descriptor ANMRR (MPEG-7 CCD) ANMRR (Corel_1k)
DCD-MPHSM 0.2604 0.3946
DCD-MPHSM with SC 0.2400 0.3756
DCD-QHDM 0.2834 0.5648
DCD-QHDM with SC 0.2434 0.4958
DCD-QHDM upper bound problem Analysis of problem 1
The upper-bound of the distance measured varies by number of color in the descriptor
Maximum of positive part is not a constant Maximum of negative part is zero So, the maximum of QHDM is not fixed This property makes DCD unable to identify completely
different images by the values measured
Positive part Negative part
1 2 1 22 2 2
1 2 1 2 1 ,2 1 21 1 1 1
( , ) 2 N N N N
i j i j i ji j i j
D F F p p a p p
Upper bound problem - example Problem 1 – The upper bound problem
Consider the following images with their DCD I1, I2 are visually more similar than I1, I3 For a similarity measure that matches human
perception, we can expect the distance between F1, F2 should be smaller than that of between F1, F3
F1 F2
1/2
F3
1/2
1/3
I2 I3I1
Upper bound problem - example But distance between F1,F3 is smaller while
measuring their distance using QHDM The extra blue color pull down the distance
D2(F1,F2)>D2(F1,F3) implies that I1 is more similar to I3 than I2
This shows that QHDM does not meet human perception
21 2
21 3
( , ) 0.81389
( , ) 0.7093
D F F
D F F
DCD-QHDM Similarity coefficient problem The similarity coefficient use the color distance
to fine tune the similarity Difficult to define a quantitative similarity
between colors, since the sensitivity of human eye depends on many conditions (e.g. light source of the room, spatial layout of the image, etc.)
1 2 1 2
1 ,2
1 2
1 / ,
0,
i j d i j d
i j
i j d
c c T c c Ta
c c T
16.67% similar
44% similar
Td
0% similar
1.2
d
Similarity coefficient problem It is easy to count 50% of area is similar.
But it is difficult to count the colors are 50% similar.
This method is unable to consider the area of matching and the color distance together.
Similarity coefficient problem - example Problem 2 – The similarity coefficient a1i,2j
problem Consider the following images
I1, I2 are visually more similar than I1, I4 For a similarity measure that matches human perception,
we can expect the distance between F1, F2 should be smaller than that of between F1, F4
F4
1
I4
F1 F2
1/2 1/2
I2I1
Similarity coefficient problem - example But distance between F1,F4 is smaller while measuring
their distance using QHDM One exactly matched color considered more important
than a whole area of similar color
D2(F1,F2)>D2(F1,F4) implies that I1 is more similar to I4 than I2
But in natural perception, images having similar color distribution is more likely to have similar semantics
This shows that QHDM does not meet human perception again
21 2
21 4
( , ) 0.81389
( , ) 0.5
D F F
D F F
Flow of MPHSM
Initial DCDs
Find a pair of colors with minimum
distance d
d<Td ?Merge colors havingminimum distance
Common Palette
N
Y
Update each DCD basedon the common palette
Histogram Intersection
Palette Merging process, visually Example
Two images with DCD, palette merging stage
Dominant Color Descriptor
Find the closest pair
Merge colors
CommonPaletteMerge colorsMerge colorsMerge colorsRemaining colors
If a remaining color is similar toany colors in the common palette. Itwill not included in common palette
About slide 23 Relationship between CBIR and Relevance
Feedback (RF) The key component is query update
Input Query
FeatureExtraction
SimilarityMeasure
RetrievalResult
User’sFeedback
QueryUpdate
Find allImages?
y
n
FinalRetrieval
Result
Image Retrieval
Relevance Feedback
Input Query
FeatureExtraction
SimilarityMeasure
RetrievalResult
User’sFeedback
QueryUpdate
Find allImages?
y
n
FinalRetrieval
Result
Image Retrieval
Relevance Feedback
MPH-RF flow
Load add DCDs
Append all DCD
Find closes pair of colors
Minimum distance
< Td ?
Merge colorsand percentages
A
A
Cut least significantcolors
Adjust histogramsum into 1
Updated query
Y
N
RF of other MPEG-7 visual descriptors Relevance feedback for MPEG-7 descriptors
Apart from the MPH-RF for DCD, we directly apply feature weighting technique on several MPEG-7 visual descriptors
RF on CLD:
RF on CSD:
' ( )N
j ij j ij j iji j
Q w DY w DCb w DCr
'N
j iji j
Q w h
RF of other MPEG-7 visual descriptors RF on SCD:
'N
j iji j
Q w coeff
MIRROR – A CBIR system using MPEG-7 visual descriptors A set of visual descriptors Relevance feedback functions is added Evaluation tools MIRROR is also a development platform of
MPEG-7 visual descriptors
Performance of color descriptors Performance of color descriptors
Evaluation tools Unmodified MPEG-7 reference software XM MPEG-7 Common Color Dataset (MPEG-7 CCD) with
5466 images and 50 sample queries Corel 1000 images dataset with 20 sample queries ANMRR performance metric (smaller means better)
MPEG-7 CCD Corel 1000 images
DescriptorANMR
R
Descriptorsize (bytes
)
Descriptor size per
image (bytes)ANMRR
Descriptorsize (bytes)
Descriptor size per
image (bytes)
DCD 0.2834 86,86915.8926
(in average)0.5468 18,538
18.538 (in average)
CLD 0.2252 43,728 8 0.4000 8,000 8
CSD 0.0399 1,401,346 256.375 0.3246 256,375 256.375
SCD 0.1645 119,569 21.875 0.3552 119,569 21.875
Performance of color descriptors Investigation of performances
Color structure descriptor performs best among color descriptors due to its large descriptor size
Dominant color descriptor performs worst, even worse than a more compact color layout descriptor
“Area of matching” is still the most efficient approach for color based CBIR
New methods will be proposed in this research to boost DCD MPEG-7 CCD Corel 1000 images
DescriptorANMR
R
Descriptorsize (bytes
)
Descriptor size per
image (bytes)ANMRR
Descriptorsize (bytes)
Descriptor size per
image (bytes)
DCD 0.2834 86,86915.8926
(in average)0.5468 18,538
18.538 (in average)
CLD 0.2252 43,728 8 0.4000 8,000 8
CSD 0.0399 1,401,346 256.375 0.3246 256,375 256.375
SCD 0.1645 119,569 21.875 0.3552 119,569 21.875
Complete results MPHSM improves DCD for both datasets to be more
close to other non-compact descriptors While using Corel_1k dataset MPHSM outperforms CLD
slightly MPHSM benefits from spatial coherency adjustment as
well as QHDM
Descriptor ANMRR (MPEG-7 CCD) ANMRR (Corel_1k)
DCD-MPHSM 0.2604 0.3946
DCD-MPHSM with SC 0.2400 0.3756
DCD-QHDM 0.2834 0.5648
DCD-QHDM with SC 0.2434 0.4958
CLD 0.2252 0.4000
CSD 0.0399 0.3246
SCD 0.1645 0.3552
Complete results MPR-RF gives significant improvement on all
combinations of similarity measures and datasets. By using MPH-RF DCD can perform as good as another
compact descriptor CLD, and very close to a lesser compact descriptor SCD.
Three iterations of relevance feedback give a significant result
Descriptor
MPEG-7 CCD Corel_1k
Before RF After 3 RFRF
ImprovementBefore RF After 3 RF
RF Improvement
DCD-MPHSM
0.2604 0.1752 0.0852 0.3946 0.3298 0.0648
DCD-QHDM 0.2834 0.2117 0.0717 0.5468 0.4900 0.0568
CLD 0.2252 0.4000
CSD 0.0399 0.3246
SCD 0.1645 0.3552
Complete results The MPH-RF improvement on DCD is more significant
than feature weighting for other color descriptors
Color structure descriptor gives impressive results among all color descriptor, and its only drawback is the descriptor size is too large.
Descriptor
MPEG-7 CCD Corel_1k
Before RF After 3 RFRF
ImprovementBefore RF After 3 RF
RF Improvement
DCD-MPHSM
0.2604 0.1752 0.0852 0.3946 0.3298 0.0648
DCD-QHDM
0.2834 0.2117 0.0717 0.5468 0.4900 0.0568
CLD 0.2252 0.1814 0.0438 0.4000 0.3571 0.0429
CSD 0.0399 0.0115 0.0284 0.3246 0.2366 0.0880
SCD 0.1645 0.1019 0.0626 0.3552 0.3276 0.0276