competence centre on information extraction and image understanding for earth observation 29th march...

29th March 2007 Category - based Semantic Search Engine 1

Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Category - based Semantic Search Engine

Mihai CostacheTélécom Paris


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Outline

General problem description

Relevance Feedback in Satellite CBIR

SVM based search engine

Results

Discussions


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Problem statement

Context

Huge volume of Earth Observation (EO) images High volume of information due to the increasing resolution (0.6 – 2.5 metesr/pixel)

Need to perform

Automatic indexation and annotation of satellite image archives (Content Based Image Retrieval, Relevance Feedback)

Detection and recognition of structures and objects within the satellite images (classification)

Reasearch areas

Information theory and statistics Machine learning and Bayesian Inference – Incremental learning Discriminative learning


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Satellite Content Based Image Retrieval System

CBIR attempts to automate the indexing process of images in the database

Descriptions based on inherent properties of images (texture, color, shape etc.) Problems:

Gap between low-level descriptors and semantic image content

Different image semantic interpretations:

• Same image but two persons

• Same person but different moments of time

Thus small retrieval precision

Solution: incorporating human perception subjectivity

Relevance Feedback


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

RF in SCBIR Systems

Relevance Feedback

Finding relevant information in satellite CBIR by human interaction

The retrieval process is improved by human decision

The user scores the retrieved documents as relevant / irrelevant

Annotations used in the learning process

Structure of a satellite indexation system with RF

Primitive feature extraction Classification Relevance Feedback (RF)

Quadrature Mirror Filters (QMF) SVM RF-SVM

Objective Subjective


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Structure of a RF system

Human-machine communication in RF

Subjective Objective


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

RF Algorithms

Query point movement - Try to find an optimal vector model for retrieval purposes

Rocchio’s AlgorithmJ. J. Rocchio, Relevance feedback information retrieval, 1971.

Viper, MARSD. McG. Squire et al., Content-based query of image databases: inspirations from text retrieval,

2000.Y. Ruiu et al., Content-based Image Retrieval with Relevance Feedback in MARS, 1997.

Re-weighting algorithms - Try to find an optimal similarity measure

MindReaderY. Ishikawa et al., Mindreader: Query Databases Through Multiple Examples,1998.

RF – Support Vector Machine (SVM)

S. Thong and Chang E., Support vector Machine Active Learning for Image Retrieval, 2001.M. Costache, H. Maitre and M. Datcu,Categorization based Relevance Feedback Search

Engine for Earth Observation Images Repositories, 2006.


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Important points in RF

Feature selection Texture, shape, color etc.

Learning scheme Bayesian, Kernel learning methods, Neural Networks, Decision Trees

Scalability In terms of storage space and query processing times

Type of search: Category search

Target search


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Proposed search engine Search engine concept based on RF algorithm Field of application: Satellite CBIR Search tool: Support Vector Machines (SVM) Bayesian inference used to infer data models

Hierarchy of information

Data driven User drivenLearing/Unlearning


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Support Vector Machine

Linear separation case Labeled data training set

Find a separation surface

Decision function f = sign(g(x)) d+ = distance from g to closest {+1}

d- = distance from g to closest {-1}

Margin area = d++d-=

Find a separating hyperplane with largest margin

0bx w g(x)

xi

xj

g > 0

g < 0

g = 0

1..Ni , 11,y , F x, y,x id

iii

w2

margin area

SV+

SV-

1b)x(wy ii

N

i

N

i

N

iiiiiiiii xywandy

b

Land

w

LxwywL

1 1 1

000)1)((2

1

0,0)2

1(max

11,1

i

N

iii

N

jijijiji

N

ii yxxyy

Most Relevant (MR)

Most Ambiguous(MA)


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

SVM based Search Engine Algorithm

1. Query selection

2. Ranking

3. User’s relevance: {+1,-1}

4. Retrieve the Most Ambiguous (MA) images (g~0)

5. Go to 3

6. Return the Most Relevant images (MR) (max g)

7. Model data (GMM) inference – generation of categories1. No of Gaussian components K obtained by Minimum Description Length (MDL)2. Parameters estimation by Expectation and Maximization (EM)

Memory

K

kkk

Tk

k

Nkp1

1

2

12

))()(2

1exp(

)2(

1)(


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

System memory

Hierarchy of information Classes Λ Categories C SV sets

Adding memory

1. Queries sessions for images within the database 2. Semantical annotation of the SVM generated classes3. Categories generation 4. Save the set of Support Vectors (SV)

Algorithm

1. Query example2. Compute the likelihod based on the saved data models3. Choose the set of SV for which the likelihood is maximized4. Start RF with the chosen SV set


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

1. D Query image

2. Θ Search engine

3. Λ

4. C

5. U airports,……, village,………..

Hierarchy construction


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

RF quality evaluation

Measure the capability of retrieving relevant data

Relevance has to be experienced Relevance is personal & situational

Precision – Recall graphs B – retrieved images A – relevant images

Recall: fraction of the relevant images which have been retrieved

R = |B∩A|/ |B |

Precision: fraction of the retrieved images which are relevant

P = |B∩A|/ |A|

Mean Precison Combines precision, relevance ranking and overall recall

N

i

iPN

p1

)(1


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Satellite database

SPOT5 ©CNES scenes 3000x3000 pixels Resolution 5m/pixel Cropped 64x64 pixels small images 46 scenes used to create an 11 classes

database 100 examples per class Primitive features: QMF coefficients (11)

Gaussian kernel: 8 images per RF step 15 RF steps Evaluation of the top 60 MP images

)2exp(),(2

jijiK

1. Clouds

2. Sea

3. Desert

4. City

5. Forest

6. Fields

7. Airports

8. Village

9. Savanna

10. Boats

11. Traffic circle


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

SVM-RF simulation results (1) – primitive feature evaluation Database composed of 600 SPOT5 images divided

in six classes Used features: Gabor, Haralick, QMF and GMRF Gaussian Kernel System evaluation: Precision-Recall graphs


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

SVM-RF simulation results (2) – automatic indexation

Database composed of 1100 SPOT5 images divided in eleven classes

The system has memory QMF features System evaluation: Precision-Recall graphs

Search engine Speed of learning Mean precision

SVM based 15 RF steps 0.9477

Category based 1 RF step 0.9368


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

SVM-RF simulation results (3) - classification

Classification (structure detection / recognition) SPOT5 image over Paris Cemeteries areas


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Conclusion

Use of hierarchy of information suitable for EO image interpretation

Semantical annotations of the generated classes

Category Bayesian based learning enhances the search capabilities by speeding the learning process

SVM supports classification of EO scenes.


Co

mp

eten

ce C

entr

e o

n I

nfo

rmat

ion

Ext

ract

ion

an

d I

mag

e U

nd

erst

and

ing

fo

r E

arth

Ob

serv

atio

n

Thank You!

competence centre on information extraction and image understanding for earth observation 29th march...

Documents

image understanding

contentbased image retrieval

information extraction

semantic search engine

image retrieval system

competence centre

relevant information

satellite content