![Page 1: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/1.jpg)
Hierarchical Image Classification over Concept Ontology
Jianping Fan Dept of Computer Science
UNC-Charlotte
Course Website: http://webpages.uncc.edu/jfan/itcs5152.html
![Page 2: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/2.jpg)
Pipeline for Traditional Image Classification System
Training Image Set
Feature Extraction
Classifier Training Classifier
Offline Training
Online Testing
Test Image
Feature Extraction
Classifier
FlowerGarden Nature……….
predictions
![Page 3: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/3.jpg)
Feature Extraction
Image-based approach
Grid-based approach
Object-based approach
Bag-of-Visual-Word
Deep Networks
![Page 4: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/4.jpg)
Input Image Salient Objects
Visual FeaturesColor histogram, Tamura texture, Locations …….
Feature Extraction
Object-Based Approach
![Page 5: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/5.jpg)
Feature Extraction
Image-Based Approach
![Page 6: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/6.jpg)
Feature Extraction
Grid-Based Approach
![Page 7: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/7.jpg)
Feature Extraction
Bag-of-Visual-Word Approach
![Page 8: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/8.jpg)
Image Representation
Salient Objects
Color histogram
Tamura Texture
Shape
Images
Color histogramWavelet Texture histogram
![Page 9: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/9.jpg)
Image Representation
Bag-of-Visual-Word
Histogram ofVisual Words
Image Grids
Color histogramWavelet Texture histogram
![Page 10: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/10.jpg)
Feature-Based Image Representation
Feature dimension
Feature dimension
Feature dimension Curse of Dimensionality
![Page 11: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/11.jpg)
Organization of Large-Scale Categories
Library: 12,000,000 books
![Page 12: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/12.jpg)
I get it!
Too easy!
Books in Library
Natural Sciences Social Sciences
DancingComputer Science
ElectricalEngineering
Computer Languages Researches
Database Multimedia
Organization of Large-Scale Categories
![Page 13: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/13.jpg)
What is the solution?
Concept Hierarchy or Ontology
Organization of Large-Scale Categories
![Page 14: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/14.jpg)
Hierarchical Concept Organization Concept Ontology
![Page 15: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/15.jpg)
Hierarchical Concept Organization Concept Ontology
![Page 16: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/16.jpg)
CalTech101
Hierarchical Concept Organization
![Page 17: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/17.jpg)
Two-Layer Ontology for ImageNet1K
Hierarchical Concept Organization
![Page 18: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/18.jpg)
Hierarchical Concept Organization
![Page 19: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/19.jpg)
Two-Layer Ontology for ImageNet10K
Hierarchical Concept Organization
![Page 20: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/20.jpg)
Two-Layer Ontology for Taobao Products
Hierarchical Concept Organization
![Page 21: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/21.jpg)
Plant Ontology
Hierarchical Concept Organization
![Page 22: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/22.jpg)
Two-Layer Ontology for Orchidaceae
Hierarchical Concept Organization
![Page 23: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/23.jpg)
Hierarchical Concept Organization
Concept Ontology
2. How to support hierarchical classifier training over concept ontology?
Multi-Level Image Annotation:Annotating Images at Different Semantic Levels
Multi-Modal Features
1. How to integrate multiple multi-modal features for classifier training?
![Page 24: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/24.jpg)
Beach
Water & Sky Water & Sand Water, Sand, Sky, …
Feature Subset 1
Feature Subset 2
Feature Subset 9
Atomic Image Concept
Different Patternsof Co-Appearances of Salient Objects
Feature Subsets
122
nCM nn
i
i
n
Classifier Training for Atomic Image Concepts
![Page 25: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/25.jpg)
Classifier Training for Atomic Image Concepts
Curse of Dimensionality # samples needed increase with # dimensions
(generally exponentially) .
Human labeling is expensive
Some features are redundant
Proposal Joint SVM boosting and feature selection
![Page 26: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/26.jpg)
Boosting SVM Classifier Training
PCA PCA PCA PCA
Subspace 1 Subspace 2 Subspace 3 … Subspace N
Higher Level Classifier
Weak Classifier 1
SV
M
…
Boosting for optimal combination
Weak Classifier 2 Weak Classifier 3 Weak Classifier N
SV
M
SV
M
SV
M
•Less training samples due to dimension reduction
•Reuse training results on low-level concepts
•More selection opportunities compared to filter and wrapper
Low-level classifiers
High-level classifier
High dimensional feature space
![Page 27: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/27.jpg)
Classifier Training for Atomic Image Concepts
Kernel-Based Data Warping
)(X
Kernel Function: )()(),( j
T
iji XXXX
![Page 28: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/28.jpg)
Classifier Training for Atomic Image Concepts
Kernel for Color Histogram
N
i ii
ii
vu
vuvu
1
22 )(
2
1),(
Statistical Image Similarity
Kernel
/),(2
),( vuevu
![Page 29: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/29.jpg)
Classifier Training for Atomic Image Concepts
Wavelet Filter Bank Kernel
![Page 30: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/30.jpg)
Classifier Training for Atomic Image Concepts
Wavelet Filter Bank Kernel
n
i
ijii yhxh
eyx 1
2 /))(),((
),(
n
i
yhxh iiiie1
/))(),((2
![Page 31: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/31.jpg)
Classifier Training for Atomic Image Concepts
Interest Point Matching Kernel
![Page 32: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/32.jpg)
Classifier Training for Atomic Image Concepts
Interest Point Matching Kernel
![Page 33: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/33.jpg)
Classifier Training for Atomic Image Concepts
Multiple Kernel Learning
1
^
),(),(i
ii yxyx 11
i
i
SVM Image Classifier
M
l
lll bXXYgXf1
^
),(sin)(
![Page 34: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/34.jpg)
Classifier Training for Atomic Image Concepts
Dual Problem
M
l
l
h
h cw1
2
122
1min
Subject to:
l
h
lkil
M
l bXwY
1)(,,0:1
1
![Page 35: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/35.jpg)
Classifier Training for Atomic Image Concepts
![Page 36: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/36.jpg)
Classifier Training for Atomic Image Concepts
Garden Scene
Beach Scene
Some Results
![Page 37: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/37.jpg)
High-Level Image Concept Modeling
Inter-Concept Similarity Modeling
Nature Scene
Garden Beach Flower View
Nature Scene: Larger Hypothesis Space & Large Variations of Visual Properties!
Garden, Beach, Flower view: Different but share common visual properties!
![Page 38: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/38.jpg)
Classifier Training for High-Level Image Concepts
Challenging Problems
Error Transmission Problems
Training Cost Issue
Knowledge Transferability and Task Relatedness Exploitation
![Page 39: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/39.jpg)
Error Transmission Problem
Classifier Training for High-Level Image Concepts
The classifiers for low-level image concepts cannot recover the errors for the classifiers of high-level image concepts!
![Page 40: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/40.jpg)
Error Transmission Problem
Classifier Training for High-Level Image Concepts
Outdoor
Garden Beach Flower View
Errors for the classifiers of atomic image concepts may be transmitted to the classifiers for the high-level image concepts!
![Page 41: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/41.jpg)
Classifier Training for High-Level Image Concepts
Training Cost Issue Multiple Hypotheses
garden
outdoor
beach flower view
Large Diversity of Contents
![Page 42: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/42.jpg)
Classifier Training for High-Level Image Concepts
Knowledge Transferability & Task Relatedness Exploitation
Outdoor
Garden Beach Flower View
They are different but strongly related!
![Page 43: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/43.jpg)
Multi-Task Learning
Which tasks are strongly related?
How to quantify the task relatedness?
How to integrate such task relatedness for training large-scale related image classifiers?
Classifier Training for High-Level Image Concepts
![Page 44: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/44.jpg)
Related Learning Tasks
Nature Scene
Garden Beach Flower View
They are different but strongly related!
Concept Ontology can provide a good environment for multi-task learning!
Classifier Training for High-Level Image Concepts
![Page 45: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/45.jpg)
Related Learning Tasks
Classifier Training for High-Level Image Concepts
![Page 46: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/46.jpg)
Classifier Training for High-Level Image Concepts
Relatedness Modelling
bXWXf T
jC j)( jj VWW 0
garden
outdoor
beach flower view
0W : Common Prediction Structure
![Page 47: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/47.jpg)
Classifier Training for High-Level Image Concepts
Joint Objective Function
C
j
N
i
C
j
jij WCVC1 1 1
2
02
21min
Subject to:
ijijjij
N
i
C
j bXVWY 1)(: 011
![Page 48: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/48.jpg)
Dual Problem
Classifier Training for High-Level Image Concepts
C
j
C
j
N
i
C
h
N
l
jlihjhjljlihih
N
i
ij XXKYY1 1 1 1 11
),(2
1max
Subject to:
0,0:1 1
11
C
j
N
i
ijijij
C
j
N
i YC
![Page 49: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/49.jpg)
Biased Classifier Training
Classifier Training for High-Level Image Concepts
m
l
l
T
l bXWYWW1
2
0 )](1[2
1min
Dual Problem
m
l
m
h
m
l
l
T
llh
T
lhlhl XWYXXYY1 1 1
0 )1(2
1min
Subject to:
0,0:1
1
l
m
l
ll
m
l YC
![Page 50: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/50.jpg)
Common Prediction Structure
Nature Scene
Garden Beach Flower View
Common Visual Properties
Classifier Training for High-Level Image Concepts
![Page 51: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/51.jpg)
)()()(1
1
XfXpXHjk C
C
j
jC
1
1
))(exp(
))(exp()(
C
j
C
C
j
Xf
XfXp
j
j
Hierarchical Boosting
Classifier Training for High-Level Image Concepts
![Page 52: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/52.jpg)
Biased Classifier for Parent Node
Classifier Training for High-Level Image Concepts
ll
m
l
l XYWW
1
0
bXWXf T
Ck)(
![Page 53: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/53.jpg)
Hierarchical Boosting to Generate Classifier for Parent Node
Classifier Training for High-Level Image Concepts
)()()(1
1
XfXpXHjk C
C
j
jC
1
1
))(exp(
))(exp()(
C
j
C
C
j
Xf
XfXp
j
j
![Page 54: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/54.jpg)
Performance Evaluation
Classifier Training for High-Level Image Concepts
![Page 55: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/55.jpg)
Performance Evaluation
Classifier Training for High-Level Image Concepts
![Page 56: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/56.jpg)
Advantages of Hierarchical Boosting
Handling inter-concept similarity via multi-task learning
Reducing training cost
Enhancing discrimination power of the classifiers
Classifier Training for High-Level Image Concepts
![Page 57: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/57.jpg)
Hierarchical Image Classification
Overall Probability
Parent Node
Children Node 1 Children Node 2 Children Node C
Path 1 Path 2 Path C
)( kC
)( kC
)( 1C )( 2C )( cC
CjCCC jkk ,....1|)(max)()(
![Page 58: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/58.jpg)
Some Results
Hierarchical Image Classification
![Page 59: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/59.jpg)
Classification Result Evaluation
Allow Users to See Global View!
![Page 60: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/60.jpg)
Classification Result Evaluation
Allow Users to See Similarity Direction!
![Page 61: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/61.jpg)
Classification Result Evaluation
Allow Users to Zoom into Images of Interest!
![Page 62: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/62.jpg)
Classification Result Evaluation: Red Flower
Allow Users to Select Query Example Interactively!
![Page 63: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/63.jpg)
Classification Result Evaluation: Sunset
Allow Users to Look for Particular Images!
![Page 64: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/64.jpg)
Some Interesting Observations
Concept Ontology for identifying inter-related learning tasks
Multi-task learning for inter-related tasks
Hierarchical learning over concept ontology
![Page 65: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/65.jpg)
Deep Multi-Task Learning over Concept Ontology
Ontology-driven task group generation
Deep multi-task learning for each task group
Hierarchical deep multi-task learning over concept ontology
![Page 66: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/66.jpg)
Traditional Deep Learning
![Page 67: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/67.jpg)
Traditional Deep Learning
Problem of AlexNet
Inter-class visual correlations are completely ignored! They are assumed to be independent!
The differences of their learning complexities are completely ignored!
Back-propagation for optimization may pay more attentions on hard ones!
![Page 68: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/68.jpg)
Traditional Deep Learning
Wish List1,000 image categories
Task Group
Task Group Task Group
Task Group
Task Group
![Page 69: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/69.jpg)
Deep Multi-Task Learning over Concept Ontology
![Page 70: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/70.jpg)
IEEE Trans. IP 2008
Ontology for Task Group Generation
![Page 71: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/71.jpg)
CalTech101
Ontology for Task Group Generation
![Page 72: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/72.jpg)
Two-Layer Ontology for ImageNet1K
Ontology for Task Group Generation
![Page 73: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/73.jpg)
Ontology for Task Group Generation
![Page 74: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/74.jpg)
Two-Layer Ontology for ImageNet10K
Ontology for Task Group Generation
![Page 75: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/75.jpg)
Two-Layer Ontology for Taobao Products
Ontology for Task Group Generation
![Page 76: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/76.jpg)
Plant Species Ontology
Ontology for Task Group Generation
![Page 77: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/77.jpg)
Two-Layer Ontology for Orchidaceae
Ontology for Task Group Generation
![Page 78: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/78.jpg)
Ontology-Driven Task Group Generation
![Page 79: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/79.jpg)
Ontology-Driven Task Group Generation
![Page 80: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/80.jpg)
The sibling object classes are semantically similar and have closer learning complexities!
Putting such semantically-related object classes in the same task group, they can be compared more sufficiently and the gradients for their objective function will be more homogeneous!---not just pay more attention on hard ones!
Assumptions for Task Group Generation
3. Ontology-Driven Task Group Generation Ontology-Driven Task Group Generation
![Page 81: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/81.jpg)
Learning Base CNNs for Each Task Group
Deep Multi-Task Learning:
![Page 82: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/82.jpg)
Deep Multi-Task Learning
Learning Base CNNs for Each Task Group
![Page 83: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/83.jpg)
Deep Multi-Task Learning
Learning Base CNNs for Each Task Group
![Page 84: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/84.jpg)
Deep Multi-Task Learning
Multi-Task Classifiers at Sibling Leaf Nodes
Learning Base CNNs for Each Task Group
![Page 85: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/85.jpg)
Learning Base CNNs for Each Task Group
Deep Multi-Task Learning
![Page 86: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/86.jpg)
Hierarchical Deep Multi-Task Learning
![Page 87: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/87.jpg)
Hierarchical Deep Multi-Task Learning
![Page 88: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/88.jpg)
Hierarchical Deep Multi-Task Learning
![Page 89: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/89.jpg)
Hierarchical Deep Multi-Task Learning
Controlling Inter-Level Error Propagation
![Page 90: Hierarchical Image Classification over Concept Ontologyover Concept Ontology Jianping Fan Dept of Computer Science UNC-Charlotte Course Website: ... Color histogram, Tamura texture,](https://reader034.vdocuments.site/reader034/viewer/2022050121/5f5192a288caa356993908d0/html5/thumbnails/90.jpg)
Hierarchical Deep Multi-Task Learning