![Page 1: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/1.jpg)
3D-Assisted Image Feature Synthesis for Novel Views of an Object
Hao Su*Fan Wang*Li YiLeonidas Guibas
* Equal contribution
![Page 2: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/2.jpg)
View-agnostic Image Retrieval
Query
Retrieval using AlexNet features
![Page 3: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/3.jpg)
Cross-view Image Comparison
![Page 4: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/4.jpg)
Cross-view Image Comparison
The comparison is between the underlying 3D objects
![Page 5: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/5.jpg)
Reconstruct 3D and then compare?
Kar et al, CVPR’15 Huang et al, SIGGRAPH’15Su et al, SIGGRAPH’14
![Page 6: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/6.jpg)
Single-image based 3D Reconstruction is hard
Fg/bg segmentation
2D-3D Correspondence
Non-convex iterative optimization
Keypoint detection
2D image part segmentation
3D shape part segmentation
Common dependencies:Many dependenciesNot Robust
Slow
![Page 7: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/7.jpg)
Our Formulation: Novel View Feature Synthesis
Observed view(HoG feature as an example)
![Page 8: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/8.jpg)
Our Novel View Feature Synthesis Results
(HoG feature as an example)
![Page 9: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/9.jpg)
Outline
Motivation
Approach
Method Diagnosis
Applications
Conclusion
![Page 10: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/10.jpg)
Key idea
Learn from a dataset of many objects with multi-view features
…
![Page 11: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/11.jpg)
Key idea
Learn from a dataset of multi-view features
d
The dataset is generated by rendering 3D models
![Page 12: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/12.jpg)
Key idea
The dataset is generated by rendering large-scale 3D models
http://shapenet.cs.stanford.edu
Learn from a dataset of multi-view features
![Page 13: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/13.jpg)
3D-assisted Feature Synthesis: Nearest Neighbour
Observed view image
Novel view feature
(HoG feature as an example)
![Page 14: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/14.jpg)
Observed view image
Novel view feature
Strong assumption:very similar model exists
(HoG feature as an example)
3D-assisted Feature Synthesis: Nearest Neighbour
![Page 15: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/15.jpg)
Observed view image
Novel view feature
(HoG feature as an example)
...
3D-assisted Feature Synthesis: Multiple Shapes
![Page 16: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/16.jpg)
Attention:Brain games start!
3D-assisted Feature Synthesis: Multiple Shapes
![Page 17: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/17.jpg)
Pipeline
(HoG feature as an example)
Observed view image
Novel view feature
![Page 18: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/18.jpg)
Observed view image
Novel view feature
Pipeline
(HoG feature as an example)
![Page 19: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/19.jpg)
Observed view image
Novel view feature
Pipeline
(HoG feature as an example)
![Page 20: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/20.jpg)
Observed view image
Novel view feature
Pipeline
(HoG feature as an example)
++ …
![Page 21: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/21.jpg)
Pipeline
(HoG feature as an example)
++ …
Observed view image
Novel view feature
![Page 22: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/22.jpg)
Pipeline
(HoG feature as an example)
++ …
Observed view image
Novel view feature
++ …0.1 0.4 0.3
Locally Linear Reconstruction
![Page 23: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/23.jpg)
Pipeline
(HoG feature as an example)
++ …
Observed view image
Novel view feature
++ …0.1 0.4 0.3
Locally Linear Reconstruction
![Page 24: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/24.jpg)
Pipeline
(HoG feature as an example)
++ …
Observed view image
Novel view feature
++ …0.1 0.4 0.3
Locally Linear Reconstruction
![Page 25: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/25.jpg)
+
++
+ …
…0.1 0.4 0.3
Observed view image
Novel view feature
Pipeline
(HoG feature as an example)
Locally Linear Reconstruction
Inter-shape relationship
![Page 26: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/26.jpg)
Surrogate Relationship Discovery
+
++
+ …
…0.1 0.4 0.3
Observed view image
Novel view feature
?
Locally Linear Reconstruction
Inter-shape relationship (HoG feature as an example)
![Page 27: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/27.jpg)
Surrogate Relationship DiscoveryObserved view
Shape Collection
Novel view
![Page 28: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/28.jpg)
Surrogate Relationship DiscoveryObserved view
Shape Collection
Surrogate suitability matrix
Novel view
![Page 29: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/29.jpg)
Formal Definition of Surrogate Suitability
Shape Collection
𝐴
𝐵
Novel view
Observed view AssumeA, 𝐵 are discrete random variables
![Page 30: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/30.jpg)
Formal Definition of Surrogate Suitability
Shape Collection
𝐴
𝐵
Novel view
Observed view AssumeA, 𝐵 are discrete random variables
(𝑎1, 𝑏1), (𝑎2, 𝑏2), are i.i.d samples of (𝐴, 𝐵)
𝑎1
𝑏1
e.g.
𝑏2
𝑎2
![Page 31: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/31.jpg)
Formal Definition of Surrogate Suitability
Shape Collection
𝐴
𝐵
Novel view
Observed view AssumeA, 𝐵 are discrete random variables
(𝑎1, 𝑏1), (𝑎2, 𝑏2), are i.i.d samples of (𝐴, 𝐵)
𝑎1
𝑏1
e.g.
𝑏2
𝑎2
𝛾 𝐴; 𝐵 = log 𝑃(𝑏1 = 𝑏2|𝑎1 = 𝑎2)
Surrogate suitability:
![Page 32: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/32.jpg)
Formal Definition of Surrogate Suitability
Shape Collection
𝐴
𝐵
Novel view
Observed view AssumeA, 𝐵 are discrete random variables
(𝑎1, 𝑏1), (𝑎2, 𝑏2), are i.i.d samples of (𝐴, 𝐵)
𝑎1
𝑏1
e.g.
𝑏2
𝑎2
𝛾 𝐴; 𝐵 = log 𝑃(𝑏1 = 𝑏2|𝑎1 = 𝑎2)
Surrogate suitability:
How well can the sameness at A
predict the sameness at B?
![Page 33: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/33.jpg)
Formal Definition of Surrogate Suitability
Shape Collection
𝐴
𝐵
Novel view
Observed view AssumeA, 𝐵 are discrete random variables
(𝑎1, 𝑏1), (𝑎2, 𝑏2), are i.i.d samples of (𝐴, 𝐵)
𝑎1
𝑏1
e.g.
𝑏2
𝑎2
𝛾 𝐴; 𝐵 = log 𝑃(𝑏1 = 𝑏2|𝑎1 = 𝑎2)
Surrogate suitability:
How well can the sameness at A
predict the sameness at B?
Cross-view transfer of relationships
![Page 34: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/34.jpg)
Estimation of Surrogate Suitability
𝐻𝑅: Renyi-entropy
Derivation shows
![Page 35: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/35.jpg)
Estimation of Surrogate Suitability
Sample complexity: tight bound Θ 𝑉𝐴 + 𝑉𝐵
Derivation shows
where 𝑉𝐴 and 𝑉𝐵 are vocabulary size of 𝐴 and 𝐵
![Page 36: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/36.jpg)
Estimation of Surrogate Suitability
Sample complexity: tight bound Θ 𝑉𝐴 + 𝑉𝐵
Derivation shows
where 𝑉𝐴 and 𝑉𝐵 are vocabulary size of 𝐴 and 𝐵
Theoretically optimal algorithm is proposed that reaches the bound
![Page 37: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/37.jpg)
Estimation of Surrogate Suitability
Sample complexity: tight bound Θ 𝑉𝐴 + 𝑉𝐵
Derivation shows
where 𝑉𝐴 and 𝑉𝐵 are vocabulary size of 𝐴 and 𝐵
Strong connection with Mutual Information
Theoretically optimal algorithm is proposed that reaches the bound
![Page 38: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/38.jpg)
More Visualization of Surrogate Suitability MatrixNovel view Observed view
𝐵
![Page 39: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/39.jpg)
More Visualization of Surrogate Suitability MatrixNovel view Observed view
𝐵
![Page 40: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/40.jpg)
More Visualization of Surrogate Suitability MatrixNovel view Observed view
𝐵
![Page 41: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/41.jpg)
Review of Pipeline
+
++
+ …
…0.1 0.4 0.3
Observed view image
Novel view feature
![Page 42: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/42.jpg)
Review of Pipeline
Inter-shape relationship
+
++
+ …
…0.1 0.4 0.3
Observed view image
Novel view feature
Inter-shape relationship:
Knowledge transfer from 3D shape database
to new instance
![Page 43: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/43.jpg)
Review of Pipeline
Inter-shape relationship
+
++
+ …
…0.1 0.4 0.3
Observed view image
Novel view feature
Intr
a-s
ha
pe
rela
tio
nsh
ip
Inter-shape relationship:
Knowledge transfer from 3D shape database
to new instance
Intra-shape relationship:
Knowledge transfer from observed view
to novel view
![Page 44: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/44.jpg)
Outline
Motivation
Approach
Method Diagnosis
Applications
Conclusion
![Page 45: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/45.jpg)
Application: Cross-view localized image comparison
![Page 46: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/46.jpg)
Cross-view Image Retrieval
![Page 47: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/47.jpg)
HoG L2
Ours (combined HoG) swivel base
Application: View-agnostic Image Retrieval
vertical bars
![Page 48: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/48.jpg)
HoG L2
Ours (combined HoG) swivel base
Application: View-agnostic Image Retrieval
vertical bars
![Page 49: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/49.jpg)
HoG L2
Ours (combined HoG) swivel base
Application: View-agnostic Image Retrieval
vertical bars
![Page 50: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/50.jpg)
Part-based View-agnostic Image Retrieval
![Page 51: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/51.jpg)
Generalizability to Many Feature Types
• Task: fine-grained retrieval (images and annotations are from ImageNet)• Metric: Average Precision
![Page 52: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/52.jpg)
Outline
Motivation
Approach
Method Diagnosis
Applications
Conclusion
![Page 53: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/53.jpg)
How many shapes are sufficient?
(Measured by Average Precision on Fine-grained retrieval for Chairs)
200
![Page 54: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/54.jpg)
How many neighboring shapes for interpolation?
(Measured by Average Precision on Fine-grained retrieval for Chairs)
80
![Page 55: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/55.jpg)
How well can one view predict another view?
Cross-view retrieval rank
Controlled diagnosis on renderings
![Page 56: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/56.jpg)
Outline
Motivation
Approach
Method Diagnosis
Applications
Conclusion
![Page 57: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/57.jpg)
Conclusion
• A novel framework for synthesizing object features at novel views
• 3D shape database provides the knowledge of feature synthesis
• For relationship transfer, surrogate suitability is defined, which is a type of “predictability” between random variables.
• A theoretically optimal estimator is proposed
![Page 58: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/58.jpg)
Thank you!
![Page 59: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/59.jpg)
![Page 60: 3D-Assisted Image Feature Synthesis for Novel Views of an ...cseweb.ucsd.edu/~haosu/slides/iccv_novelview_website.pdf · •A novel framework for synthesizing object features at novel](https://reader033.vdocuments.site/reader033/viewer/2022042304/5ed0265f137588523c69a38b/html5/thumbnails/60.jpg)