![Page 1: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/1.jpg)
Reasoning About Object Affordances in a Knowledge Base Representation
Yuke Zhu, Alireza Fathi, and Li Fei-FeiECCV14
Presented by Fereshteh [email protected]
![Page 2: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/2.jpg)
Affordance
“Properties of an object [...] that determine what actions a human can perform on them”
--Gibson 1979
![Page 3: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/3.jpg)
Affordance
• Combination of :
• An affordance label (e.g. edible)
• A human pose representation of the action (e.g. skeleton form)
• Relative position of the object with respect to human pose (e.g. next to)
![Page 4: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/4.jpg)
Knowledge structure
![Page 5: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/5.jpg)
Knowledge structure
Entity: Apple
![Page 6: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/6.jpg)
Knowledge structure
Entity: Apple Visual Attributes
![Page 7: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/7.jpg)
Knowledge structure
Entity: Apple Visual Attributes
Affordance
![Page 8: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/8.jpg)
Main Goal
• Predict affordances of unseen objects
• Infer richer information beyond visual similarity
• Knowledge based approach for reasoning and answering various types of questions
![Page 9: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/9.jpg)
Knowledge Base (KB)
Nodes: EntitiesEdges: General rules to characterize relations
![Page 10: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/10.jpg)
Knowledge Base (KB)
Nodes: Entities
Edges: General rules to characterize relations
• Visual attributes (e.g. round)• Physical attributes (weight & size)• Categorical attributes (e.g. apple)• Affordance (e.g. edible)
![Page 11: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/11.jpg)
Knowledge Base (KB)
Nodes: EntitiesEdges: General rules to characterize relations
• attribute-attribute• attribute-affordance• human-object-interaction
• attribute-pose, affordance-pose, attribute-location,affordance-location
![Page 12: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/12.jpg)
System overview
![Page 13: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/13.jpg)
• Seed KB with 40 objects & actions (Stanford 40 dataset)
• 14 affordance (Stanford 40 dataset)
• 100 images for each object (ImageNet)
![Page 14: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/14.jpg)
• WordNet: hypernym hierarchy
• Freebase: animal synopsis
• Amazon & eBay: physical attributes (weight & size)
![Page 15: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/15.jpg)
• WordNet: hypernym hierarchy
• Freebase: animal synopsis
• Amazon & eBay: Physical attributes (weight & size)
![Page 16: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/16.jpg)
Affordances:Manual labeling
Future: Use Google N-gram
![Page 17: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/17.jpg)
Pose descriptor Relative Locations
![Page 18: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/18.jpg)
33 pre-trained visual attribute classifierDescribe shape, material & parts of objects
![Page 19: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/19.jpg)
Learning KB using Markov Logic
• Markov Logic Network (MLN)
• Unify MRF with first-order logic
Possible worlds feature function
![Page 20: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/20.jpg)
• Markov Logic Network (MLN)
• Unify MRF with first-order logic
Likelihood of the formulae being trueL-BFGS Optimization
Learning KB using Markov Logic
![Page 21: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/21.jpg)
negative weight
positive weightentities (atomic formulae in MLN)
![Page 22: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/22.jpg)
![Page 23: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/23.jpg)
zero-shot affordance prediction
![Page 24: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/24.jpg)
zero-shot affordance prediction
Base features
Visual attributes rank svm
categorical attributes
, Base featureslogistic regression classifier
physical attributes
Base features
![Page 25: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/25.jpg)
zero-shot affordance prediction
Base features
Visual attributes rank svm
categorical attributes
, Base featureslogistic regression classifier
physical attributes
Base features
First-order inference to predict affordances
![Page 26: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/26.jpg)
zero-shot affordance prediction
KB models complex general rules
Classifiers fail to take correlations into account
![Page 27: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/27.jpg)
Estimating human pose
Pose: (torso, lower body, upper body)
cluster centroids
Set of ground-truth poses of the canonical affordance of the object
![Page 28: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/28.jpg)
Question Answering
MLN infers the probability or the most likely state of each query
from the evidence
![Page 29: Reasoning About Object Affordances...Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu, Alireza Fathi, and Li Fei-Fei ECCV14 Presented by Fereshteh Sadeghi](https://reader030.vdocuments.site/reader030/viewer/2022040607/5eb8d381cb141d30056d85dd/html5/thumbnails/29.jpg)
why KB