beckman background knowledge
TRANSCRIPT
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 1/44
Using Background Knowledge
to Improve Visual Learning
Derek Hoiem
Beckman Director’s Seminar
March 11, 2009
Work with: Ali Farhadi, Ian Endres, Gang Wang, Santosh Divvala, James Hays
David Forsyth, Alexei Efros, Martial Hebert
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 2/44
What I’d like to make possible with computer vision
Household Robot Intelligent Vehicle
Security Photo Organization
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 3/44
What we can do (with the right dataset)
• Recognize faces
• Categorize scenes
• Detect, segment, and track objects
• 3D from multiple images or stereo
• Classify actions
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 4/44
But we’re a long way from “Rosie”
Computer vision has been divided into many
task- and dataset-specific problems
– Difficult to coordinate pieces – Poor generalization to unfamiliar environments
– Massive engineering and data collection effort
required for every task/dataset
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 5/44
Goal
Use background knowledge: generalize known
solutions to new problems or dataset
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 6/44
The Challenge
How can we use what we know to make learning
new things easier and more robust?
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 7/44
This Talk
• Three uses of background knowledge
– Contextual knowledge
– Compositional knowledge
– Organizational knowledge
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 8/44
I. Contextual Knowledge
Goal: Use knowledge of objects and spatial
layout to better detect a new object.
Work with Santosh Divvala, James Hays, Alexei Efros, Martial Hebert
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 9/44
Object Detection without Context
Search over many positions and scales
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 10/44
Object Detection without Context
Cat?
Cat?
Cat?
In each window: is this a cat?
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 11/44
Training a Detector
Color
Texture
Edges
xx
x x
x
x
x
xx
oo
oo
o
Classifier FeaturesExamples
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 12/44
Object Detection without Context
In each window: is this a cat?
,
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 13/44
Object Detection without Context
• Top five cat detections in a challenging dataset
Detector: Felzenszwalb et al. CVPR 2008 Dataset: PASCAL VOC 2008
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 14/44
What do we know that can help us?
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 15/44
What do we know that can help us?
Knowledge of Other Objects and Scenes
Similar ImagesLarge Set of
Loosely
AnnotatedImages
Associated Keywords
BabyPuppy
Sand
House KittenHelps tell us howlikely the object is to
appear in this image.
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 16/44
.
.
Occlusion Boundaries Depth EstimatesSurface Layout
Helps tell us where and how
big the object is likely to be.
Knowledge of Spatial Layout
What do we know that can help us?
Hoiem et al. 2005,2007
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 17/44
Context: Likelihood of Presence
1. Object presence
Contains Cat No Cat
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 18/44
Context: Likelihood of Presence
Image
Gist
Surface Layout
Associated Keywords
BabyPuppy
Sand
House Kitten
gist: Torralba Oliva 2003
Likely to contain a cat?
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 19/44
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 20/44
Context: Likelihood of Size
• Predict height of object based on depth, surface orientations,gist, and image position
Predicted Height
Candidate
Bounding Box
Size from Gist: Torralba Oliva 2003
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 21/44
Rescoring Candidate Objects
Presence Scores
Position Scores
Size Scores
Appearance Score(from detector)
Independently Trained
Classifiers
Bounding BoxScore
Linear Weights L1-Regularized Logistic
Regression
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 22/44
Context improves detection
Top 5: Before Context
Top 5: After Context
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 23/44
Context improves detection accuracy
Average Precision (Higher is Better)
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 24/44
Context changes the error patterns
•More confusion
– Cats and Dogs
– Dogs and Sheep
– Motorbike and Bicycle
• Less confusion
– Objects and background
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 25/44
II. Compositional Knowledge
Goal: Describe new objects using attributes
learned from other objects.
Work with Ali Farhadi, Ian Endres, David Forsyth
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 26/44
A name doesn’t tell us much
Name: Cat
Name: Dog
Name: Horse
Known Objects New Object
Name: Unknown
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 27/44
But what if we learn attributes?
Name: Cat
Properties: four legs, tail,
eyes, ears, furry, has
stripes, gray
Name: Dog
Name: Horse
Known Objects New Object
Properties: four legs,
eyes, ears, snout, tan,
muscular
Properties: four legs, tail,
mane, eyes, ears, snout,
tan
Name: Unknown
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 28/44
We can infer what object is like
Name: Cat
Properties: four legs, tail,
eyes, ears, furry, hasstripes, gray
Name: Dog
Name: Horse
Known Objects
Name: Unknown
Properties: four legs, eyes,
snout, tan, muscular
Properties: four legs, tail,
mane, eyes, ears, snout,
tan
Properties: four legs,eyes, ears, snout,
stripes, mane
New Object
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 29/44
Learning Attributes
• Learn to distinguish between things that have
an attribute and things that don’t
• Train one classifier per attribute
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 30/44
Learning Correlated Attributes
Problem• Many attributes are strongly correlated through the
object category
Most cars are “made of metal” and have “wheels”
When we try to learn “has
wheels”, we may accidentally
learn “made of metal”
Has Wheels, Made of Metal?
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 31/44
Decorrelating Attributes
Solution• Select features that can distinguish between two
classes
– Things that have wheels
– Things that do not, but have other attributes in common
“Has Wheels” “No Wheels”
Vs.
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 32/44
Learning to Describe Objects
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 33/44
Describing New Objects
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 34/44
Identifying Unusual Attributes
Absence of Typical Attributes
Presence of Atypical Attributes
752 reports
68% are correct
951 reports
47% are correct
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 35/44
Recognition from Description
• Learn new classes by describing them to the
algorithm
• Goat = “Is furry, four legged, has snout, has horn”
• 12-Class Classification Accuracy = 32.5%
• Chance = 8%
• As good as having 8 visual examples with original imagefeatures
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 36/44
III. Organizational Knowledge
Goal: Help a person organize his photos usingimage similarity learned from Flickr groups.
Work with Gang Wang, David Forsyth
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 37/44
Taming the Digital Explosion
Photos are easy to take and store.
But it’s still difficult to organize them.
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 38/44
Solution: Learn from photo sharing sites
• Billions of images in Flickr
• Hundreds of thousands of categories
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 39/44
Learn similarity
• Downloads hundreds of groups, each
containing thousands of photos
• Train classifier to predict whether a photo islikely to belong in each group
– Gang Wang created super-fast online training
method for kernelized SVMs
• Images are similar if they are likely to belong
to the same group
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 40/44
We can find similar images
Query Image
Retrieved Images Using
Feature Similarity
Retrieved Images Using
Similarity Learned from Flickr
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 41/44
We can say how two images are similar
Fireworks (15.6)Christmas (7.6)
Rain (4.0)
Water drops (2.5)
Candles (2.0)
Sports (2.6)
Dances (2.0)
Weddings (1.0)
Toys (0.5)
Horses (0.5)
Painting (2.4)
Art (1.2)
Macro-flowers (0.9)
Hands (0.9)
Skateboarding (0.6)
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 42/44
Conclusions
• Background knowledge is a key missing component in today’s
computer vision algorithms
• Existing knowledge can make learning easier
– Provides new abilities (say two things are similar or different)
– More complete visual models (better accuracy, more reasonable
mistakes)
– Better able to handle new objects and situations
• We need to start designing systems that accumulate
visual knowledge
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 43/44
Thank you
7/31/2019 Beckman Background Knowledge
http://slidepdf.com/reader/full/beckman-background-knowledge 44/44