sentences and pictures - university of illinoisluthuli.cs.uiuc.edu/~daf/talks/cvpr-sp.pdf · •in...
TRANSCRIPT
![Page 1: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/1.jpg)
Sentences and pictures:not just “more words” and pictures
D.A. Forsyth, UIUCwith
Julia Hockenmaier, Derek Hoiem, Ian Endres, Ali Farhadi, Nicolas Loeff, Cyrus Rashtchian, Gang Wang
all of UIUCwith echoes from
Kobus Barnard (U. Arizona), Pinar Duygulu (Bilkent U.), Nando de Freitas (UBC
![Page 2: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/2.jpg)
Words and pictures: Implicit csp
Barnard et al 01, 01
![Page 3: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/3.jpg)
• In its simplest form, missing variable problem• Pile in with EM• given correspondences, conditional probability table is easy (count)• given cpt, expected correspondences could be easy
• Caveats• might take a lot of data; symmetries, biases in data create issues
Words and pictures: Explicit csp
“the beautiful sun”
“le soleil beau” “sun sea sky”Brown, Della Pietra, Della Pietra & Mercer 93;
Melamed 01 See Duygulu et al 02
![Page 4: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/4.jpg)
How to generalize words and pictures?
• More accuracy• More words• Predict more structure
![Page 5: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/5.jpg)
Accuracy
Y. Mori et al 99Duygulu et al, 02Jeon et al 03
Jeon et al 04Lavrenko et al 03
Feng et al 04
Carneiro et al, 05
Celebi et al 05
Feng et al 04Metzler et al 04
Yavlinsky et al, 05
Viitaniemi et al 07
![Page 6: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/6.jpg)
More words
• Easy case• learn with larger vocabularies• tricky bits, but...
• Hard case• what do we do about out-of-example words?• one simple answer doesn’t work (later)
![Page 7: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/7.jpg)
Structure
• Correlated words• waves go with beaches not cats
• Attributes• has nose
• Adjectives• green hat
• Relations• cat on mat
• Sentences• A dolphin holds a basketball as it swims on its back
![Page 8: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/8.jpg)
Correlated Words
• Simple method:• rack up some features, build a bunch of linear classifiers one per word• works poorly
• few examples per word• many features, only some are stable
Word data (observed) Image representation (observed)
D !MX
Learn this
![Page 9: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/9.jpg)
Correlated words
• Idea• some features are not helpful• a low dimensional subspace is good at predicting most things (Ando
+Zhang, )• We can find this space by penalizing rank in the matrix of linear classifiers
Word data (observed) Image representation (observed)
Learn this
D ! GFX
![Page 10: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/10.jpg)
Loeff Farhadi 08
![Page 11: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/11.jpg)
It was there and we predicted it
It was there and we didn’t
It wasn’t and we did
Loeff Farhadi 08
![Page 12: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/12.jpg)
Correlated word predictors are quite good
Loeff Farhadi 08
![Page 13: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/13.jpg)
Structure
• Correlated words• waves go with beaches not cats
• Attributes• has nose
• Adjectives• green hat
• Relations• cat on mat
• Sentences• A dolphin holds a basketball as it swims on its back
![Page 14: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/14.jpg)
General architecture
Farhadi et al 09; cf Lampert et al 09
![Page 15: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/15.jpg)
How is an object different from typical?
• Pragmatics suggests this is how adjectives are chosen• If we are sure it’s a cat, and we know that• an attribute is different from normal• the detector is usually reliable• we should report the missing/extra attribute
![Page 16: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/16.jpg)
Missing attributes
![Page 17: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/17.jpg)
Extra attributes
![Page 18: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/18.jpg)
Structure
• Correlated words• waves go with beaches not cats
• Attributes• has nose
• Adjectives• green hat
• Relations• cat on mat
• Sentences• A dolphin holds a basketball as it swims on its back
![Page 19: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/19.jpg)
“Pink” from Google
Yanai Barnard 05
![Page 20: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/20.jpg)
“Pink” after 10 EM iterations
Yanai Barnard 05
![Page 21: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/21.jpg)
Wang et al 09
![Page 22: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/22.jpg)
Structure
• Correlated words• waves go with beaches not cats
• Modifiers• pink cadillac
• Attributes• has nose
• Modifier-noun pairs• green hat
• Relations• cat on mat
• Sentences• Two women wearing jeans , one with a blue scarf around her head , sit and talk .
Gupta and Davis 08, but there is still a lot here
![Page 23: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/23.jpg)
Structure
• Correlated words• waves go with beaches not cats
• Attributes• has nose
• Adjectives• green hat
• Relations• cat on mat
• Sentences• A dolphin holds a basketball as it swims on its back
Gupta and Davis 08, but there is still a lot here
![Page 24: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/24.jpg)
Relations distort participants
![Page 25: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/25.jpg)
Relations distort participants
![Page 26: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/26.jpg)
Relations distort participants
![Page 27: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/27.jpg)
Structure
• Correlated words• waves go with beaches not cats
• Attributes• has nose
• Adjectives• green hat
• Relations• cat on mat
• Sentences• A dolphin holds a basketball as it swims on its back
![Page 28: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/28.jpg)
Two girls take a break to sit and talk .
Two women are sitting , and one of them is holding something .
Two women chatting while sitting outside
Two women sitting on a bench talking .
Two women wearing jeans , one with a blue scarf around her head , sit and talk .
![Page 29: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/29.jpg)
A crowd of young adults in a dark room .
A girl in a brown shirt and a blue jean skirt is dancing with a young man dressed in a blue shirt wearing a black backpack .
A group of people standing in a dark building .
A large group of people dancing in a bar
Dancing at club and two guys bucking up
![Page 30: Sentences and pictures - University Of Illinoisluthuli.cs.uiuc.edu/~daf/Talks/CVPR-SP.pdf · •In its simplest form, missing variable problem • Pile in with EM • given correspondences,](https://reader034.vdocuments.site/reader034/viewer/2022042120/5e9a337419309e397c6e142e/html5/thumbnails/30.jpg)
Conclusions
• Real progress in accuracy
• Structure is still hard, but rewarding
• Big problem: predicting sentences