peekaboom: a game for locating objects in images roy liu carnegie mellon university joint work with...
Post on 15-Jan-2016
217 views
TRANSCRIPT
![Page 1: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/1.jpg)
Peekaboom: A Game for Locating Objects in Images
Roy Liu
Carnegie Mellon University
Joint work with Luis von Ahn and Manuel Blum
![Page 2: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/2.jpg)
Object Location in Images
Given an image, determine what objects there are and locate them:
Woman
Man
Umbrella
Tree
Sailboat
Dog
![Page 3: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/3.jpg)
As Things Stand Now
No algorithm is known for taking an image and determining what objects are in it, let alone telling you where they are.
Fortunately, this talk isn’t about developing such an algorithm. Let’s try a different approach.
![Page 4: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/4.jpg)
Harnessing the Power of Human Cycles
“Math is hard. Let’s go shopping!” –BarbieOn similar line of thinking:
• Programming computers to locate objects in images is hard, so…
• Let’s not think about it.• Instead, let’s get humans to do the work for
us.
![Page 5: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/5.jpg)
Problems
Wait! Your average human probably wants:
• Enjoyment – they want to have a good time• Incentives – they want something in return
How do we address both?
![Page 6: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/6.jpg)
A Game
Have people do the work for us by playing a game.
Many design issues arise:• What will be the core idea of the game?• How do we collect data?• How do we ensure the quality of the data?
![Page 7: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/7.jpg)
An Earlier Idea: Luis von Ahn’s ESP Game
Two players, with no communication, each try to guess what the other is thinking about a particular image they both see.
If they agree on a word, the game moves on and increases both players’ scores.
![Page 8: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/8.jpg)
A Sample Run
Player 2 Guesses
• Woman• Shirt• Denim• Girl• Model
Player 1 Guesses
• Pants• Model• Lady
Server: Agreed, “Model”
![Page 9: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/9.jpg)
Why ESP Works
By agreeing on a word, the players:• Say what it is – we call this assigning a “label”
to the image.• Check their own work – the fact that two
strangers agree on a label is a witness of the label’s quality.
![Page 10: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/10.jpg)
The Limitations of ESP
The ESP Game can label images (and consequently tell you what’s in them), but it cannot:• Find the objects being labeled.• Determine the way in which the object
appears – does the label “car” refer to the text “car” or an actual car in the image?
![Page 11: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/11.jpg)
Completing the Image Cycle
unlabeled images ESP game serverlabeled images
located images Peekaboom game server
![Page 12: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/12.jpg)
A New Idea: Peekaboom
Two players are assigned the roles of “revealer” and “guesser”.
The revealer sees an image with a label. The guesser sees nothing.
The revealer shows the guesser parts of the image. If the guesser guesses correctly, the game moves on.
![Page 13: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/13.jpg)
Statement of Purpose
We would like to collect data about images systematically and en masse.
We hope our collection will form the basis for data sets that can be used to train computer vision algorithms.
![Page 14: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/14.jpg)
A (Simplified) Trial Run
![Page 15: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/15.jpg)
The Revealer clicks on parts of the image and shows them to the Guesser.
The Guesser guesses:•Flower•Petal•Butterfly
Server: Correct, Butterfly
![Page 16: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/16.jpg)
Why Peekaboom Works
By getting the guesser to guess correctly, the revealer locates objects by clicking on the relevant parts of the image:
![Page 17: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/17.jpg)
But Wait, There’s More
Peekaboom not only locates objects, but also:• Gives the context necessary to identify them.• Classifies the image as “Text”, “Noun”, or
“Verb” by way of hints.Let’s examine how Peekaboom does both.
![Page 18: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/18.jpg)
•Pings help separate the context of object with the object itself.•They help the guesser distinguish trunk from other possibly correct labels like “elephant”, “tusk”, and “ear”.
The label: trunk
Object Context
![Page 19: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/19.jpg)
Hints
The label “car” is ambiguous --
this is “car”
this is also “car”
The hints help distinguish the manner in which the label “car” appears:
this is the object “car”
this is the text “car”
![Page 20: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/20.jpg)
The Architecture of Peekaboom
game server
players
compiler
researchers
players
labeled images
located images
raw data
![Page 21: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/21.jpg)
Peekaboom is…
funnovelaesthetically appealingnetworkedscalablewidely deployable
![Page 22: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/22.jpg)
Conclusions
Peekaboom will be released to a general audience within two months.
We hope that it will solve difficult AI tasks while achieving:• Low costs – One game server.• Quality – Accurately locate objects in images.• Quantity – Locate objects in millions of
images.
![Page 23: Peekaboom: A Game for Locating Objects in Images Roy Liu Carnegie Mellon University Joint work with Luis von Ahn and Manuel Blum](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649d5d5503460f94a3c16d/html5/thumbnails/23.jpg)
Advertisement
The best way to understand this talk is to try the game out for yourself:
www.peekaboom.org
We look forward to collecting your cycles!