studying relationships between human gaze, description, and computer vision

11
Studying Relationships Between Human Gaze, Description, and Computer Vision Kiwon Yun 1 , Yifan Peng 1 Dimitris Samaras 1 , Gregory J. Zelinsky 1,2 , Tamara L. Berg 3 1 Department of Computer Science, Stony Brook University 2 Department of Psychology, Stony Brook University 3 Department of Computer Science, University of North Carolina Computer Vision and Pattern Recognition (CVPR) 2013

Upload: davis

Post on 23-Feb-2016

43 views

Category:

Documents


0 download

DESCRIPTION

Studying Relationships Between Human Gaze, Description, and Computer Vision. Kiwon Yun 1 , Yifan Peng 1 Dimitris Samaras 1 , Gregory J. Zelinsky 1,2 , Tamara L. Berg 3 1 Department of Computer Science, Stony Brook University 2 Department of Psychology, Stony Brook University - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Studying Relationships Between Human Gaze, Description, and Computer Vision

Studying Relationships Between Human Gaze, Description, and Computer Vision

Kiwon Yun1, Yifan Peng1

Dimitris Samaras1, Gregory J. Zelinsky1,2, Tamara L. Berg3 1Department of Computer Science, Stony Brook University

2Department of Psychology, Stony Brook University3Department of Computer Science, University of North Carolina

Computer Vision and Pattern Recognition (CVPR) 2013

Page 2: Studying Relationships Between Human Gaze, Description, and Computer Vision

OverviewUser behavior while freely viewing images contains an abundance

of information - about user intent and depicted scene content.

• Human gaze “where” the important things are in an image.

• Description “what” is in an image, which parts of an image are important to the viewer.

• Computer vision “what” might be “where” in an image. However, it will always be noisy and have no knowledge of importance.

Page 3: Studying Relationships Between Human Gaze, Description, and Computer Vision

OverviewUser behavior while freely viewing images contains an abundance

of information - about user intent and depicted scene content.2. From these exploratory analyses,

we build prototype applications for gaze-enabled object detection and annotation.

Human gaze

DescriptionAn old black woman wearing a turban and a headdress and a dog next to her wearing the same red headdress

Image content

1. We conduct several experiments to better understand the relationship between gaze, description, and image content.

Page 4: Studying Relationships Between Human Gaze, Description, and Computer Vision

SUN09• 104 images, free-viewing for 5 seconds• eye movements from 8 observers, each of whom provided a scene description.

Datasets

PASCAL VOC• 1,000 images, free-viewing for 3 seconds• eye movements from 3 observers• 5 natural language descriptions per image from different observers.

Page 5: Studying Relationships Between Human Gaze, Description, and Computer Vision

Experiments and Analyses

• People are more likely to look at people, other animals, televisions, and vehicles.• People are less likely to look at chairs, bottles, potted plants, drawers, and rugs.• Animate objects are much more likely to be fixated than inanimate objects (0.636 vs. 0.495).

Gaze vs. Object Type

Probability of being fixated when present for various object categories (top: PASCAL, bottom: SUN09)

Page 6: Studying Relationships Between Human Gaze, Description, and Computer Vision

Experiments and Analyses

Gaze vs. Location on Objects

person horse bird bus train tv

bicycle chair table cabinet curtain plant

• Animate objects are much more likely to be described than inanimate objects (0.843 vs. 0.545).

What objects do people describe?

Page 7: Studying Relationships Between Human Gaze, Description, and Computer Vision

Experiments and Analyses

What is the relationship between gaze and description?

P (fixated | described) P (described | fixated)PASCAL 0.866 0.952SUN09 0.737 0.725

S1: A man is reading the label on a beverage bottle. S2: A man looking at the bottle of beer that he is holding. S3: The man in a white tee shirt is holding a beer bottle and looking at it. S4: The scraggly haired man is holding up and admiring his bottle of beer. S5: Young man with curly black hair holding a beer bottle.

Fixated objects: bottle, person. Described objects: bottle, person

Page 8: Studying Relationships Between Human Gaze, Description, and Computer Vision

Gaze-Enabled Computer Vision

Analysis of Human Gaze with Object Detectors• Potential for gaze to increase the performance of object detectors varies by

object category.

Page 9: Studying Relationships Between Human Gaze, Description, and Computer Vision

Gaze-Enabled Computer Vision

Gaze-Enabled Object Detection and Annotation• Combine gaze and automated object detection methods to create a collaborative

system for detection and annotation.

Page 10: Studying Relationships Between Human Gaze, Description, and Computer Vision

Gaze-Enabled Computer Vision

Gaze-Enabled Object Detection and Annotation

Page 11: Studying Relationships Between Human Gaze, Description, and Computer Vision

Conclusion

• Through a series of behavioral studies and experimental evaluations, we explored the information contained in eye movements and description, and analyzed their relationship with image content.

• We also examined the complex relationships between human gaze and outputs of current visual detection methods.

• In future work, we will build on this work in the development of more intelligent human-computer interactive systems for image understanding.

[1] Studying Relationships Between Human Gaze, Description, and Computer Vision, Kiwon Yun, Yifan Peng, Dimitris Samaras, Gregory J. Zelinsky, and Tamara L. Berg, Computer Vision and Pattern Recognition (CVPR) 2013 (Oregon/USA)

[2] Specifying the Relationships Between Objects, Gaze, and Descriptions for Scene Understanding, Kiwon Yun, Yifan Peng, Hossein Adeli, Tamara L. Berg, Dimitris Samaras, and Gregory J. Zelinsky, Visual Science Society (VSS) 2013 (Florida/USA)