stanford hci group / cs376 vision-based interaction€¦ · 640x480 sony firewire camera for video;...
TRANSCRIPT
1
stanford hci group / cs376
http://cs376.stanford.eduScott Klemmer · 28 November 2006
Vision-Based Interaction
2
2
cs547: Blake Ross and Asa Dotzler
Mozilla: Creating simple software in a geek-driven culture
3
3
The first vision-based interface
Myron Krueger used computer vision to create Responsive Environments (1970s)“Reaction is the Medium”http://www.artmuseum.net/w2vr/timeline/videoplace_video.html
4
4
How it works
Video and background are separated in analog using chroma key techniques(think broadcast news)The first and last points of each raster are stored in the computer, and represent the person’s outline
5
5
Vision-based UIs: “Verbs”
Detecting and Tracking elements of a certain type in a sceneCapturing contents of detected objectsRecognizing individual members in an object class
6
6
Vision-based UIs: “Verbs”
Detecting and Tracking elements of a certain type in a scene
7
7
Vision-based UIs: “Verbs”
Capturing contents of detected objects
8
8
Vision-based UIs: “Verbs”
Recognizing individual members in a class
9
9
Vision-based UIs: “Nouns”
People (one or multiple)BodiesFacesHands
DocumentsObjects
10
10
Vision-based UIs: “Nouns”
People (one or multiple)BodiesFacesHands
DocumentsObjects
11
11
Vision-based UIs: “Nouns”
People (one or multiple)BodiesFaces Hands
DocumentsObjects
12
12
Background SubtractionI N F R A S T R U C T U R E
Two Cameras
640x480 Sony Firewire camera for video; Mounted behind board; eliminates occlusion; Used for Tracking Notes
3 megapixel Epson still USB camera; Treat as ~1fps ‘video’ camera; Mount in front, use for ink capture
13
13
Image Moments (of Inertia)
0th moment is mass(total number of pixels)
14
14
Image Moments (of Inertia)
1st moment is center
15
15
Image Moments (of Inertia)
2nd moment is orientation
16
16
Tools for Vision appsIntel’s OpenCV
C API to highly optimized image processing functions (threshold, dilate, optical flow, …)http://www.intel.com/research/mrl/research/opencv
Fast to run! Slow to developGreat for vision folks; too low-level for app folks
Papier-MâchéJava API (and to some extent visual UI) for vision (and other physical input)http://guir.berkeley.edu/papier-macheFast to develop! Slow to runGreat for app folks; ~5 fps can sometimes be too slow
17
17
Good Vision Books
Computer Vision: A Modern ApproachDavid Forsyth and Jean Ponce (2003)Fantastic book; but goal is more theoretical understanding than practical application
Robot VisionBerthold Horn (1987)More focused on apps and interactive algorithmsSomewhat out of date
18
18
Next Time… Software Tools
Past, Present, and Future of User Interface Software Tools, Brad Myers, Scott E. Hudson, Randy Pausch
Natural Programming Languages and Environments, Brad A. Myers, John F. Pane, Andy Ko