supporting beyond-surface interaction for tabletop display systems by integrating ir projections...
TRANSCRIPT
Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections
Hui-Shan Kao
Advisor : Dr. Yi-Ping Hung
2
Outline
Introduction Related Work System Design Interaction Techniques Applications Conclusion
3
Outline
Introduction Related Work System Design Interaction Techniques Applications Conclusion
4
Introduction
Interaction with surfaces mainly support Multi-touch Tangible input
Interaction beyond surfacesCombine with mobile displays
iPad Tablet PCPico ProjectoriPhone
Introduction5
Two scenarios Type 1:
Using pico projector Multi-resolution presentation Augmenting personal information
Type 2: Using tablet PC
Interactive 3D viewer
Type2 Type1
Challenge6
How to know the 3D spatial relationship with surface
in real-time ??
Type2 Type1
7
Outline
Introduction Related Work System Design Interaction Techniques Applications Conclusion
8
Localization of Device
Enable 3D interaction on tabletop display Need to recognize the 6DOF of device
The way to know 6DOF Magnetic tracker
Penlight
H.Song, T.Grossman, G. Fitzmaurice, F. Guimbretiere, A. Khan, R. Attar, and G. Kurtenbach. Penlight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proc. CHI ’09, 2009
9
Localization of Device
Vision based tracker Markers with known structure
Interactive handheld projector ARToolkit
Visible
Marker !
Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. Multi-user interaction using handheld projectors. In Proc. UIST ’07, 2007.D. Wagner and D. Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc CVWW’07: Proceedings of 12th Computer Vision Winter Workshop, 2007.
10
Outline
Introduction Related Work System Design
Hardware Configuration Interaction Techniques Applications Conclusion
11
Hardware Configuration
Color Projection visible content
IR Camera IR
Camera
Color Projector
IR Camera
IR Projector
IR Projection invisible markers
Mirror
12
Hardware Configuration
Multi-touch Surface Glass layer support touch force Diffuser layer display image
Placement of glass layer and diffuser layer Glass on top of diffuser Diffuser on top of glass
13
Hardware Configuration
Option 1: Glass on top of diffuser
pico-projection reflection• degrade illumination• damage user’s eye
IR & colorprojectorIR camera
Pico-projector
IR camera
diffuser
touch-glass
Hard to Solve!
14
Hardware Configuration
Option 2: Diffuser on top of Glass
IR-projection reflection• spot effect in IR images
IR & color projectorIR camera IR camera
Pico-projector
spot spot
touch-glass
diffuser
15
Hardware Configuration
Removing IR spot by using two cameras
Weighting Mask (2)IR Camera(2)
Weighting Mask (1)IR Camera(1)
Stitched View
IR Camera 1
IR Camera 2
16
Outline
Introduction Related Work System Design Interaction Techniques
Adaptive Markers for Camera Estimation Dynamic Markers for Multi-Touch
Applications Conclusion
17
3D Position Estimation
3D position?
Type2 Type13D position?
18
ARToolKitPlus for 3D estimation Fiducial marker
Self-identify by ID
Projecting a grid of markers on tabletop Each marker with
Unique ID Associated position
Compute the camera’s position based on the marker perceived
3D Position Estimation
(0,0)
(1,1)
ID = 0
19
Multi-Level Markers
Uni-level marker Camera might observe the markers too small or
too big
Multi-level marker System resizes the IR markers according to
camera position
20
Marker Split and Merge
Adapting the maker size Markers merge, when camera moves far
enough Markers split, when camera moves too
close Ensure camera to see at least 4 markers
21
Cooperating with multi-touch
Marker on for camera positioning Marker off for finger detection Foreground-ROI detection for marker
on/off
22
23
Cooperating with multi-touch
?
24
Cooperating with multi-touch
25
Background Simulation
Offline Save each marker as a patch image and
record the position of marker
_ = patch patch
collection
(offline) :: for each marker
base view
26
Background Simulation
Online As the layout re-arrange, the simulated
background can be built by the saving patch in real time
_ =simulate
d backgrou
nd
(online) :: for each marker in layout ( )
patch of marker
base view
27
Background Simulation
28
ROI Generation
1st frame after finger touch
2nd and the other frames
29
IR Projecto
r
IR Cameras
IR Camera of Mobile Device
Smoothing
Simulated Background
Observed Image
Foregrounds
Tangible Objects
Finger Touches
Layout Manager
Prediction
Color
Projector
ROI
Applications
KalmanFiltering
Real scene
Summary of Foreground Detection
30
Outline
Introduction Related Work System Design Interaction Techniques Applications Conclusion
31
Three Applications
Three applications provide intuitive and natural manipulation.
iLamp
iFlashlight
iView
32
iLamp
iLamp
Combine a pico projector and an IR camera
Project seamless high-resolution content, bringing more detailed information
Act as a desk lamp for personal use
iLamp34
? Real-timeRc Tc
Rpc Tpc
Rp Tp
[Rp |Tp ] = [Rpc |Tpc ] x [Rc |Tc ]
Rpc Tpc
35
Pico projector and camera calibration
Off-line : Find transformation between camera and pico projector The projector can be viewed as a dual of camera Rpc Tpc are the Rotation and translation of the
attached camera
Rpc Tpc
36
On-line : Find transformation between pico projector and tabletop Estimate Rc Tc in real-time Compute Rp Tp
[Rp |Tp ] = [Rpc |Tpc ] x [Rc |Tc ]
Pico projector and camera calibration
Rpc Tpc
37
iFlashlight
iFlashlight
A mobile version of iLamp, can be moved easily.
Multi-user cooperation
39
iView
An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.
iView
41
Outline
Introduction Related Work System Design Interaction Techniques Applications Conclusion
42
Conclusion
A new interactive surface based on the programmable invisible markers.
Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera.
Bring another level of information on interactive surface.
43
Thank you