bidi screen a thin, depth-sensing lcd for 3d interaction using light fields matthew hirsch ...
DESCRIPTION
http://web.media.mit.edu/~mhirsch/bidi/ Watch video: http://www.youtube.com/watch?v=kXuxK6IeQfo BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar 1MIT Media LabTRANSCRIPT
BiDi ScreenDepth and Lighting Aware Interactive Display
Matthew HirschMIT Media Lab
Douglas LanmanBrown University
Ramesh RaskarMIT Media Lab
Henry HoltzmanMIT Media Lab
BiDi Screen
Inspiration
Light Sensitive Displays Depth Cameras
Multitouch
Beyond Multi-touch: Hover Interaction
• Seamless transition of multitouch to gesture
• Thin package, LCD
Beyond Multi-touch: Mobile
Laptops
Mobile
Results: Analysis
BiDi Screen
• Capture 3-D to enable depth and lighting aware interaction
• Collocated image capture and display
• Walk-up interaction
• Mobility and portability – thin device
Design Overview
Dis
play
with
em
bedd
ed o
ptic
al s
enso
rs
Sharp Microelectronics Optical Multi-touch Prototype
Design Overview
Dis
play
with
em
bedd
ed o
ptic
al s
enso
rs
LCD , displaying mask
Opt
ical
sen
sor a
rray
~2.5 cm~50 cm
Design Vision
Object Collocated Captureand Display
Bare Sensor
Spa
tial L
ight
Mod
ulat
or
Design Overview
Mask
Array of Virtual Cameras
Sensor
Manipulating an object with 3D gesture
Alternatives to capture depth from an LCD
• Adapted Touch
• Capacitive
• Resistive
• Optical
• Camera arrays
• Behind screen
• To sides
• Depth Sensors/Cameras
Adapting Traditional Touch?
• Resistive touch screen
• Confined to screen
• Capacitive
• Not lighting dependant
• Calibration problems for gesture
• No existing technology trend for gesture
?
Camera Arrays
• Cameras behind screen
• Interfere with backlight
• Expensive, Large
• Han , SecondLight, ThinSight
• No tech curve
• Cameras to side
• Transition region
• Second modality
?
Depth Cameras and Sensors
• Depth Cameras
• Behind screen w/ projector (MSR TouchlLight)
• No longer a thin device
• No light sensitivity
• Touch difficult
Theory: Depth from light-field capture
• LCD used for Spatial Heterodyne Light Field Capture and Display
• Pinholes or tiled broadband masks
• Separate Sensor and Mask
• Fourier Refocusing
• Depth from focus
• Blob tracking (gesture)
Theory: Lightfield
θx
y θ
x
.
Sensor integrates these rays
(Sensor)
Theory: Lightfield Frequency Domain
fθ
fx
θ
x
Lightfield FT
Fourier Slice Theorm
FT
Theory: LF Skew in Free-Space Propogation
θ
x
x
y
mask
θ
mask
Theory: Convolution with Delta Functions
Theory: Transforms
fθ
fx
θ
x
Lightfield FT
fmaskmaskFT
Tiled Broadband Mask
Theory: Spatial Heterodyning
x
y
mask
Multiplication in prime domain is convolution in frequency domain
Theory: Spatial Heterodyning
fθ
fxfmask
fθ
fx
Theory: Spatial Heterodyning
fθ
fx
Reconstructed LightfieldSpectrum
Band Limited
Desired PrototypeActual Prototype
Pipeline
Max ContrastOperator
Pipeline
Theory: Depth and Spatial Resolution
Theory: Depth and Spatial Resolution
Results: Analysis
Manipulating an object with 3D gesture
Navigating a 3D world with 3D gesture
Lighting a vritual scene with a real light
Limitations
• Requires separation between display and sensor (adds thickness)
• 2.5 cm for 50 cm range in our prototype
• 750mm for 8 cm range in iPhone like device
• Sensitive to room lighting or requires its own light source
• This is true of many optical systems
• Time multiplexing of display/capture
• Requires fast capture and screen refresh to stay below flicker fusion rate
• 240 Hz LCDs coming to market
Conclusions
• Future Work
• Dynamic Masks
• Change frequency characteristics to match scene
• Video capture / Video chat (higher resolution)
• SIGGRAPH Asia 2009 Paper
• BiDi Screen: Depth and Lighting Aware Interaction and Display
Conclusions
• Enable multitouch and gesture interaction on a thin display screen
• Walk-up interaction does not require gloves / fiducials
• Capture depth using array of vitual cameras
• Thin portable devices possible with area sensor