M-TEEVE: Real-Time 3D Video Interaction and Broadcasting Framework

for Mobile Devices

Shu Shi, Won Jeon, Klara Nahrstedt, and Roy CampbellDepartment of Computer Science

University of Illinois at Urbana-Champaign{shushi2, wonjeon, klara, roy}@illinois.edu

IMMERSCOM, Berkeley, CA, May 27-29, 2009

Thursday, June 16, 2011

Overview

• Introduce mobile devices to real-time 3D video rendering• Challenges in bandwidth and computation

• Remote rendering framework

• Use mobile devices to interact with 3D video • Eye tracking

• Orientation tracking

2Thursday, June 16, 2011

Outline

• Background• 3D Video

• TEEVE

• M-TEEVE Framework• Proxy-Based Remote Rendering Framework

• View-Dependent 3D Video Compression

• Mobile Interaction• Eye Tracking

• Orientation Tracking

• Implementations

• Conclusion

3Thursday, June 16, 2011

Background

• 3D Video• Stereo video and Free-viewpoint video

4Thursday, June 16, 2011

User

Background

• TEEVE Framework

5

Acquisition

Rendering Transmission

3D Cameras

Reconstruction Servers

RenderingServer

Display

Gateway Server

Site 1

Internet 2

Site 2

Site 3

Site n

... ...

Depth Calculation

Color Capturing

Thursday, June 16, 2011

Background

• TEEVE

6Thursday, June 16, 2011

User

M-TEEVE Framework

7

Acquisition

Rendering Transmission

3D Cameras

Reconstruction Servers

RenderingServer

Display

Gateway Server

Site 1

Internet 2

Site 2

Site 3

Site n

... ...

• M-TEEVE

• Mobile Tele-Immersion

Thursday, June 16, 2011

M-TEEVE Framework

• Challenges• Transmission bandwidth

• Rendering computation

• 3D Video Rendering• 3D Video Display

• Rendering 3D video streams

• Rendering graphics background

• Display 3D video scene

• 3D Video Control• Video playback control

• Viewpoint navigation

8

Depth Image Streams

Compressed Rendered Scene

M-TEEVE

Control Information

Gateway Server

Proxy

Display Units

Control Units

Thursday, June 16, 2011

M-TEEVE Framework

• Advantages• Compatibility with original

TEEVE system

• Interface between mobile devices implementation and other parts of TEEVE

• Support various devices with limited computing resources

• Support of multi-control and multi-display

• Disadvantages• Interaction delay

9

Depth Image Streams

Compressed Rendered Scene

M-TEEVE

Control Information

Gateway Server

Proxy

Display Units

Control Units

Thursday, June 16, 2011

Mobile Interaction

• Viewpoint Navigation• Zooming -- forward, backward

• Rotation -- clockwise, counterclockwise

• Translation -- left, right, up, down

• Circling -- left, right, up, down

10Thursday, June 16, 2011

Mobile Interaction

• Eye Tracking• Connecting eye movements to Translation, Zooming and

Rotation

11

width

Thursday, June 16, 2011

Mobile Interaction

• Orientation Tracking

12

View direction of viewers

Orientation of mobile devices

Virtual Camera

3D video objects

Thursday, June 16, 2011

Implementations

• On both Nokia N800 and iPhone

13Thursday, June 16, 2011

Implementations

• Proxy• Workstation with Gbps wired network

• Compress rendered scene to JPEG images

• Control Units• Touch screen and button events

• Camera, accelerometers

• Wii remotes

• Display Units• Up to 25 fps

• Multi-views

• Interaction delay less than 150ms

14Thursday, June 16, 2011

Implementations

• Eye Tracking• OpenCV

• Feature points selection

• Lucas-Kanade algorithm for optical flow calculation

• Performance• 5 fps with resolution of 160x120

15

0

50

100

150

200

250

300

0 50 100 150 200 250 300

Exe

cu

tatio

n t

ime

(m

s)

Frame number

Eye tracking Performance

LaptopN800

Exec

utio

n T

ime

(ms)

Frame Number

N800 Tablet

Laptop

Thursday, June 16, 2011

Implementations

• Orientation Tracking• Accelerometers in iPhone

• iPhone SDK

• High detection rate

• Gravity orientation

• Filters for noise reduction

16Thursday, June 16, 2011

Conclusion

• Proxy-based remote rendering framework solves the problems of transition bandwidth and rendering computation

• Cameras and accelerometers provide new methods for 3D video interaction

• The problem of interaction delay needs to be reduced in the future work

17Thursday, June 16, 2011

Thank you!

• Acknowledgement• The work is supported by NSF grant CNS 05-20182 and

CNS 07-20702

• TEEVE group at UIUC and work by UC Berkeley

• Project Info• http://cairo.cs.uiuc.edu/teleimmersion/

• Contact• Shu Shi: shushi2@illinois.edu

18Thursday, June 16, 2011

Interaction Delay

19

Wired Network

Mobile Devices

Proxy GatewayServer

Wireless Network

3D Video Frame

F2

F1

F3

Image FrameAfter Rendering

R1

R2

R3

... ...

... ...

Fi

... ...

User Changes the Rendering Viewpoint

Proxy Updates the Rendering Viewpoint

First Frame Processed at the Updated

Rendering ViewpointFirst Result Frame at the Updated

Rendering Viewpoint

InteractionDelay

Thursday, June 16, 2011

View-Dependent 3D Video Compression

20

Multi Depth Image Streams

Input

Viewpoint

3D Points Downsampling

Invisible Points

ViewpointLayer

Stream FrameBasicLayer

Visible Points3D Points

Projection

3D Points

3D Points Reconstruction

Viewpoint Listener

Viewpoint from Control

Units

3D Point Reconstruction

Image MapDisplay Selection

FrameParsing

ViewpointLayer

BasicLayer

EnhancedLayer

Viewpoint Module

Compare with Current Viewpoint

Current Viewpoint

?Display

3D Point Rendering

Stream Frame

Stream Frame

......

.....

.....

Stream Frame

Stream Frame

......

Stream Frame

......

......

......

......

EnhancedLayer

Encoding

DecodingThursday, June 16, 2011