fifty shades of augmented reality: creating connection using ar

67
FIFTY SHADES OF AUGMENTED REALITY: CREATING CONNECTION USING AR Mark Billinghurst [email protected] March 24 th 2017 Laval Virtual

Upload: mark-billinghurst

Post on 05-Apr-2017

1.634 views

Category:

Technology


0 download

TRANSCRIPT

FIFTY SHADES OF AUGMENTED REALITY: CREATING CONNECTION USING AR

Mark [email protected]

March 24th 2017

Laval Virtual

• jg

Ancient Communication Tools

Evolution of Communication Tools

TECHNOLOGY TRENDS

Modern Technology Trends

1. Improved Content Capture• Move from sharing faces to sharing places

2. Increased Network Bandwidth• Sharing natural communication cues

3. Implicit Understanding• Recognizing behaviour and emotion

These three trends combined with AR/MR technology can create new types of collaborative experiences

Content CaptureRealism

Time

Photo

Film

Live Video

Panorama

360 Video

3D Space

1850 1900 1940 1990 2000 2010

Capturing Reality• Panoramas (1790s)

• Immersive paintings• Photography (1820-30s)

• Oldest surviving photo (Niépce, 1826)• Stereo imagery (1830s)

• Wheatstone (1832)• Brewster (1851)

• Movies (1870s)• Muybridge (1878)• Roundhay Garden Scene (1888)

Sharing Faces to Sharing Spaces

• Moving from desktop to handheld cameras

Capturing 360 images

Kodak 360 Fly 360 Gear 360 Theta S Nikon

LG 360 Pointgrey Ladybug Panono 360 Bublcam

360 Stereo Capture

Samsung Project Beyond

Jaunt VR camera

https://www.youtube.com/watch?v=FnBIvq8jzds

3D Space Capture

Google Project Tango

Occipital Structure Sensor

https://www.youtube.com/watch?v=XA7FMoNAK9M

Content CaptureRealism

Time

Photo

Film

Live Video

Panorama

360 Video

3D Space

1850 1900 1940 1990 2000 2010

2D Static

Immersive

Live

Experience

Networking SpeedsLog (b/s)

Time

100 b/s

10 Kb/s

1 Mb/s

1980 1985 1990 1995 2000 2010

100 Mb/s

2005

Networking Technology

• Telephone (1876)• Bell Picture Phone (1955)

• B+W live video calls• Mobile Phones (1985)• Desktop Video Conferencing (1990’s)

• Netmeeting, Skype

https://www.youtube.com/watch?v=OBkcYyenNBM

Mobile Conferencing (2010-)

• Video conferencing• Skype, facebook

• Live broadcasting• Meerkat, Periscope• Support for 360 video

Network Innovation

Universal Connectivity

Networking SpeedsLog (b/s)

Time

100 b/s

10 Kb/s

1 Mb/s

1980 1985 1990 1995 2000 2010

100 Mb/s

2005

Text

Audio

Natural

Video

Interaction TechnologyNatural

Time

PunchCard

Keyboard

Mouse

Speech

Gesture

Emotion

1950 1960 1980 1990 2000 2010

Thought

Natural Gesture Input

• Freehand input• Depth sensors, rich two handed gestures

Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S. (2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).

https://www.youtube.com/watch?v=A-xXrMpOHyc

Physiological Sensing

Emotiv Empatica

Interaction TechnologyNatural

Time

PunchCard

Keyboard

Mouse

Speech

Gesture

Emotion

1950 1960 1980 1990 2000 2010

Thought

Implicit

Explicit

MIXED REALITY CONFERENCING

Milgram’s Reality-Virtuality Continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

RealEnvironment

AugmentedReality (AR)

AugmentedVirtuality (AV)

VirtualEnvironment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

Mixed Reality Conferencing in SciFi

AR Video Conferencing (2001)

• Bringing conferencing into real world• Using AR video textures of remote people• Attaching AR video to real objects

Billinghurst, M., & Kato, H. (2002). Collaborative augmented reality. Communications of the ACM, 45(7), 64-70.

Demo: AR Video Conferencing

MR Point Cloud Sharing (2016)• Use video-see through HMD

• Occulus Rift• Depth camera on HMD

• Softkinetic• Create point cloud

• Remote hand gestures• User’s space

Demo: Point Cloud Capture

Holoportation (2016)

• Augmented Reality + 3D capture + high bandwidth • http://research.microsoft.com/en-us/projects/holoportation/

Holoportation Video

https://www.youtube.com/watch?v=7d59O6cfaM0

Social VR

• Facebook Social Virtual Reality, AltspaceVR• Bringing Avatars into VR space• Natural social interaction

Demo: Facebook Social VR (2016)

https://www.youtube.com/watch?v=yxHwWHHg4Vs

AR/VR Collaboration (2017)

• Make 3D copy of real space• AR user in real world, VR user in 3D copy of real space

• Hololens for AR user, HTC Vive for VR user• Share virtual body cues (head, hands, gaze information)

Real World Virtual World

Virtual Cues

• AR/VR displays• Gesture input

• Leap Motion• Room scale tracking

Demo: AR/VR Conferencing

EMPATHIC COMPUTING

Natural Collaboration

Implicit Understanding

Experience Capture

Natural Collaboration

Implicit Understanding

Experience Capture

Empathic Computing

Empathy

“Seeing with the Eyes of another,

Listening with the Ears of another,

and Feeling with the Heart of another..”

Alfred Adler

Empathic Computing

Systems that allow us to share what we are seeing, hearing and feeling with others..

Example: Google Glass

• Camera + Processing + Display + Connectivity

• Ego-Vision Collaboration (But with Fixed View)

Current Collaboration on Wearables

• First person remote conferencing/hangouts• Limitations

• Single POV, no spatial cues, no annotations, etc

Social Panoramas (ISMAR 2014)

• Capture and share social spaces in real time• Supports independent views into Panorama

Reichherzer, C., Nassani, A., & Billinghurst, M. (2014, September). [Poster] Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 303-304). IEEE.

Implementation

• Google Glass • Capture live image panorama (compass + camera)

• Remote device (tablet)• Immersive viewing, live annotation

User Interfaces

Glass View

Tablet View

Social Panorama Demo

https://www.youtube.com/watch?v=vdC0-UV3hmY&t=1s

JackIn – Live Immersive Video Streaming

• Jun Rekimoto – University of Tokyo/Sony CSL

JackIn Hardware

• Wide angle cameras – 360 degree video capture• Live video stitching

JackIn Demo

https://www.youtube.com/watch?v=Lkuz6LhlBf8

• zxcz

JospehTame – Tokyo Marathon

• Live streaming from Tokyo marathon• http://josephta.me/en/tokyo-marathon/

Empathy Glasses (CHI 2016)

• Combine together eye-tracking, display, face expression• Impicit cues – eye gaze, face expression

++

Pupil Labs Epson BT-200 AffectiveWear

Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

AffectiveWear – Emotion Glasses

• Photo sensors to recognize expression• User calibration• Machine learning• Recognizing 8 face expressions

Empathy Glasses in Use

• Eye gaze pointer and remote pointing• Face expression display• In future integrated eye-tracking/display

Empathy Glasses Demo

https://www.youtube.com/watch?v=CdgWVDbMwp4

Empathic VR Environments (2017)

• Player and Viewer• Viewer slaved to player

• Share emotional signals • Heart rate, GSR

• Remote affect measuring

Sharing VR Experiences

• HTC Vive HMD• Empathic glove • Empatica E4

VR Environments

• Butterfly World: calm scene, collect butterflies• Zombie Attack: scary scene, fighting zombies

Demo Video

https://www.youtube.com/watch?v=SaiHxps-Ofw

AR and VR for Empathic Computing• VR systems are ideal for trying experiences:

• Strong story telling medium • Provide total immersion/3D experience• Easy to change virtual body scale and representation

• AR systems are idea for live sharing:• Allow overlay on real world view/can share viewpoints• Support remote annotation/communication• Enhance real world task

CONCLUSIONS

Conclusions•Trend towards Empathic Computing

• Understanding, Experiencing, Sharing

•AR/VR Enables Empathic Experiences• Changes perspective • Sharing space/experience• Supports annotation/communication

•Many directions for future research

www.empathiccomputing.org

@marknb00

[email protected]