vsmm 2016 keynote: using ar and vr to create empathic experiences
TRANSCRIPT
![Page 1: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/1.jpg)
USING AR AND VR TO CREATE EMPATHIC EXPERIENCES Mark Billinghurst [email protected]
October 19th 2016
VSMM 2016
![Page 2: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/2.jpg)
• daf
![Page 3: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/3.jpg)
Hiroshi Ishii – AWE 2014
![Page 4: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/4.jpg)
1977 – Star Wars
Augmented Reality
![Page 5: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/5.jpg)
Augmented Reality
• Defining Characteristics • Combines Real and Virtual Images
• Both can be seen at the same time • Interactive in real-time
• The virtual content can be interacted with • Registered in 3D
• Virtual objects appear fixed in space
![Page 6: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/6.jpg)
ARToolKit (1999)
First open source AR SDK http://artoolkit.sourceforge.net/
![Page 7: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/7.jpg)
Shared Space (1999)
• Face to Face interaction, Tangible AR metaphor • Easy collaboration with strangers • Users acted same as if handling real objects
![Page 8: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/8.jpg)
MagicBook (2001)
• First AR story book • Transitional AR to VR interface
![Page 9: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/9.jpg)
AR Tennis (2005)
• First collaborative AR game on mobile phone
![Page 10: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/10.jpg)
Mobile AR Advertising (2007)
• First mobile AR ad campaign (Saatchi & Saatchi)
![Page 11: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/11.jpg)
GeoBoids (2012)
• Outdoor AR game • Capture creatures
• Use whistling to attract creatures • Google Glass/Handheld AR
![Page 12: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/12.jpg)
GeoBoids Demo
https://www.youtube.com/watch?v=x3MKR3xVRM0
![Page 13: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/13.jpg)
Pokemon GO..
![Page 14: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/14.jpg)
Pokemon GO Effect
• Fastest App to reach $500 million in Revenue • Only 63 days after launch, > $1 Billion in 6 months • Over 500 million downloads, > 25 million DAU • Nintendo stock price up by 50% (gain of $9 Billion USD)
![Page 15: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/15.jpg)
AR Business Today
• Around $600 Million USD in 2014 (>$2B 2015) • 70-80+% Games and Marketing
![Page 16: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/16.jpg)
“Do you want to sell sugar water for the rest of your life or do you want to come with me and change the world?"
Steve Jobs to John Sculley 1983
![Page 17: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/17.jpg)
Mark’s Midlife Crisis
• Get Married • Sabbatical at Google • Looking for new opportunities • Resigned my job • Moved to a new country • Now creating a new research group
![Page 18: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/18.jpg)
• df
From Hiroshi Ishii
![Page 19: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/19.jpg)
• asfas
![Page 20: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/20.jpg)
Interaction Technology Natural
Time
Punch Card
Keyboard
Mouse
Speech
Gesture
Emotion
1950 1960 1980 1990 2000 2010
Thought
![Page 21: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/21.jpg)
Physiological Sensing
Emotiv Empatica
![Page 22: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/22.jpg)
Interaction Technology Natural
Time
Punch Card
Keyboard
Mouse
Speech
Gesture
Emotion
1950 1960 1980 1990 2000 2010
Thought
Implicit
Explicit
![Page 23: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/23.jpg)
Content Capture Realism
Time
Photo
Film
Live Video
Panorama
360 Video
3D Space
1850 1900 1940 1990 2000 2010
![Page 24: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/24.jpg)
Live Streaming/3D Space Capture
Google Project Tango Samsung Project Beyond
![Page 25: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/25.jpg)
Content Capture Realism
Time
Photo
Film
Live Video
Panorama
360 Video
3D Space
1850 1900 1940 1990 2000 2010
2D Static
Immersive
Live
Experience
![Page 26: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/26.jpg)
Networking Log (b/s)
Time
100 b/s
10 Kb/s
1 Mb/s
1980 1985 1990 1995 2000 2010
100 Mb/s
2005
![Page 27: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/27.jpg)
Network Innovation
Universal Connectivity
![Page 28: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/28.jpg)
Networking Log (b/s)
Time
100 b/s
10 Kb/s
1 Mb/s
1980 1985 1990 1995 2000 2010
100 Mb/s
2005
Text
Audio
Natural
Video
![Page 29: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/29.jpg)
Holoportation
• Augmented Reality + 3D capture + high bandwidth • http://research.microsoft.com/en-us/projects/holoportation/
![Page 30: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/30.jpg)
Holoportation Demo
https://www.youtube.com/watch?v=7d59O6cfaM0
![Page 31: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/31.jpg)
Natural Collaboration
Implicit Understanding
Experience Capture
![Page 32: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/32.jpg)
Natural Collaboration
Implicit Understanding
Experience Capture
Empathic Computing
![Page 33: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/33.jpg)
EMPATHIC COMPUTING Systems that Create and Share Understanding
![Page 34: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/34.jpg)
Empathy
“Seeing with the Eyes of another,
Listening with the Ears of another,
and Feeling with the Heart of another..”
Alfred Adler
![Page 35: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/35.jpg)
![Page 36: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/36.jpg)
Empathic Computing
1. Understanding: Systems that can understand your feelings and emotions
2. Experiencing: Systems that help you better experience the world of others
3. Sharing: Systems that help you better share the experience of others
Sensors
VR
AR
![Page 37: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/37.jpg)
1. Understanding: Affective Computing
• Ros Picard – MIT Media Lab • Systems that recognize emotion
![Page 38: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/38.jpg)
Appliances That Make You Happy
• Jun Rekimoto – University of Tokyo/Sony CSL • Smile detection + smart appliances
![Page 39: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/39.jpg)
Happiness Counter Demo
https://vimeo.com/29169237
![Page 40: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/40.jpg)
2. Experiencing: Virtual Reality
"Virtual reality offers a whole different medium to tell stories that really connect people and create an empathic connection."
Nonny de la Peña http://www.emblematicgroup.com/
![Page 41: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/41.jpg)
Using VR for Empathy
• USC Project Syria (2014) • Experience of Terrorism • Project Homeless (2015)
• Experience of Homelessness
![Page 42: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/42.jpg)
Demo: Project Syria
https://www.youtube.com/watch?v=Uuszow5giaQ
![Page 43: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/43.jpg)
CHILDHOOD
• Kenji Suzuki, University of Tsukuba • What does it feel like to be a child? • VR display + moved cameras + hand restrictors
![Page 44: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/44.jpg)
CHILDHOOD Demo
https://vimeo.com/128641932
![Page 45: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/45.jpg)
3. Sharing
Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
![Page 46: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/46.jpg)
• sdfs
![Page 47: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/47.jpg)
• sdfgs
![Page 48: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/48.jpg)
![Page 49: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/49.jpg)
• axcvxca
![Page 50: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/50.jpg)
Movies are like a machine that generates Empathy
Roger Ebert
![Page 51: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/51.jpg)
Technical Requirements
• Basic Requirements • Make the technology transparent
• Wearable, unobtrusive
• Technology for transmitting • Sights, Sounds, Feelings of another
• Audio, video, physiological sensors
![Page 52: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/52.jpg)
Wearable AR for Empathic Interfaces
• Wearable AR can: • Be unobtrusive • Capture emotion • Share sights and sounds • Provide two way communication • Enhance interaction in the real world
![Page 53: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/53.jpg)
Changing Perspective
• CamNet (1992) • British Telecom
• Wearable Teleconferencing • audio, video
• Remote collaboration • Sends task space video
• Similar CMU study (1996) • cut performance time in half
![Page 54: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/54.jpg)
WACL: Remote Expert Collaboration
• Wearable Camera/Laser Pointer • Independent pointer control • Remote panorama view
![Page 55: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/55.jpg)
WACL: Remote Expert Collaboration
• Remote Expert View • Panorama viewing, annotation, image capture
Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H., & Billinghurst, M. (2004, October). Remote collaboration using a shoulder-worn active camera/laser. In Wearable Computers, 2004. ISWC 2004. Eighth International Symposium on (Vol. 1, pp. 62-69).
![Page 56: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/56.jpg)
Example: Google Glass
• Camera + Processing + Display + Connectivity • Ego-Vision Collaboration (But with Fixed View)
![Page 57: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/57.jpg)
Social Panoramas (ISMAR 2014)
• Capture and share social spaces in real time • Supports independent views into Panorama
![Page 58: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/58.jpg)
Implementation
• Google Glass • Capture live image panorama (compass + camera)
• Remote device (tablet) • Immersive viewing, live annotation
![Page 59: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/59.jpg)
Interface
Glass View
Tablet View
![Page 60: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/60.jpg)
Social Panorama
https://www.youtube.com/watch?v=vdC0-UV3hmY
![Page 61: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/61.jpg)
Lessons Learned
• Good • Communication easy and natural • Users enjoy have view independence • Very natural capturing panorama on Glass • Sharing panorama enhances the shared experience
• Bad • Difficult to support equal input • Need to provide awareness cues
![Page 62: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/62.jpg)
JackIn – Live Immersive Video Streaming
• Jun Rekimoto – University of Tokyo/Sony CSL
![Page 63: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/63.jpg)
JackIn Hardware
• Wide angle cameras – 360 degree video capture • Live video stitching
![Page 64: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/64.jpg)
JackIn Demo
https://www.youtube.com/watch?v=mxFuQcYL4D8
![Page 65: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/65.jpg)
JospehTame – Tokyo Marathon
• Live streaming from Tokyo marathon • http://josephta.me/en/tokyo-marathon/
![Page 66: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/66.jpg)
• zxcz
![Page 67: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/67.jpg)
Capturing Space: Real World Capture
• Hands free AR • Portable scene capture (color + depth) ! Projector/Kinect combo, Remote controlled pan/tilt
• Remote expert annotation interface
![Page 68: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/68.jpg)
Remote Expert View
![Page 69: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/69.jpg)
Example: CoSense (CHI 2015)
• Real time sharing - Emotion, video, and audio • Wearable (sender) – Send emotion and view • Desktop (receiver) - See remote view and emotion
Google Glass e-Health 2.0 board
+
![Page 70: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/70.jpg)
Implementation
Data Capture
Feature Detection
Emotion Recognition
Emotion Representation
Empathic User Interface
Hardware
User Interface
![Page 71: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/71.jpg)
Wearable Interface
• Google Glass + e-Health + Spydroid + SSI • Measure GSR, pulse oxygen, ECG, voice pitch • Share video and audio remotely • Representative emotions sent back to Glass user
!
!
![Page 72: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/72.jpg)
Desktop Interface
![Page 73: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/73.jpg)
Lessons Learned
• Good • System was wearable • Sender and receiver mirrored emotion • Minimal cues provided best experience
• Bad • System delays • Need for good stimulus • Difficult to represent emotion
![Page 74: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/74.jpg)
Gaze and Video Conferencing • Gaze tracking
• Implicit communication cue • Shows intent
• Task space collaboration • HMD + camera + gaze tracker
• Expected Results • Gaze cues reduce need for communication • Allow remote collaborator to respond faster
![Page 75: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/75.jpg)
Equipment
• Custom eye-tracker • Head mounted camera • Head mounted display
![Page 76: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/76.jpg)
Experiment Set Up • Lego assembly • Two assembly areas • Remote expert
![Page 77: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/77.jpg)
Experiment Design
• 4 conditions varying eye-tracking/pointer support • 13 pairs subjects • Measures
• Performance time • Likert scale results, Ranking results, User preference
![Page 78: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/78.jpg)
Task Performance
• Performance Time (seconds)
![Page 79: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/79.jpg)
Ranking
• Median Ranking Values
![Page 80: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/80.jpg)
Key Results • Both the pointer and eye tracking visual cues helped participants to perform significantly faster
• The pointer cue significantly improved perceived quality of collaboration and co-presence
• Eye-tracking improved the collaboration quality, and sense of being focused for the local users, and enjoyment for the remote users
• The Both condition was ranked as the best in user experience, while the None condition was worst.
![Page 81: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/81.jpg)
Empathy Glasses (CHI 2016)
• Combine together eye-tracking, display, face expression • Impicit cues – eye gaze, face expression
+ +
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
![Page 82: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/82.jpg)
AffectiveWear – Emotion Glasses
• Photo sensors to recognize expression • User calibration • Machine learning • Recognizing 8 face expressions
![Page 83: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/83.jpg)
Integrated System • Local User
• Video camera • Eye-tracking • Face expression
• Remote Helper • Remote pointing
![Page 84: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/84.jpg)
System Diagram
• Two monitors on Remote User side • Scene + Emotion Display
![Page 85: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/85.jpg)
Empathy Glasses Demo
![Page 86: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/86.jpg)
Ranking Results
"I ranked the (A) condition best, because I could easily point to communicate, and when I needed it I could check the facial expression to make sure I was being understood.”
Q2: Communication Q3: Understanding partner
HMD – Local User Computer – Remote User
![Page 87: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/87.jpg)
Lessons Learned • Pointing really helps in remote collaboration
• Makes remote user feel more connected
• Gaze looks promising • Shows context of what person talking about • Establish shared understanding/awareness
• Face expression • Used as an implicit cue to show comprehension
• Limitations • Limited implicit cues • Task was a poor emotional trigger • AffectiveWear needs improvement
![Page 88: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/88.jpg)
Empathic VR Environments
• Player and Viewer • Viewer slaved to player
• Share emotional signals • Heart rate, GSR
• Remote affect measuring
![Page 89: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/89.jpg)
Emotion Classification
• Classify emotions from physiological data • 8 emotional states
• Collaboration with Sensaura • http://www.sensauratech.com
![Page 90: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/90.jpg)
Heart Rate for Different VR Apps. • Games: theBlu, BrookHaven, Night Café
quiet scene – scary scene – peaceful scene • Devices: HTV VIVE, Empatica E4
Heart Rate The Blu BrookHaven Night Café
Mean 84.89 BPM 98.23 BPM 73.01 BPM
![Page 91: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/91.jpg)
Sharing VR Experiences
• Player controls viewer position (not view) • Measure and share physiological cues
Viewer
Player
![Page 92: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/92.jpg)
Demo Video
![Page 93: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/93.jpg)
AR and VR for Empathic Computing • VR systems are ideal for trying experiences:
• Strong story telling medium • Provide total immersion/3D experience • Easy to change virtual body scale and representation
• AR systems are idea for live sharing: • Allow overlay on real world view/can share viewpoints • Support remote annotation/communication • Enhance real world task
![Page 94: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/94.jpg)
Looking to the Future
What’s Next?
![Page 95: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/95.jpg)
Scaling Up
• Seeing actions of millions of users in the world • Augmentation on city/country level
![Page 96: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/96.jpg)
AR + Smart Sensors + Social Networks
• Track population at city scale (mobile networks) • Match population data to external sensor data • Mine data for applications
![Page 97: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/97.jpg)
Example: MIT SENSEable City Lab
http://senseable.mit.edu/wikicity/rome/
![Page 98: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/98.jpg)
Example: CSIRO WeFeel Tool
• Emotionally mining global Twitter feeds
• http://wefeel.csiro.au
![Page 99: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/99.jpg)
European Cup – July 10th
GOAL!
![Page 100: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/100.jpg)
Research Challenges
• How to capture emotion?
• How to measure empathy?
• Interface/interaction models?
• How to communicate emotion?
• How to create strong empathic bonds?
• How to scaling up to city/country scale?
![Page 101: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/101.jpg)
Potential Applications • Education • Sports training • Rich life logging • Remote meeting support • Psychological treatments • Virtual Travel/Entertainment • Surrogate Adventure Tourism • First responders (stress, team cohesion)
![Page 102: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/102.jpg)
CONCLUSION
![Page 103: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/103.jpg)
Harvard Grant Study • $20 million, 75 years study
• 268 Harvard graduates • 456 disadvantaged people • Led by George Valliant
• What makes us happy? • warmth of relationships throughout life have the greatest positive impact on "life satisfaction".
![Page 104: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/104.jpg)
“The seventy-five years and twenty million dollars expended on the Grant Study points to a straightforward five-word conclusion: Happiness is love. Full stop.” George Valliant
![Page 105: VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences](https://reader031.vdocuments.site/reader031/viewer/2022022414/586fd9b71a28ab18428b5b11/html5/thumbnails/105.jpg)
Conclusions
• Empathic Computing • Sharing what you see, hear and feel
• AR/VR Enables Empathic Experiences • Removing technology • Changing perspective • Sharing space/experience
• Many directions for future research