designing outstanding ar experiences

DESIGNING OUTSTANDING AUGMENTED REALITY

EXPERIENCES

Mark Billinghurst mark.billinghurst@unisa.edu.au

May 17th 2016

Shenzen, China

•  adsf

•  jg

The Digital Divide

• Screens are windows into digital spaces • Separation between digital and physical world

Augmented Reality

Vuforia Smart Terrain

Augmented Reality Technology that: 1.  Combines Real and Virtual Images

• Both can be seen at the same time

2.  Interactive in real-time • The virtual content can be interacted with

3.  Registered in 3D • Virtual objects appear fixed in space

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

• Web based AR •  Flash, HTML 5 •  Marketing, education

• Outdoor Mobile AR •  GPS, compass tracking •  Viewing Points of Interest in real world

• Handheld AR •  Vision based tracking •  Marketing, gaming

•  Location Based Experiences •  HMD, fixed screens •  Museums, point of sale, advertising

Typical AR Experiences

AR Business Today

• Around $600 Million USD in 2014 (>$2B 2016) • 70-80+% Games and Marketing

Market Projections

cf. 2014 computer game market = $84 Billion USD

DESIGNING AR EXPERIENCES

What Makes a Good AR Experience? • Compelling

• Engaging, ‘Magic’ moment

• Intuitive, ease of use • Uses existing skills

• Anchored in physical world • Seamless combination of real and digital

Example: colAR (Quiver)

• Turn colouring book pages into AR scenes • Markerless tracking, use your own colours..

• Try it yourself: http://www.colARapp.com/

• Interface Components • Physical components • Display elements

• Visual/audio • Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Interface Elements

AR Interface Design • Combines physical + virtual object design • Physical

• Tangible controllers and objects

• Virtual • Virtual graphics and audio

Physical Design •  Industrial Design

• Type of Objects •  Purposely built – affordances

•  “Found” – repurposed

•  Existing – already at use in marketplace

How are these used?

”… the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...]

Affordances provide strong clues to the operations of things.”

(Norman, The Psychology of Everyday Things 1988, p.9)

Affordance

Physical vs. Virtual Affordances • Physical Affordances

•  Look and feel of real objects • Shape, texture, colour, weight, etc •  Industrial Design

• Virtual Affordance •  Look of virtual objects • Copy real objects •  Interface Design

Design of AR Enhanced Objects

• Make affordances obvious

• Object affordances visible

• Give feedback

• Provide constraints

• Use natural mapping

• Use good cognitive model

Example: Haunted Book/AR Book

• Camera hidden in lamp object • AR content seamlessly integrated into real book • Natural page turning/manipulation interaction

Interface Design Path 1/ Demonstration: Working Prototype

2/ Copying: Adoption of Interaction Techniques from other interface metaphors

3/ Creation: Development of new interface metaphors appropriate to the medium

4/ Theory: Development of formal theoretical models for predicting and modeling user actions

Desktop WIMP

Virtual Reality

Augmented Reality

Example: VR Interfaces

• Copying: Virtual Windows/keyboards • Creation: World in Miniature

AR Interaction Metaphors

• AR Lens/Window • simple (conceptually!), unobtrusive

• 3D User Interfaces (VR) • expressive, creative, require attention

• Tangible Interfaces (TUI) • Embedded into conventional environments

• Tangible AR • Combines TUI input + AR display

AR Lens

•  Information is registered to real-world context • Hand held AR displays

•  Interaction •  2D/3D virtual viewpoint control •  Limited input/interactivity

• Applications • Context-aware information

Tourism, gaming

3D AR Interfaces • Virtual objects displayed in 3D physical space and manipulated •  HMDs and 6DOF head-tracking

•  6DOF hand trackers for input

•  Interaction •  Viewpoint control

•  Traditional 3D interaction: •  manipulation, selection, etc.

•  VR techniques

Tangible User Interfaces (Ishii 97)

• Create digital shadows for physical objects

• Foreground • graspable UI

• Background • ambient interfaces

Tangible Interface: Augmented Groove

• Collaborative Instrument • Physically Based Interaction • Map actions to Midi output

• Translation, rotation • Tilt, shake

Lessons from Tangible Interfaces • Advantages

• Physical objects make us smart • Objects aid collaboration • Objects increase understanding

• Disadvantages • Difficult to change object properties •  Limited display capabilities • Separation between object and display

Back to the Real World • AR overcomes limitation of TUIs

•  enhance display possibilities • merge task/display space •  provide public and private views

• TUI + AR = Tangible AR • Apply TUI methods to AR interface design •  TUI for input, AR for output

Tangible AR Design Principles • Tangible AR Interfaces use TUI principles

• Physical controllers for moving virtual content • Support for spatial 3D interaction techniques • Time and space multiplexed interaction • Support for multi-handed interaction • Match object affordances to task requirements • Support parallel activity with multiple objects • Allow collaboration between multiple users

Tangible AR: Tiles • Tiles semantics

• data tiles • operation tiles

• Operation on tiles • proximity • spatial arrangements • space-multiplexed

Poupyrev, I., Tan, D., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001). Tiles: A mixed reality authoring interface. In INTERACT 2001 Conference on Human Computer Interaction (pp. 334-341).

Example: LevelHead

Case Study: LevelHead

• Physical Components •  Real blocks

• Display Elements •  Virtual person and rooms

• Interaction Metaphor •  Blocks are rooms

Design for Technology Limitations • Understand the platforms used and design for limitations

• Hardware, software platforms

• Eg Handheld AR game with visual tracking • Use large screen icons • Consider screen reflectivity •  Support one-hand interaction • Consider the natural viewing angle • Do not tire users out physically • Do not encourage fast actions • Keep at least one tracking surface in view

Art of Defense Game

Seamless Design

• Design to reduce seams in the user experience •  Eg: AR tracking failure, change in interaction mode

• Paparazzi Game • Change between AR tracking to accelerometer input

Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media, and Humanities, p.19-28, October 26-29, 2011

Consider the Whole User

Consider Your User

• Mobile Phone AR User • Probably Mobile • One hand interaction • Short application use • Need to be able to multitask • Use in outdoor or indoor environment • Want to enhance interaction with real world

Social Acceptance

• People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses •  20% of mobile AR browser users experience social issues

• Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)

TAT Augmented ID

Google Glass Anyone?

Designing AR Experiences • Create a compelling experience •  Intuitive and ease of use • Anchor in the real world • Design affordances for physical + virtual elements • Create the appropriate Interaction Metaphor • Design for technology limitations • Consider the whole user (Social, cultural, ..)

RESEARCH DIRECTIONS

Looking to the Future

What’s Next?

Research Enables New Experiences • Gesture/multimodal Input

• Natural user interaction

• Collaborative experiences • Capturing rich communication cues

• Experience capture • Sharing surroundings

• Empathic Computing • Creating understanding

• Etc..

Gesture Based Interaction

• Use free hand gestures to interact • Depth camera, scene capture

• Multimodal input • Combining speech and gesture

HIT Lab NZ Microsoft Hololens

Meta SpaceGlasses

Social Panoramas

• Google Glass • Capture live image panorama (compass + camera)

• Remote device (tablet) •  Immersive viewing, live annotation

Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 303-304). IEEE.

Empathy Glasses

• Combine together eye-tracking, display, face expression •  Implicit cues – eye gaze, face expression

+ +

Pupil Labs Epson BT-200 AffectiveWear

Remote Collboration

• Eye gaze pointer and remote pointing • Face expression display •  Implicit cues for remote collaboration

Holoportation

• Augmented Reality + 3D capture + high bandwidth •  http://research.microsoft.com/en-us/projects/holoportation/

CONCLUSION

Conclusions • AR enables connection back to real world

• Great AR experiences need good design • Physical + virtual components, interaction metaphor • Design for technology limitations • Need to consider whole user

• Many directions for future research • Natural interaction, collaboration, experience capture

www.empathiccomputing.org

@marknb00

mark.billinghurst@unisa.edu.au