hololens inspiration session

ETTU

Inspiration session

HoloLens

Agenda• About realities

• Microsoft HoloLens device

• Fields of application

• Overview

• Input controls

• Sound

• Spatial perception & understanding

• Other cool stuff

• Data & Cloud

• Design considerations

Which reality?

Generation of realistic images,

sounds and other sensations

that replicate a real environment

or create an imaginary setting

An immersive experience created

entirely from computer-generated

Content. Also similar to 360 degree video

Virtual Reality

Direct or indirect view of a physical,

real-world environment

whose elements are augmented

by computer-generated sensory input

such as sound, video, graphics or GPS data

An overlay of content on the real world where

that content is not anchored to or part of it

Augmented Reality

Merging of real and virtual worlds

to produce new environments

and visualizations where physical

and digital objects co-exist

and interact in real time.

Overlay of synthetic content on the real world

that is anchored to and interacts with the real world

Mixed Reality

Immersive headsets

• Everything becomes Mixed Reality

• SteamVR

• AltSpaceVR

Now and the future!

October 2016

HoloLens v1

October/ November 2017

Immersive headsets

2019HoloLens v3

Microsoft HoloLens device

History of HoloLens

• Codenamed Project HoloLens

• Chief inventor Alex Kipman• Official title is technical fellow

• He dreamed up Kinect in at the end of 2007 and set a vision which incorporated HoloLens

• It is the start of a transforming world

• In the new reality, sensors will be anywhere

• A visual computing platform controlled by speech and gesture

Satya Nadella - “The emergence of

the next computing interface”

Microsoft Hololens

• Windows 10 device

• First Holographic computer running on Windows 10

• Contains a CPU, GPU and HPU

• Human understanding• Gesture and voice input

• Gaze tracking

• Spatial sound

• Uses wi-fi to connect and has no wires attached

• Design for comfort

Sensors, optics and speakers

Environment

camera

High definition

lenses

Spatial sound

with

speakers

Depth

camera

Video

camera

Demo

Holograms

Fields of application

Fields of application•Enables the user to visualize

surrounding which is not directly

visible

•Get visual insight of an object or

model

•Modelling of non-existing objects

visualization

•Enrich real-world environment

objects with information

•Show information as part of the

real-world on a wall or table

•Show additional information to

objects, persons or activities

Provisioning

of information

•Support through linked view, voice

and gestures

•Meetings around Holographic

objects

•Working together on projects

whereby visualization is an

important part

•Working together when distance is

involved

Working

together

•Experience learning

•Getting more realistic teaching by

moving 2D presentations in books

to 3D holographic models

Education

•Holographic media

•Enrich media with additional

holographic objects

Media &

entertainment

•Play games as part of your real-

world environment

•Game interaction with your walls,

ceiling and floor

Gaming

• A company needs to add additional

electrical wiring.

• Detailed wiring plans are projected

over a wall or ceiling

• Based on detailed plans shown

through HoloLens he can decide

where to add

Example of visualization

Detecting

electrical wiring

• Detailed information about the elevator through a dashboard

• A 3D representation of parts of the elevator to allow quickly find the problem

• Ability to contact others throughphone and mail for help

Example of visualization

Elevator

maintenance

• Getting better insights of BI data

through 3D models

• Extend the desktop with 2D/3D

information screens

Example of provisioning of

informationEnhance your BI

desktop

• Sharing the same view, scenarios

and data of a car simulation with

sensors

• Getting help from each other and

make it easier to explain on what

you are working on

Example of working together

Working

together on

car safety

• Getting a realistic 3D projection of the human body

• It allows easy layering of the body and mark different body parts

• By walking around the projection and viewing and marking body parts will help students to learn more quickly

Example of education

Learning

about the

human body

• Become part of different squares in Rome

• Rome is viewed as a 360 degreeview around you in all directions

• Images are enhanced wihholographic additions like the inside of the church and the fountain in the front.

Example of media and

entertainment…

• HoloLens allows us to play games

which become part of our real-world

environment

• Fragments is a CSI game in which

you, together with virtual people

solve crime delicts

• Virtual people use your room as if

there are actually there

Example of gaming

Mixed reality

during

gameplay

Demo

HoloMaps

Input controls

Gaze

• The direction of the

HoloLens pointing is called

the gaze

• Hits an object or part of

the environment

• Uses a Vector3 based value

to define its so called

“forward” direction

Gestures

• Recognize input by tracking the position of either or both hands

• Gesture frame

• Recognized input from hands• Bloom

• Press, hold and release

• Follow the hand (my own )

• Gestures• Hold

• Manipulation

• Navigation

Voice

• Use words or sentences to

control your environment

• English language only at the

moment

• Phrase recognition using

Language Understanding

Intelligent Service (LUIS)

Spatial Sound

• Simulates 3D sound using direction, distance and environmental simulations

• Above, below, behind, to the side, etc.

• Attach sound to holographic objects

• Works also when object is not in line of sight

• Used to draw attention

• Audio engine in HoloLens

• CPU and memory considerations

• 10-12 spatial sound voices

Spatial Sound

• Gaze mixing• Highlighting objects

• Play a sound on the object to get the users attention

• Audio Haptics• Reactive audio for touchless

interactions

• Play a sound when user selects an object or when his hands appear inside the gesture frame

• Immersion• Ambient sounds surrounding the user

• Support your scene with background sounds or music

Cortana

• Start with “Hey Cortana”

• Some thing you can say

• Move <app name> here

• Take a picture

• Start or stop recording

• Call <contact>

• How much battery do i have left?

• Restart or Shutdown

• Cortana can be turned on/off

Spatial Perception & understanding

Spatial perception

• Spatial perception and

mapping

• Rooms

• Different ways of detecting

and working with spatial

perception

• Meshes & planes

Spatial perception is the

ability to be aware of your

relationships with the

environment around you

and with yourself

Spatial perception

• Device scans your

environment and builds a

digital model in real time

• It allows HoloLens to see

different surfaces, walls

and ceiling

• Possible to simulate a

physical space

• E.g. Projecting a terain

over your floor

Demo

Scanning & building

Pinning

• Device can project a hologram into a room and keep it locked in position

• Called pinnnig

• Objects are not moving relative to you but you can move around the object

• Allows you to view the object from any angle

Demo

Spatial Design Concept

Other cool stuff

Vuforia

• Allows you to develop holographic apps which recognize specific things in the environment and attach experiences to them• Recognizes specific images and

objects in the environment

• Connects holograms to targets

• Vuforia engine• Extend the capabilities of your

holographic app

• Targets can be stored on Device or in the Cloud

Sharing holographic experiences

• Two or more people are

experiencing the same

holographic objects

• Accomplished to anchor sharing

HoloLens device editions comparison

• Kiosk mode• Limiting running apps to enable

demo or showcase experiences

• MDM• Manage Multiple HoloLens devices

through solutions like Microsoft Intune

• Business Store Portal• Private enterprise store for your

companies apps

• Secure boot• Only boot software which is trusted

for your organization

• Windows Update for Business• Controlled OS updates to device

Data & Cloud

Data storage

• Only allowed to store files to a number of local folders

• Save or open from another App like OneDrive

• HoloLens does not support connecting external hard drives or SD Cards

• There is no File Explorer application

• You could use the Windows Device Portal

Cloud

• Device supports connectivity to a broad set of service

• Azure services> Cognitive services

> AI / Bots

> Custom services

• Office 365> Microsoft Graph

> SharePoint Online

• Other> Bing

> Custom

Design considerations

User perspective

• Ensure the application is comfortable to the user

• Avoid abrupt movements

• Allow them to see the world in the background

• Animation from down, left or right or fade in

• Move content slowly and smoothly toward the user

• Keep a minimal distance of 3 meters

Design patterns for

mixed reality

User perspective

• Design your application from

an user’s unique point of view

• Determine if the user moves

around

• User is sitting down or

standing

• The user is the camera!

• Let them control the

movement

Design patterns for

mixed reality

Holographic frame

• Use the center of the holographic frame for your holographic objects

• Help the user through guideness

• Arrows pointing to a specific location

• Light trails

• Pointers

• Spatial sound

• Voice prompts

• Make sure your holograms fit the frame

• They will feel natural Design patterns for

mixed reality

Thank you!

Alexander Meijers

<the end/>

hololens inspiration session

Technology