HoloLens developmentAchmea TechnightApeldoorn, februari 2017

Alexander Meijers

Even kennismaken...• Alexander Meijers• Architect @ ETTU

Apps | Hololens | Azure SharePoint | Office 365 | Dev | Functional |

Architecture Love the Community Speaker | Blogger | 3 kids Fitness |

Airsoft Warhammer 40K

@ameijers alexander@appzinside.com

AgendaSESSIE 1

• History, emplacement and fields of application

• Microsoft HoloLens inside out• HoloLens development• Unity Explained

• Camera location, position and forward

SESSIE 2

• Holographic Objects and the gaze • Gestures and pinning objects• Spatial perception• Meshes to planes

History, emplacement and fields of application

History of HoloLens• Codenamed Project HoloLens• Chief inventor Alex Kipman

• Official title is technical fellow• He dreamed up Kinect in at the end of 2007

and set a vision which incorporated HoloLens

• It is the start of a transforming world• In the new reality, sensors will be anywhere • A visual computing platform controlled by speech and gesture

Satya Nadella - “The emergence of the next computing interface”

Digital workspace“Mixed reality is becoming part of the Smart Workplace”

Modern workplace

Advanced intranet

Traditional intranet

Mail & Calendaring

Mobile access

Document management

Unified Communications

Enterprise Search

Selfservice applicaties

Business proces automation

Office applications

Microblogging

Instant Messaging

Social Networking

CRM

Supply chain management

Extranet collaboration

NewsSocial Business

tools

RSS feeds

Intranet collaboration

HR systems Corporate information

PhonebookPolicies and regulations

Line of Business Systems

Corporate App store

Smart workplaceAugmented/Virtual reality

Internet of Things

Machine Learning

Virtual Digital Assistants

“Augment reality is a live direct or indirect view of a physical,

real-world environment whose elements are augmented by

computer generated sensory input such as sound, video,

graphics and data”

Different approaches

• Augment Reality (AR)• Mixed reality• Interact with the physical, real-

world environment• Virtual Reality (VR)

• No transparency to the physical real-world environment

Fields of application•Enables the user to visualize surrounding which is not directly visible

•Get visual insight of an object or model

•Modelling of non-existing objects

visualization

•Enrich real-world environment objects with information

•Show information as part of the real-world on a wall or table

•Show additional information to objects, persons or activities

Provisioning of information

•Support through linked view, voice and gestures

•Meetings around Holographic objects

•Working together on projects whereby visualization is an important part

•Working together when distance is involved

Working together

•Experience learning •Getting more realistic teaching by moving 2D presentations in books to 3D holographic models

Education

•Holographic media•Enrich media with additional holographic objects

Media & entertainment

•Play games as part of your real-world environment

•Game interaction with your walls, ceiling and floor

Gaming

Detecting electrical wiring

• A company needs to add additional electrical wiring.

• Detailed wiring plans are projected over a wall or ceiling

• Based on detailed plans shown through HoloLens he can decide where to add

Example of visualization

Elevator maintenance

• Detailed information about the elevator through a dashboard

• A 3D representation of parts of the elevator to allow quickly find the problem

• Ability to contact others through phone and mail for help

Example of visualization

Enhance your BI desktop

• Getting better insights of BI data through 3D models

• Extend the desktop with 2D/3D information screens

Example of provisioning of information

Working together on car safety

• Sharing the same view, scenarios and data of a car simulation with sensors

• Getting help from each other and make it easier to explain on what you are working on

Example of working together

Learning about the human body

• Getting a realistic 3D projection of the human body

• It allows easy layering of the body and mark different body parts

• By walking around the projection and viewing and marking body parts will help students to learn more quickly

Example of education

…

• Become part of different squares in Rome

• Rome is viewed as a 360 degree view around you in all directions

• Images are enhanced wih holographic additions like the inside of the church and the fountain in the front.

Example of media and entertainment

Mixed reality during gameplay

• HoloLens allows us to play games which become part of our real-world environment

• Fragments is a CSI game in which you, together with virtual people solve crime delicts

• Virtual people use your room as if there are actually there

Example of gaming

Microsoft HoloLens inside out

Microsoft Hololens• First Holographic computer running on

Windows 10• Contains a CPU, GPU and HPU• Human understanding

• Gesture and voice input• Gaze tracking• Spatial sound

• Uses wi-fi to connect and has no wires attached

• Design for comfort

Microsoft Hololens specifications• Windows 10 device based on

32 bit architecture• 64GB flash • 2GB memory• 1GB Holographic Processor Unity (HPU)

• First of its kind• Device is more powerfull than a laptop

• No overheating due to warm air flows to the sides• 2-3 hours active and 2 weeks standby• Weight 579g

Microsoft Hololens specifications• Contains depth camera

• Field of vision that spans 120 by 120 degrees

• 18 sensors flooding the device with terabytes of data every second

• Tricks your brain into perceiving holographic images• Light engine in which light particles are

bouncing million times• Photons enter the two lenses• Ricochet some between layers of blue,

green and red glass• Hitting back of the eye

Microsoft Hololens specifications• Device can project a

hologram into a room and keep it locked in position• Called pinnnig

• Objects are not moving relative to you but you can move around the object• Allows you to view the

object from any angle

Microsoft Hololens specifications• Device scans your

environment and builds a digital model in real time• It allows HoloLens to see

different surfaces, walls and ceiling

• Possible to simulate a physical space• E.g. Projecting a terain

over your floor

Sensors, optics and speakersEnvironment

camera

High definition lenses

Spatial soundwith

speakers

Depth camera

Video camera

Integration coming up• Windows Insiders

will unexpected see a “Holographic” entry on the main page of Settings

• Fast ring only• Build 15007• It has been removed

from the current builds

HoloLens development

Development tools• Visual Studio 2015 Update 3• Windows 10 SDK (version 1511 or later)

• There is no separate SDK for HoloLens• Unity 5.5 or higher• HoloLens emulator

• HoloLens Emulator (build 10.0.14393.0)• Hyper-V• Contains DirectX project templates for

Visual Studio• HoloLens device

Vuforia• Allows you to develop holographic apps

which recognize specific things in the environment and attach experiences to them• Recognizes specific images and objects

in the environment

• Vuforia engine• Extend the capabilities of your

holographic app• Includes support for OpenGL ES 3.x• Uses Cloud recognition services

Build Lifecycle HoloLens project

• Create Unity Project• Configure HoloLens

settings• Create scene

Unity

• Configure build settings

• Build and generate Visual Studio project

Convert• Open project with

Visual Studio• Pair with HoloLens• Build Visual Studio

project

Visual Studio

• Deploy application• Start Application

HoloLens

Develop for HoloLens• HoloLens Device

• Developer mode needs to be turned on

• Pair the device with your development machine

• Retrieve IP address of HoloLens Device• Ask Cortana• Settings Network & Internet

Advanced options

Windows Device Portal• 3D View• Mixed Reality Capture• Performance

• Performance tracing• System performance• Processes

• Apps• Maintenance• Crash dumps

• Additional tools• Logging• File Explorer• Kiosk Mode• Simulation• Virtual Input

Unity Explained

Unity explained Scene

Main Camera

DirectionalLight

Scripts

Components

Script Coupled as component

Game Objects

Assets

Unity Explained - Prefabs• Predefined Game Objects• Contains components with

predefined values• Prefabs used in Scene

inherit the components and settings

Unity explained – Script for GameObject• Derived from MonoBehaviour• Contains methods which are called

by Unity through the a broadcast messaging system

• void Awake() – Called when class is instantiated

• void Start() – Called when object is enabled once before the first Update()

• void Update() – Called once per frame

• More methods are available

Unity explained – Message systempublic void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options = SendMessageOptions.RequireReceiver);

• Used for calling methods on GameObjects• Calls the methods also on its Children• Examples

• this.BroadcastMessage(“OnSelect”);• focusedObject.BroadcastMessage(“OnSelect”);

Components• Transform

• Position of object• Rotation of object• Scale (size) of object

• Mesh Renderer & Mesh Filter• Renders the object• Mostly done using a mesh

• Mesh Collider• Determines the collision for the object

• Script• Perform actions on the object• Handle messages of the object

Paint 3D• Creates fbx files• Easy to import• Need some additional work to

use them in Unity• Removing nodes• Moving meshes up• Set all components at highest

level> Mesh Render> Mesh Filter> Mesh Collider> Other

http://www.remix3d.com

Positioning Unity and HoloLens

forward (x,y,z)Camera (0, 0, 0)Rotation (x,y,z)

Camera (1, -1, 4)Rotation (x,y,z)

HolographicObject Pinned

HoloLensmoved to

new location

View directionGaze

Positioning Unity and HoloLens

{{

Democamera location,

position and forward

Wat zijn onze conclusies?• Nog geen volwassen product• Integratie met SharePoint Online loopt voorop• Alleen toepassen voor eenvoudige data verwerking

AgendaSESSIE 1

• History, emplacement and fields of application

• Microsoft HoloLens inside out• HoloLens development• Unity Explained

• Camera location, position and forward

SESSIE 2

• Holographic Objects and the gaze • Gestures and pinning objects• Spatial perception• Meshes to planes

{{

DemoHolographic objects

and the gaze

{{

Demogestures and

pinning objects

{{

Demospatial perception

{{

Demomeshes to planes

Developer resources

GitHub development resources• Galaxy Explorer - application build by

Microsoft as part of the Share Your Idea campaign.

• HoloLens Companion Kit - This is a toolkit style repo for code bits and components that *do not run on HoloLens* but pair with HoloLens to build experiences.

• HoloToolkit-Unity – Toolkit for HoloLens and contains Unity specific components

Other resources• Windows Dev Center – Documentation about

learning the fundamentals of holographic development

• Academy – hands on with code, tutorials and lessons

• Windows Holographic Developer Forum – Forum about HoloLens App Development

• Unity tutorials – Learning about the engine, services and production

• Windows blog – Lots of business articles found about HoloLens

• Vuforia – Examples, articles and support on the use of HoloLens with Vuforia

Bedankt voor jullie aandacht! Alexander Meijers, ETTU

<the end/>