ieee vr-searis 2014 keynote - middlevr - philosophy and architecture

Sébastien ‘Cb’ Kuntz @Cb_VRGeek

Philosophy and architectureSEARIS 2014

Plan

• Introduction

• Origin of MiddleVR

• Goals

• Demo

• Philosophy & Architecture

• Conclusion

(French railways)

“Creating Virtual Reality Games:

The Fundamentals”

Founder & President

Sébastien ‘Cb’ Kuntz

Zone hunter (Virtuality-1994)

FRENCH RAILWAYS - SNCF

• Polhemus magnetic tracker, treadmill, 5DT Glove• C++ / OpenSceneGraph / VRPN / OpenAL

FRENCH RAILWAYS - SNCF

VIRTOOLS

VIRTOOLS VR PACK

• VR tools to simplify VR

Virtual reality

Presence in a virtual world

RidiculousIf it’s real VR, you probably

look ridiculous !

• Build hardware

To create and maintain presence,you need to ...

• Build software

• Build user experience

The challenges of creating a VR software

“VR is mainly a software problem”

Jaron Lanier

Device management

Physical devicePhysical device DriverDriver ApplicationApplication

Device management

Tracker 1Tracker 1Driver Tracker Driver Tracker

11

ApplicationApplication

Tracker 2Tracker 2 Driver Driver Tracker 2Tracker 2

Correct perspective

(c) Johnny Lee

Correct perspective

Stereoscopy

Warping & Blending

Warping & Blending

Graphics cards limitations

• Number of outputs

• Processing power

multiple graphics cards

• Pro

• More outputs

• More processing power

• Con

• 3D engines are not optimized for multi-pipe

Cluster

• Pro

• More outputs

• More processing power

• Con

• 3D engines don’t support clustering

• No seams across displays require 3 layers of sync

• 3D objects synchro (Scenelock)

• New images display (Swaplock)

• Left/Right eyes (Genlock)

Cluster

Interactions

• Many different 3D user interactions

• Navigation

• Selection

• Manipulation

• Draw

• System control

• etc.

Interactions are dependent on hardware

Menus

Haptics(force feedback)

Deployment

• Modify application to

• Change drivers

• Change displays

• Change interactions

• Platform for creating interactive 3d applications

• Widely used

• 2.5 million registered developers

• High-quality rendering

• Deferred renderer

• Occlusion culling

• Realtime shadows

• Ambient occlusion

• Light mapping

Unity 3D Features• Programming

• C#, Javascript

• Profiler

• Physics

• NVidia PhysX

• Cloth

• Soft/rigid bodies

• Ragdolls

• Cars

3D formats

FBX, Collada, 3DS, OBJ…

Audio engine

Networking

Terrain

Editor

• Simple to use

• Create your own tools inside Unity

Asset store

• Buy / sell assets

• 3D models

• Textures

• Shaders

• Scripts

• Editor plugins

DesktopDesktop

• Build hardware

To create and maintain presence,you, the author, need to ...

• Build software

• Build user experience

Existing VR software

• VRPN

• OpenTracker

• VRJuggler

• Virtools

• Worldviz

• …

VRPN

VR Juggler

VIRTOOLS VR PACK

Worldviz Vizard

• Based on OpenSceneGraph

• Python

Worldviz Vizard

The need for betterVR software tools

• Hardware getting cheaper

• But engineers are not getting cheaper

• Need to be more efficient and spend less time creating apps

• Democratization

• VR developers are not all engineers anymore

Goals

• Simplify VR

• Simplify creation of VR applications

• Simplify deployment of VR applications

• Create new VR tools

• Adapting existing 3D applications

What is ?

• Generic VR plugin (C++ SDK)

• Functionalities○Manage input devices

•3D trackers, keyboard, mouse, joystick○Manage display

•Viewports, cameras, stereoscopy, cluster○High-level building-blocks

○Interactions○Menu

Architecture

MiddleVR (C++)MiddleVR (C++)

DriversDrivers OpenGLOpenGL DirectDirectXX

NetworNetworkk

C# WrapperC# Wrapper

GUIGUI UnityUnity

Lua Lua WrapperWrapper

Lua Lua WrapperWrapper

Python Python WrapperWrapperPython Python

WrapperWrapper

MiddleVR Features

Supported Devices• Via VRPN :

• A.R.T• Vicon• Optitrack• Wiimote

• Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers,

• Ascension Flock of Bird

• Intersense IS600, IS900

• 3dTech HiBall-3000 Wide Area Tracker,

• Advanced Realtime Tracking Gmbh DTrack Client,

• World Viz Precision Position Tracker PPT 1.2,

• Natural Point Optitrack Rigid Body Toolkit,

• Phasespace optical tracking OWL,

• ...

• Native drivers• Oculus Rift• Leap Motion• Kinect (Microsoft SDK)• PNI SpacePoint Fusion• GameTrak• Razer Hydra• SpaceMouse• Motion Analysis (beta)

Haptics(force feedback)

VR Camera• MiddleVR computes all the cameras’

parameters automatically

• Correct perspective

• Scale 1

Stereoscopy

• Active stereoscopy (OpenGL Quad-Buffer)

• Which Unity can’t natively handle

• Passive stereoscopy

Viewports management

Simply configure your system

Navigations

Homography

Cluster & Multi-pipe

• Scenelock

• Swaplock

• Genlock

• Multi-pipe (soon)

Supported displays

• Virtually any HMD• VR-Wall, Powerwall• Workbench, Holobench• HoloStage• CAVEs• 3D TVs• zSpace• Igloo Vision• …

Goals

• Simplify creation of VR applications

• Simplicity of adding VR into application

• Simplicity of API

• High-level tools

• Interactions: Selection, manipulation, navigation, menus, GUIs

Example of device management

Driver 1

Driver 2

Device management

MiddleVR

Class reference

The value of Simplicity

The value of simplicity

“Perfection is achieved, not when there is nothing more to add, but

when there is nothing left to take away”

Antoine de St Exupery

The value of simplicity

• For each feature you add

• Test

• Document

• Support / maintain

• Each feature costs !!

The value of simplicity• Huge value in simplicity !

• User side

• Faster to learn, less to remember

• Developer side

• Faster to code

• Easier maintain

• Less bugs

• Business side

• Need less coders

• More interesting features

• Easy download / setup > Quickly validate on user’s hardware !

• Easier support

API in simple english• Long discussions for naming of functions, classes, variables

• Consistent api

• SetNumButtons, SetButtonsNum, SetNbButtons > SetButtonsNb

• French not good at english

• Analog ? Digital ? Proxy ?

• Use natural terms depending on device

• Joystick->GetAxisValue()

• Joystick->IsButtonPressed()

• Keyboard->IsKeyPressed()

• Why “Register” ? Why not “Add” ?

• Why so complex, want to sound very clever ?

• AddDevice, AddCamera, …

• API in degrees, because more intuitive / simple to work with

Simplified API

• Simple but complete API

• Like legos

• Rotate

• Node3D->Rotate( Quat )

• Node3D->Rotate( Euler )

• Node3D->Rotate( Axis, Angle)

• Quat( quat ), Quat( Euler ), Quat( Axis, Angle)

The value of simplicity

“Simplicity is the ultimate sophistication”

DaVinci

The value of simplicity• Design user interface for the 80%

• Don’t ruin the experience for the 20%

• Simple things should be simple

• Complex things can be complex

• Put in an “advanced menu”

• Don’t let beauty of design ruin user experience

• Like real life architects who want their design to be beautiful but totally unusable on a day to day

• Example of Lego

• Simple but complete

Optimizations

• Engineers have a strong tendency to over-optimise

• Complexifies code

• Code must be above all readable

• To avoid bugs

• To maintain maintenability

• If you want to optimise

• First profile your code

• Optimize what is *really* taking time

• not what you *think* takes time

Empathy

“The ability to understand and share the feelings of another.”

Test with users

Hardware SDK are mostly too complicated

Drivers• Why not just a few simple functions with

sensible defaults ?

• Init()

• Update()

• GetValue()

• Destroy()

• And add more functions for more advanced

Documentation

• User Guide

• Class references

Goals

• Simplify deployment of VR applications

• GUI to configure VR systems

• Dynamic reconfiguration of application

• Ship with standard configurations

• Abstractions

Deployment

• Abstractions

• Devices

• 3D Nodes (User)

• Interactions (soon)

Device management

Driver Tracker Driver Tracker 11

ApplicationApplication

Driver Driver Tracker 2Tracker 2

Device abstraction

Virtual Virtual DeviceDevice

ApplicationApplication

Driver Tracker Driver Tracker 11

Driver Driver Tracker 2Tracker 2

Device management

DriverDriver

Virtual Virtual DeviceDevice

ApplicationApplication

SimulatioSimulationn

ReplayReplay

Display

• Cameras & viewport created dynamically

User• Human-centric rather than techno-

centric

Virtual Device Virtual Device 11

HandHand(3D Node)(3D Node) ApplicationApplication

Virtual Device Virtual Device 22

Inverse Inverse KinematicsKinematics

User abstraction

Interactions• MiddleVR includes high-level

interactions

• Navigation

• Selection

• Manipulation

• … More to come

• Future: Interactions abstractions

• Dynamically change interactions based on hardware

Interactions abstraction?

Grab-airGrab-air

NavigationNavigation

ApplicationApplication

PointPoint

SteeringSteering

SelectionSelection

ManipulationManipulation

RaycastinRaycastingg

Go-goGo-go

HandHand

HomerHomer

HandHand

WIMWIM

• How to create portable interactions ?

• Navigations can be portable

• Except collisions

• Generalise what is generalisable

• + Custom part for engine dependent

• Collisions

• Highlights

• Gizmos

Interactions

Menus• Ability to create menus and GUIs in HTML5

(soon)

Create immersive HTML5 GUI

C++ Reflexion

• Also for configuration file

• Cluster synchronisation

• Accessing drivers specific information

Working with multiple 3D engines

• Different coordinate systems

• vrCoordinateSystem( Front, Up, Right )

• +/- X,Y,Z

• OpenGL:

• Front= -Z

• Up = +Y

• Right = +X

• DirectX:

• Front=+Z

• Up=+Y

• Right=+X

Working with multiple 3D engines

• C++

• Compiling

• STL, Boost

• Strings (Wide strings)

• Link issues

• Different compilers (VC2005/2008/2010/2012)

• Different languages

• C wrapper

• Lua wrapper

• Simplicity• Creation• Deployment

•Adaptability•Hardware• Software

Our future

• Games/apps development:

ConclusionConclusion

http://www.imin-vr.com/download

Thank you !Questions ?

Sebastien.Kuntz@imin-vr.com@Cb_VRGeek

@imin_vr