1

Enable Your Virtual Reality

& Augmented Reality

Technology on ARM

Nizar Romdan

Director of Ecosystem

Media Processing Group

17th June 2015

2

AR is open and partly immersive – you can see through and around it

AR puts virtual things into users’ real worlds, augmenting them

AR is great fun for games, but maybe not as much fun as VR

AR could be used outdoors for various use cases

VR is closed and fully immersive – you are inside it

VR puts users inside virtual worlds, immersing them

VR is great for games and 3D films, that’s what it was designed for

VR is primarily a living room, office or seated experience

Augmented Reality vs Virtual Reality – Similar or Different?

Augmented Reality (AR) and Virtual Reality (VR) are inherently different!

3

Augmented Reality

4

AR is overlaying location based/user

specific/geographic information on a

real-time camera feed

Several other definitions exist

Current AR techniques tend to be

more CPU focused than GPU

Process the camera feed Feature

Recognition Overlay information

Gravimetric, Fiduciary, Markerless, QR,

Natural Feature

Utilise the power of the Mali GPU to overlay

3D effects on to the view of the real world

Introducing Augmented Reality

5

Augmented Reality Use Cases

GAMING INTUITIVE UI

NAVIGATION, SECURITY SHOPPING, SOCIAL MEDIA, LBS

Applications Registration & Tracking

Featu

re R

eco

gn

itio

n

3D

Gra

ph

ics

6

Augmented Reality Challenges

“True to Life”

3D Graphics

Immersive

User Experience

Battery Life

Responsiveness

Robustness

Relevant Content

Reliable

Network

Ergonomically

Attractive

Fast

Sensors, Tracking

Key

Challenges

Influenci

ng

fac

tors

7

Virtual Reality

8

VR is not new – People were discussing VR the 90’s

Expensive and bulky hardware

Inadequate technology offered limited graphics capabilities

Virtuality – Sega VR – Nintendo Virtual Boy

VR is now accessible

Emergence of low-cost hardware based on existing platforms

Mobile platforms offering console-like performance offering more

realism

VR is about immersing users into Virtual worlds

All the virtual world is graphically rendered

Introducing VR Virtuality

Sega VR

9

VR Use Cases

Games Experiences Training

Education TV/Film Advertising

10

Virtual Reality Segmentation

Samsung Gear VR Zeiss VR One Google Cardboard

Oculus Rift Sony Morpheus HTC Vive (Steam VR)

Mobile

Desktop/Console

11

Headset based on Smartphone

Partnership between Samsung Mobile & Oculus

First version for Galaxy Note 4 (Mali-T760)

Second version for Galaxy S6 (Mali-T760)

Resolution 1080p – 60 Hz

First VR appstore

Introducing GearVR

12

Stereoscopic Display (via headset) – Providing wearer with a sense of depth

Render left eye and right eye independently

Barrel distortion effect applied to rendered output via post processing

Sensor input for head-tracking – To add realism to the output image

Sensor input for interaction (hands-tracking) – To provide an immersive experience

How Virtual Reality Works

Left eye

Right eye

Eye gap (IPD)

Left eye Right eye

Lens

Lens

2x CPU

Processing

2x Vertex

processing

Increased Fragment

Processing

13

Virtual Reality Technical Challenges

14

Mali Well Positioned for Mobile VR Samsung Gear VR is first real VR device (based on Mali-T760)

Mali Bandwidth Reduction Features ASTC & AFBC provide significant advantage

Mali Driver Extensions Minimize latency for mobile VR under development

Mali Ecosystem Engagements with middleware and game studios started

Mali VR SDK Under planning – H2’15 release

Geomerics Enlighten

Best lighting solution for VR

ARM Enabling Mobile VR Experiences

Enlighten Enables Immersive and Realistic VR Experiences

VR requirements:

→90+ fps for best user experience

→High resolution

→One view per eye

→Low latency

Limited GPU resources

for immersive 3D effects

Enlighten computes indirect lighting on the

CPU, independent of GPU

Engine

CPU Light and

colour data

Precompute

data

Enlighten output

Geometry Final scene

GPU

16

ARM VR Ecosystem

VR HARDWARE VR MIDDLEWARE & TOOLS VR CONTENT

Resolution

Games

17

Thank You!

malideveloper.arm.com/cn

Special Thanks to

nDreams & Starship, UK