levels of realism: from virtual reality to real virtuality - perception and virtual time

© 2006 WMG

© 2007 WMG

Levels of Realism: From Virtual Reality to Real

Virtuality - Perception and Virtual Time Alan ChalmersAlan Chalmers† Andrej Ferko Andrej Ferko‡

† Warwick Digital Laboratory, UKWarwick Digital Laboratory, UK‡ Comenius University, SlovakiaComenius University, Slovakia

2Warwick Digital LabWarwick Digital Lab© 2007 WMG

““Holy Grail”Holy Grail”

Realism in Real-Time Real-Time ≥ 25 fps

Realism …?

Courtesy of recomCGI and Steffen Schraegle, produced via SpheronVR Technology

3Warwick Digital LabWarwick Digital Lab© 2007 WMG

IntroductionIntroduction

Virtual reality

1960’s

• Morton Heilig’s Sensorama

• Ivan Sutherland’s head mounted display

Modern multi-modal VR systems

• CAVE’s, data-gloves, etc.

Needs real-time

Trade off realism to achieve this

GPUs make things faster

• Not physically based

4Warwick Digital LabWarwick Digital Lab© 2007 WMG

Types of realismTypes of realism

1998: Lengyel’s rendering spectrum

Images which are appearance based

Geometry which is physically based

2001: Ferwerda

Physical realism

• Visual simulation

Photorealism

• Visual response

Functional realism

• Visual information

5Warwick Digital LabWarwick Digital Lab© 2007 WMG

Physical realismPhysical realism

Exact match

Spectral irradiance values

Match at all points within the real and virtual scene

Scene modelled precisely

Geometry

Materials

Light transport

BUT

All include approximations

Especially displays (even HDR is only 3,000 cd/m2)

6Warwick Digital LabWarwick Digital Lab© 2007 WMG

PhotorealismPhotorealism

Resultant rendered image

Indistinguishable from photograph of real scene

Other forms

Lumigraph [Gortler et al. 1996]

Image based lighting [Debevec et al. 1996]

But

Make rendered images more like a photograph

Blurring

Increasing Gaussian noise

7Warwick Digital LabWarwick Digital Lab© 2007 WMG

Being thereBeing there

8Warwick Digital LabWarwick Digital Lab© 2007 WMG

Functional realismFunctional realism

Image provides

Sufficient information to enable viewer to complete task

Correct relative size and shape

But not material properties

9Warwick Digital LabWarwick Digital Lab© 2007 WMG

Levels of realismLevels of realism

One-to-one mapping

Virtual world to Real world experience

Adopt same learning strategy

Real and virtual worlds

High confidence in results

10Warwick Digital LabWarwick Digital Lab© 2007 WMG

Believable realismBelievable realism

Believable Realism!

11Warwick Digital LabWarwick Digital Lab© 2007 WMG

Scruffy texturesScruffy textures

12Warwick Digital LabWarwick Digital Lab© 2007 WMG

ClassificationClassification

Is PhysicsIP

Not PhysicsNP

Not BelievableNB

Is BelievableIB

Real virtuality

Photorealism

IBNP IBIP

NBNP NBIPAvatars

Uncanny valley

Cartoons NPR

Photographs

Games

Virtual Reality

Functional realism

Abstract art

Vermeer

13Warwick Digital LabWarwick Digital Lab© 2007 WMG

Comparing real and virtual scenesComparing real and virtual scenes

Physical test environment with diffuse materials

[Meyer et al. 1986]

Comparing photograph with real scene

[Rushmeier et al. 1995]

Visual Difference Predictor (VDP)

[Daly 1993], [Myszkowski 1998], [Mantiuk et al. 2005]

Karol’s new method

Myszkowski et al. @ SIGGRAPH 2008

14Warwick Digital LabWarwick Digital Lab© 2007 WMG

Holistic approachHolistic approach

Three different human visions

Monocular, binocular, and ambient

Four parallel processes

Head, eye, visual perception, cognitive processes

Visual equivalence

[Ramanarayanan et al. 2007]

Judgement of lightness

[McNamara et al. 2000]

15Warwick Digital LabWarwick Digital Lab© 2007 WMG

Multi-modalitiesMulti-modalities

Human perceives environment with all senses

Sight

Sound

Smell

Touch

Taste

Crossmodal effects can be considerable

16Warwick Digital LabWarwick Digital Lab© 2007 WMG

PresencePresence

Typically a measure of technical immersion

Higher quality more immersion

But

Immersion in books and films

Can quantify user’s engagement with VE

Not a measure of realism

17Warwick Digital LabWarwick Digital Lab© 2007 WMG

PreconditioningPreconditioning

Human imagination

Dungeons & Dragons

Virtual Vietnam

Habituation

Familiarity may make user perceive less

18Warwick Digital LabWarwick Digital Lab© 2007 WMG

Nature of taskNature of task

Top down process

Attention under volitional control

What we perceive is strongly dependent on the task

Looking for a street sign

Finding a coffee shop

On patrol in (hostile) environment

19Warwick Digital LabWarwick Digital Lab© 2007 WMG

There-realityThere-reality

Same perceptual response as if “there” in real world

Exploits limits of human perception

Only render in high quality what user is attending to

Selective rendering

Benefits from crossmodal interactions

Can save significant computational effort

Experience

Active – infinitive limited by technology

Passive

20Warwick Digital LabWarwick Digital Lab© 2007 WMG

The perception equationThe perception equation

Function of task (t) and preconditioning (ρ):

P(t, ρ) = ωv V + ωa A + ωs S + ωt T + ωf F

V=Visuals, A=Audio, S=Smell, T=Taste, F=Feel

Σωi = 1

ωi are threshold values

Above threshold there is no perceptual difference

21Warwick Digital LabWarwick Digital Lab© 2007 WMG

ExampleExample

Preconditioning

Experienced sailor

Sound of water across hull

Feel of wind on skin

22Warwick Digital LabWarwick Digital Lab© 2007 WMG

Other approachesOther approaches

Eg. Neurolinguistic programming (NLP)

Sensory input channels

Visual

Audio

Kinesthetic

Olfactory

Gustatory

Can be used to represent eg. desire, emotions …

Initially: Interpersonal communication model

Now: Business and management training

23Warwick Digital LabWarwick Digital Lab© 2007 WMG

Virtual time issueVirtual time issue

Perceptual equation re-read…

Sensory input channels in real and virtual time (Borges, 2 past)

Visual

Audio

Kinesthetic

Olfactory

Gustatory

Higher level of perception… VR as a semiotic system

How to define and measure interestingness?

Up to now: data mining, Koestler, engagement in virtual museums

24Warwick Digital LabWarwick Digital Lab© 2007 WMG

ConclusionsConclusions

In virtual reality striving to achieve

More realism

While maintaining real-time performance

Many applications don’t need physical realism

For those that do

May not be possible to compute total physical realism

Perceptual realism

• Same user response as if there in real scene being portrayed

• Can save significant computation by exploiting human perception

25Warwick Digital LabWarwick Digital Lab© 2007 WMG

ChallengesChallenges

Delivery system

Virtual cocoon (light weight, portable system)

Determine the “weights” ωi in the perception equation

Empirical studies

fMRI brain imaging

Weights are individual

Some generic similarity

Do NOT have to be determined precisely

As long as we are ABOVE the thresholdPerceptually high-fidelity virtual environments

Real virtuality

26Warwick Digital LabWarwick Digital Lab© 2007 WMG

AcknowledgementsAcknowledgements

Pavel Zemcik and MEMICS team for inviting

SCCG for enabling long-term discussions

EPSRC for funding “…towards Real Virtuality”

Alan’s group for 23 years of exciting research

David Howard + Christopher Moir + red wine Real Virtuality

Further details:

alan.chalmers@warwick.ac.uk

ferko@sccg.sk

27Warwick Digital LabWarwick Digital Lab© 2007 WMG

Levels of Realism: From Virtual Reality to Real

Virtuality - Perception and Virtual Time Alan ChalmersAlan Chalmers† Andrej Ferko Andrej Ferko‡

† Warwick Digital Laboratory, UKWarwick Digital Laboratory, UK‡ Comenius University, SlovakiaComenius University, Slovakia