levels of realism: from virtual reality to real virtuality - perception and virtual time

© 2006 WMG

© 2007 WMG

Levels of Realism: From Virtual Reality to Real

Virtuality - Perception and Virtual Time Alan ChalmersAlan Chalmers† Andrej Ferko Andrej Ferko‡

† Warwick Digital Laboratory, UKWarwick Digital Laboratory, UK‡ Comenius University, SlovakiaComenius University, Slovakia

2Warwick Digital LabWarwick Digital Lab© 2007 WMG

““Holy Grail”Holy Grail”

Realism in Real-Time Real-Time ≥ 25 fps

Realism …?

Courtesy of recomCGI and Steffen Schraegle, produced via SpheronVR Technology


IntroductionIntroduction

Virtual reality

1960’s

• Morton Heilig’s Sensorama

• Ivan Sutherland’s head mounted display

Modern multi-modal VR systems

• CAVE’s, data-gloves, etc.

Needs real-time

Trade off realism to achieve this

GPUs make things faster

• Not physically based


Types of realismTypes of realism

1998: Lengyel’s rendering spectrum

Images which are appearance based

Geometry which is physically based

2001: Ferwerda

Physical realism

• Visual simulation

Photorealism

• Visual response

Functional realism

• Visual information


Physical realismPhysical realism

Exact match

Spectral irradiance values

Match at all points within the real and virtual scene

Scene modelled precisely

Geometry

Materials

Light transport

BUT

All include approximations

Especially displays (even HDR is only 3,000 cd/m2)


PhotorealismPhotorealism

Resultant rendered image

Indistinguishable from photograph of real scene

Other forms

Lumigraph [Gortler et al. 1996]

Image based lighting [Debevec et al. 1996]

But

Make rendered images more like a photograph

Blurring

Increasing Gaussian noise


Being thereBeing there


Functional realismFunctional realism

Image provides

Sufficient information to enable viewer to complete task

Correct relative size and shape

But not material properties


Levels of realismLevels of realism

One-to-one mapping

Virtual world to Real world experience

Adopt same learning strategy

Real and virtual worlds

High confidence in results


Believable realismBelievable realism

Believable Realism!


Scruffy texturesScruffy textures


ClassificationClassification

Is PhysicsIP

Not PhysicsNP

Not BelievableNB

Is BelievableIB

Real virtuality

Photorealism

IBNP IBIP

NBNP NBIPAvatars

Uncanny valley

Cartoons NPR

Photographs

Games

Virtual Reality

Functional realism

Abstract art

Vermeer


Comparing real and virtual scenesComparing real and virtual scenes

Physical test environment with diffuse materials

[Meyer et al. 1986]

Comparing photograph with real scene

[Rushmeier et al. 1995]

Visual Difference Predictor (VDP)

[Daly 1993], [Myszkowski 1998], [Mantiuk et al. 2005]

Karol’s new method

Myszkowski et al. @ SIGGRAPH 2008


Holistic approachHolistic approach

Three different human visions

Monocular, binocular, and ambient

Four parallel processes

Head, eye, visual perception, cognitive processes

Visual equivalence

[Ramanarayanan et al. 2007]

Judgement of lightness

[McNamara et al. 2000]


Multi-modalitiesMulti-modalities

Human perceives environment with all senses

Sight

Sound

Smell

Touch

Taste

Crossmodal effects can be considerable


PresencePresence

Typically a measure of technical immersion

Higher quality more immersion

But

Immersion in books and films

Can quantify user’s engagement with VE

Not a measure of realism


PreconditioningPreconditioning

Human imagination

Dungeons & Dragons

Virtual Vietnam

Habituation

Familiarity may make user perceive less


Nature of taskNature of task

Top down process

Attention under volitional control

What we perceive is strongly dependent on the task

Looking for a street sign

Finding a coffee shop

On patrol in (hostile) environment


There-realityThere-reality

Same perceptual response as if “there” in real world

Exploits limits of human perception

Only render in high quality what user is attending to

Selective rendering

Benefits from crossmodal interactions

Can save significant computational effort

Experience

Active – infinitive limited by technology

Passive


The perception equationThe perception equation

Function of task (t) and preconditioning (ρ):

P(t, ρ) = ωv V + ωa A + ωs S + ωt T + ωf F

V=Visuals, A=Audio, S=Smell, T=Taste, F=Feel

Σωi = 1

ωi are threshold values

Above threshold there is no perceptual difference


ExampleExample

Preconditioning

Experienced sailor

Sound of water across hull

Feel of wind on skin


Other approachesOther approaches

Eg. Neurolinguistic programming (NLP)

Sensory input channels

Visual

Audio

Kinesthetic

Olfactory

Gustatory

Can be used to represent eg. desire, emotions …

Initially: Interpersonal communication model

Now: Business and management training


Virtual time issueVirtual time issue

Perceptual equation re-read…

Sensory input channels in real and virtual time (Borges, 2 past)

Visual

Audio

Kinesthetic

Olfactory

Gustatory

Higher level of perception… VR as a semiotic system

How to define and measure interestingness?

Up to now: data mining, Koestler, engagement in virtual museums


ConclusionsConclusions

In virtual reality striving to achieve

More realism

While maintaining real-time performance

Many applications don’t need physical realism

For those that do

May not be possible to compute total physical realism

Perceptual realism

• Same user response as if there in real scene being portrayed

• Can save significant computation by exploiting human perception


ChallengesChallenges

Delivery system

Virtual cocoon (light weight, portable system)

Determine the “weights” ωi in the perception equation

Empirical studies

fMRI brain imaging

Weights are individual

Some generic similarity

Do NOT have to be determined precisely

As long as we are ABOVE the thresholdPerceptually high-fidelity virtual environments

Real virtuality


AcknowledgementsAcknowledgements

Pavel Zemcik and MEMICS team for inviting

SCCG for enabling long-term discussions

EPSRC for funding “…towards Real Virtuality”

Alan’s group for 23 years of exciting research

David Howard + Christopher Moir + red wine Real Virtuality

Further details:

[email protected]

[email protected]


Levels of Realism: From Virtual Reality to Real

Virtuality - Perception and Virtual Time Alan ChalmersAlan Chalmers† Andrej Ferko Andrej Ferko‡

† Warwick Digital Laboratory, UKWarwick Digital Laboratory, UK‡ Comenius University, SlovakiaComenius University, Slovakia

levels of realism: from virtual reality to real virtuality - perception and virtual time

Documents