User Issues in 3D TV & Cinema

Martin S. BanksVision Science Program

UC Berkeley

Issues in 3D TV & Cinema

Technical Issues• Developing content• Sufficient resolution over time: temporal aliasing• Sufficient separation between two eyes’ images: “ghosting”

User Issues• Perceptual distortions due to incorrect viewing position• Flicker & motion judder due to temporal sampling• Maintaining depth across scene cuts• Window violations• Residual ghosting• Visual discomfort due to vergence-accommodation conflict• Appropriate blur relative to other depth signals• Conflict between visually-induced motion & vestibular signals

Technical Issues• Developing content• Sufficient resolution over time: temporal aliasing• Sufficient separation between two eyes’ images: “ghosting”

User Issues• Perceptual distortions due to incorrect viewing position• Flicker & motion judder due to temporal sampling• Maintaining depth across scene cuts• Window violations• Residual ghosting• Visual discomfort due to vergence-accommodation conflict• Appropriate blur relative to other depth signals• Conflict between visually-induced motion & vestibular signals

Issues in 3D TV & Cinema

Technical Issues• Developing content• Sufficient resolution over time: temporal aliasing• Sufficient separation between two eyes’ images: “ghosting”

User Issues• Perceptual distortions due to incorrect viewing position• Flicker & motion judder due to temporal sampling• Maintaining depth across scene cuts• Window violations• Residual ghosting• Visual discomfort due to vergence-accommodation conflict• Appropriate blur relative to other depth signals• Conflict between visually-induced motion & vestibular signals

Issues in 3D TV & Cinema

Fo

cal d

ista

nce

Ve

rge

nce

dis

tan

ce

Vergence & Accommodation: Natural Viewing

3 6

1.5

3

4.5

6

Vergence Distance (diopters)

00

Fo

cal D

ista

nce

(d

iop

ters

)

Fo

cal d

ista

nce

Ve

rge

nce

dis

tan

ce

1.5 4.5

Vergence & Accommodation: Natural Viewing

3 6

1.5

3

4.5

6

Vergence Distance (diopters)

00

zone of clear singlebinocular vision

Fo

cal D

ista

nce

(d

iop

ters

)

Fo

cal d

ista

nce

Ve

rge

nce

dis

tan

ce

1.5 4.5

Vergence & Accommodation: Natural Viewing

3 6

1.5

3

4.5

6

Vergence Distance (diopters)

00

Percival's zoneof comfort

zone of clear singlebinocular vision

Fo

cal D

ista

nce

(d

iop

ters

)

Fo

cal d

ista

nce

Ve

rge

nce

dis

tan

ce

1.5 4.5

Vergence & Accommodation: Natural Viewing

Fo

cal d

ista

nce

Ve

rge

nce

dis

tan

ce

Vergence & Accommodation: Stereo Display

3 6

1.5

3

4.5

6

Vergence Distance (diopters)

00

Fo

cal D

ista

nce

(d

iop

ters

)

1.5 4.5

Fo

cal d

ista

nce

Ve

rge

nce

dis

tan

ce

Vergence & Accommodation: Stereo Display

Percival's zoneof comfort

zone of clear singlebinocular vision

Displays with Nearly Correct Focus Cues

Two multi-focal displays we’ve developed:

1.Fixed-viewpoint, volumetric display with mirror system & 3 focal planes (Akeley, Watt, Girshick, & Banks, SIGGRAPH, 2004).

2.Fixed-viewpoint, volumetric display with switchable lens & 4 focal planes (Love, Hoffman, Kirby, Hands, Gao, & Banks, Optics Express, 2009)

Multi-focal Display

Akeley, Watt, Girshick & Banks (2004), SIGGRAPH.

Akeley, Watt, Girshick & Banks (2004), SIGGRAPH.

Multi-focal Display

Akeley, Watt, Girshick & Banks (2004), SIGGRAPH.

Multi-focal Display

Depth-weighted Blending

• Depth-weighted blending along lines of sight

• Weights dependent on dioptric distances to planes

Akeley, Watt, Girshick, & Banks (2004), SIGGRAPH.

Do V-A Conflicts Cause Fatigue/Discomfort?

• 600-ms stimulus at near or far vergence-specified distance

• Appeared at each focal distance

Hoffman, Girshick, Akeley, & Banks (2008), Journal of Vision

Do V-A Conflicts Cause Fatigue/Discomfort?

** ** ** **

cues-inconsistent

cues-consistent

Sev

erity

of S

ympt

om

1

3

5

7

9

How tired are your eyes?

How clear is your vision?

How tired or sore are your

neck & back?

How do your eyes feel?

How does your head feel?

** = p < 0.01 (Wilcoxen test)

Hoffman, Girshick, Akeley, & Banks (2008), Journal of Vision

Do V-A Conflicts Cause Fatigue/Discomfort?

*******Which session was

more fatiguing?

Which session irritated

your eyes more?

Which session gave you

more headache?

Which session did

you prefer?

cues-consistent much worse than

inconsistent

cues-inconsistent much worse than

consistent

no difference

** = p < 0.01 (Wilcoxen test)

Hoffman, Girshick, Akeley, & Banks (2008), Journal of Vision

Do V-A Conflicts Cause Fatigue/Discomfort?

Discomfort & 3D Cinema

Discomfort & 3D Cinema

Discomfort & 3D Cinema

Discomfort & 3D Cinema

Discomfort & 3D Cinema & TV

Technical Issues• Developing content• Sufficient resolution over time: temporal aliasing• Sufficient separation between two eyes’ images: “ghosting”

User Issues• Perceptual distortions due to incorrect viewing position• Flicker & motion judder due to temporal sampling• Maintaining depth across scene cuts• Window violations• Residual ghosting• Visual discomfort due to vergence-accommodation conflict• Appropriate blur relative to other depth signals• Conflict between visually-induced motion & vestibular signals

Issues in 3D TV & Cinema

Almost never view pictures from correct position.

Retinal image thus specifies different scene than depicted.

Do people compensate, and if so, how?

Viewing Pictures

Stimuli

Vishwanath, Girshick, & Banks (2005), Nature Neuroscience.

Stimulus: simulated 3D ovoid with variable aspect ratio.

Task: adjust ovoid until appears spherical.

Vary monitor slant Sm to

assess compensation for oblique viewing positions.

Spatial calibration procedure.

If compensate, will set ovoid to sphere on screen (ellipse on retina).

Observation Point

Sm

CRT

Experimental Task

Vishwanath, Girshick, & Banks (2005), Nature Neuroscience.

Center of Projection

Observation Point

No compensation: set ovoid to make image on retina circular:

retinal coordinates

screen coordinates

Predictions

Center of Projection

Observation Point

Compensation: Set ovoid to make image on screen circular:

Predictions

retinal coordinates

screen coordinates

Asp

ect R

atio

(sc

reen

coo

rds)

1

1.2

1.4

-40 -20 0 20 40

invariance predictions

Viewing Angle Sm (deg)

Sm

Predictions

Predictions

1

1.2

1.4

-40 -20 0 20 40

invariance predictions

retinal predictions

Asp

ect R

atio

(sc

reen

coo

rds)

Viewing Angle Sm (deg)

Sm

Results

1

1.2

1.4

-40 -20 0 20 40

monoc-aperture

invariance predictsretinal predicts

Asp

ect R

atio

(sc

reen

coo

rds)

Viewing Angle (deg)

JLL

Vishwanath, Girshick, & Banks (2005), Nature Neuroscience.

Results

1

1.2

1.4

-40 -20 0 20 40

monoc-aperture

binoc-no aperture

invariance predictsretinal predicts

JLL

Results

Asp

ect R

atio

(sc

reen

coo

rds)

Viewing Angle (deg)

Vishwanath, Girshick, & Banks (2005), Nature Neuroscience.

Compensation for Incorrect Viewing Position

• Pictures not useful unless percepts are robust to changes in viewing position.

• People compensate for oblique viewing position when viewing 2d pictures.

• Two theories of compensation: pictorial & surface. Data clearly favor surface compensation.

• Two versions of surface method: global & local. Data clearly favor local slant.

2D Pictures vs 3D Pictures

• Two eyes presented same image

• Binocular disparities specify orientation & distance of picture surface; hence useful for compensation

2D

• Two eyes presented different images

• Binocular disparities specify orientation & distance of picture surface and layout of picture contents; hence not useful for compensation

• Two eyes presented same image

• Binocular disparities specify orientation & distance of picture surface; hence

useful for compensation

3D

• Two eyes presented same image

• Binocular disparities specify orientation & distance of picture surface; hence useful for compensation

2D

2D Pictures vs 3D Pictures

Stereo (3D) Pictures

• For most applications, viewers will not be at correct position.

• Retinal disparities thus specify a different layout than depicted.

• Do people compensate?

• Is correct seating position for a 3D movie more important than for 2D movie?

Stereo Picture Geometry

display surface

stereo projectors

display surface

stereo projectors

depicted hinge

Stereo Picture Geometry

display surface

stereo projectors

depicted hinge

Stereo Picture Geometry

display surface

stereo projectors

depicted hinge

disparity-specified hinge

Stereo Picture Geometry

perceiveddihedralangle?

display surface

stereo projectors

depicted hinge

disparity-specified hinge

Stereo Picture Geometry

Predictions

0 25 450

30

60

90

120

Viewing Angle (deg)

35°

17.5°

0°

-17.5°

-35°

Hin

ge S

ettin

g (d

eg) Invariance: Hinge settings

are 90° for all viewing angles and base slants

Retinal disparity: Hinge settings vary significantly with viewing angle & base slant

Viewing Angle (deg)

0 25 45

30

60

90

120

Hin

ge S

ettin

g (d

eg)

Results

non-stereo pictures

Viewing Angle (deg)

0 25 45

30

60

90

120

stereo pictures

0 25 45

30

60

90

120

Hin

ge S

ettin

g (d

eg)

Results

non-stereo pictures

Viewing Angle (deg)

0 25 45

30

60

90

120

0 25 45

30

60

90

120

Hin

ge S

ettin

g (d

eg)

non-stereo pictures stereo pictures

Results

Summary

• User issues in 3D cinema & TV

• Vergence-accommodation conflicts cause visual fatigue & discomfort

Can be handled by attending to viewer’s distance from screen & range of disparities presented relative to screen

• Perceptual distortions due to incorrect viewing position

Compensation is good with non-stereo pictures

Compensation is significantly poorer with stereo pictures suggesting that viewer position could be more important

Acknowledgments

• Kurt Akeley (Microsoft)

• Simon Watt (Univ. of Wales, Bangor)

• Ahna Girshick (NYU)

• David Hoffman (UC Berkeley)

• Robin Held (UC Berkeley)

• Funding from NIH, NSF, & Sharp Labs