matting and transparency 15-463: computational photography alexei efros, cmu, fall 2007

52
Matting and Transparency 15-463: Computational Photograph Alexei Efros, CMU, Fall 200

Post on 19-Dec-2015

230 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Matting and Transparency

15-463: Computational PhotographyAlexei Efros, CMU, Fall 2007

Page 2: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

How does Superman fly?

Super-human powers?

OR

Image Matting?

Page 3: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

“Pulling a Matte”Problem Definition:

• The separation of an image C into– A foreground object image Co,

– a background image Cb,

– and an alpha matte

• Co and can then be used to composite the foreground object into a different image

Hard problem• Even if alpha is binary, this is hard to do automatically

(background subtraction problem)• For movies/TV, manual segmentation of each frame is

infeasible • Need to make a simplifying assumption…

Page 4: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Average/Median ImageWhat can we do with this?

Page 5: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Background Subtraction

--

==

Page 6: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Crowd Synthesis (with Pooja Nath)

1. Do background subtraction in each frame

2. Find and record “blobs”

3. For synthesis, randomly sample the blobs, taking care not to overlap them

Page 7: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Background SubtractionA largely unsolved problem…

Estimatedbackground

Difference Image

ThresholdedForeground on blue

One videoframe

Page 8: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Blue Screen

Page 9: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Blue Screen mattingMost common form of matting in TV studios & movies

Petros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement.

A form of background subtraction:• Need a known background• Compute alpha as SSD(C,Cb) > threshold

– Or use Vlahos’ formula: = 1-p1(B-p2G)

• Hope that foreground object doesn’t look like background – no blue ties!

• Why blue?• Why uniform?

Page 10: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

The Ultimatte

p1 and p2

Page 11: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Blue screen for superman?

Page 12: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Semi-transparent mattes

What we really want is to obtain a true alpha matte, which involves semi-transparency• Alpha between 0 and 1

Page 13: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Review: two issues

Semi-transparent objects

Pixels too large

Page 14: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Review: alpha channelAdd one more channel:

• Image(R,G,B,alpha) Sprite!

Encodes transparency (or pixel coverage):• Alpha = 1: opaque object (complete coverage)• Alpha = 0: transparent object (no coverage)• 0<Alpha<1: semi-transparent (partial coverage)

Example: alpha = 0.7

Partial coverage or semi-transparency

Page 15: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Matting Problem: Mathematical Definition

Page 16: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Why is general matting hard?

Page 17: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Solution #1: No Blue!

Page 18: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Solution #2: Gray or Flesh

Page 19: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Triangulation Matting (Smith & Blinn)

How many equations?

How many unknowns?

Does the background need to be constant color?

Page 20: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

The Algorithm

Page 21: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Triangulation Matting Examples

Page 22: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

More Examples

Page 23: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

More examples

Page 24: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Problems with Matting

Images do not look realistic

Lack of Refracted Light

Lack of Reflected Light

Solution:Modify the Matting Equation

Page 25: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Environment Matting and Compositing

slides by Jay Hetler

Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin

Page 26: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Environment Matting EquationC = F + (1- )B +

C ~ Color F ~ Foreground color B ~ Background color ~ Amount of light that passes through the

foreground ~ Contribution of light from Environment that travels

through the object

Page 27: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Explanation of

R – reflectance imageT – Texture image

Page 28: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Environment Mattes

Page 29: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

PerformanceCalibration

Matting: 10-20 minutes extraction time for each texture map (Pentium II 400Mhz)

Compositing: 4-40 frames per second

Real-Time?

Page 30: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

How much better is Environment Matting?

Alpha Matte Environment Matte Photograph

Page 31: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

How much better is Environment Matting?

Alpha Matte Environment Matte Photograph

Page 32: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Movies!

Page 33: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Fast Separation of Direct and Global

Images

Using High Frequency IlluminationShree K. Nayar

Gurunandan G. Krishnan

Columbia University

SIGGRAPH ConferenceBoston, July 2006

Support: ONR, NSF, MERL

Michael D. Grossberg

City College of New York

Ramesh Raskar

MERL

Page 34: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

source

surface

P

Direct and Global Illumination

A

A : Direct

B

B : Interrelection

C

C : Subsurface

D

participating medium

D : Volumetric translucent surface

E

E : Diffusion

camera

Page 35: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

],[],[],[ icLicLicL gd

direct globalradiance

Direct and Global Components: Interreflections

surface

i

camera

source

P

g jiLjiAicL ],[],[],[

j

BRDF and geometry

Page 36: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

High Frequency Illumination Pattern

surface

camera

source

fraction of activated source elements

],[],[],[ icLicLicL gd +

i

Page 37: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

High Frequency Illumination Pattern

surface

fraction of activated source elements

camera

source

],[],[],[ icLicLicL gd + - ],[],[ icLicL g )1(

i

Page 38: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

:2

1 min2LLg

Separation from Two Images

direct global

,minmax LLLd

Page 39: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Other Global Effects: Subsurface Scattering

translucent surface

camera

source

i

j

Page 40: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Other Global Effects: Volumetric Scattering

surface

camera

source

participating medium

i

j

Page 41: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Diffuse Interreflections

SpecularInterreflections

Volumetric Scattering Subsurface

Scattering

Diffusion

Page 42: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Scene

Page 43: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Scene

Direct Global

Page 44: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Real World Examples:

Can You Guess the Images?

Page 45: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Eggs: Diffuse Interreflections

Direct Global

Page 46: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Wooden Blocks: Specular Interreflections

Direct Global

Page 47: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Kitchen Sink: Volumetric Scattering

Volumetric Scattering:

Chandrasekar 50, Ishimaru 78

Direct Global

Page 48: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Peppers: Subsurface Scattering

Direct Global

Page 49: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Hand

Direct Global

Skin: Hanrahan and Krueger

93,

Uchida 96, Haro 01, Jensen et

al. 01,

Cula and Dana 02, Igarashi et

al. 05, Weyrich et al. 05

Page 50: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Face: Without and With Makeup

GlobalDirect

GlobalDirectWithout Makeup

With Makeup

Page 51: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Blonde Hair

Hair Scattering: Stamm et al. 77,

Bustard and Smith 91, Lu et al. 00

Marschner et al. 03

Direct Global

Page 52: Matting and Transparency 15-463: Computational Photography Alexei Efros, CMU, Fall 2007

www.cs.columbia.edu/CAVE