compositing & blending

Compositing & Blending

Tamara Berg Computational Photography

Many slides from: Alexei Efros & Derek Hoiem

Image Compositing & Blending

This Class How do I put an object from one image into another?

Slide from: Derek Hoiem

Image Compositing

Recent news

Original

“Enhanced” Version

http://www.guardian.co.uk/world/2010/sep/16/mubarak-doctored-red-carpet-picture

Compositing Procedure 1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)

Composite by David Dewey

2. Blend them into the composite (in the right order)

Three methods

1.  Cut and paste

2.  Laplacian pyramid blending

3.  Poisson blending

Method 1: Cut and Paste

Method 1: Cut and Paste Method: •  Segment using intelligent scissors •  Paste foreground pixels onto target region

Method 1: Cut and Paste

Problems: •  Small segmentation errors noticeable •  Pixels are too blocky •  Won’t work for semi-transparent materials

Feathering Near object boundary pixel values come partly from foreground and partly from background

Method 1: Cut and Paste (with feathering)

Alpha compositing

Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem

Alpha compositing with feathering

Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem

Proper blending is key

Alpha Blending / Feathering

= Iblend = αIleft + (1-α)Iright

Setting alpha: simple averaging

Alpha = .5 in overlap region

Setting alpha: center seam

Alpha = logical(dtrans1>dtrans2)

Distance Transform bwdist!

Setting alpha: blurred seam

Alpha = blurred

Distance transform

Setting alpha: center weighting

Alpha = dtrans1 / (dtrans1+dtrans2)

Distance transform

Ghost!

Alpha Blending / Feathering

= Iblend = αIleft + (1-α)Iright

Affect of Window Size

1 left

right 0

Large windows -> lots of ghosting

Affect of Window Size

Small window –> seams are too obvious

Good Window Size

“Optimal” Window: smooth but not ghosted

Medium window -> just right!

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

To avoid ghosting •  window <= 2*size of smallest prominent feature

Natural to cast this in the Fourier domain

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright)

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + …

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain

i with Frighti

•  Sum feathered octave images in frequency domain •  Compute IFFT to get final blended result.

i with Frighti

•  Sum feathered octave images in frequency domain •  Compute IFFT to get final blended result.

Actually better implemented in the spatial domain

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983] •  In computer graphics, a mip map [Williams, 1983]

- Reminder

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

Figure from David Forsyth

Gaussian Pyramid

Laplacian Pyramid

Lowpass Images

Laplacian Pyramid

Bandpass Images

Lowpass Images

Laplacian Pyramid

Lowpass Images

Laplacian Pyramid

Lowpass Images

Laplacian Pyramid

Lowpass Images

Bandpass Images

Pyramid Blending

Left pyramid Right pyramid blend

•  At low frequencies, blend slowly •  At high frequencies, blend quickly

Method 2: Laplacian Pyramid Blending

Implementation: 1. Build Laplacian pyramids for each image 2. Build a Gaussian pyramid of region mask 3. Blend each level of pyramid using region mask

from the same level

4. Collapse the pyramid to get the final blended image

)1(2112iiiii RLRLL −⋅+⋅=

Burt and Adelson 1983

Region mask at level i of Gaussian pyramid

Image 1 at level i of Laplacian pyramid

laplacian level

left pyramid right pyramid blended pyramid

Pyramid Blending

Blending Regions

Horror Photo

Season Blending (St. Petersburg)

Simplification: Two-band Blending Brown & Lowe, 2003

•  Only use two bands: high freq. and low freq. •  Blends low freq. smoothly •  Blend high freq. with no smoothing: use binary alpha

Low frequency (λ > 2 pixels)

High frequency (λ < 2 pixels)

2-band Blending

Linear Blending

2-band Blending

Practice Quiz Poisson Image Editing: 1) “Solving the Poisson equation computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions” In blending: What does function refer to? What is an example prescribed vector field?

2) How might you implement texture flattening using Poisson techniques? Photomontage: 3) What are the 2 main techniques used in this paper?

4) What is an image objective? Name 3 possible objectives. Seam Carving: 5) What is the definition of a seam?

6) What algorithm would you use to efficiently find the optimal seam for removal in an image? Describe generally how to implement this.

Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background

Perez et al. 2003

Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background

Perez et al. 2003

What’s with Poisson?

Solving Poisson – computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions

What’s that mean?

Given a guidance field (e.g. gradient field from source image, or from a composite image)

Calculate the image that best fits this vector field

In general no image exists whose gradient exactly matches input. Best-fit image in least squares sense can be calculated by solving Poisson equation.

Perez et al. 2003

HW2: Gradient Domain Editing

General concept: Solve for pixels of new image that satisfy constraints on the gradient and the intensity •  Constraints can be from one image (for filtering) or more (for

blending)

HW2: Poisson Blending

A good blend should preserve gradients of source region without changing the background

Treat pixels as variables to be solved

•  Minimize squared difference between gradients of foreground region and gradients of target region

•  Keep background pixels constant

Perez et al. 2003

Example

Source: Evan Wallace

Gradient Visualization

Source: Evan Wallace

+ Specify object region

Other results

Perez et al. 2003

What do we lose?

Perez et al. 2003

Foreground color changes Background pixels in target region are replaced

Blending with Mixed Gradients Use foreground or background gradient with

larger magnitude as the guiding gradient

Perez et al. 2003

Concealment

Transparent Objects

Things to remember

Three ways to blend/composite 1.  Alpha compositing

–  Need nice cut (intelligent scissors) –  Should feather

2.  Laplacian pyramid blending –  Smooth blending at low frequencies, sharp at high

frequencies –  Usually used for stitching

3.  Gradient domain editing –  Also called Poisson Editing –  Explicit control over what to preserve –  Changes foreground color (for better or worse) –  Applicable for many things besides blending

Don’t blend, CUT!

So far we only tried to blend between two images. What about finding an optimal seam?

Moving objects become ghosts

Davis, 1998 Segment the mosaic

•  Single source image per segment •  Pick a good seam (cut) so that when you merge the

segments is looks ok.

- Details about this next week!

Kwatra et al, 2003

Putting it all together Compositing images

•  Have a clever blending function –  Feathering –  Center-weighted –  blend different frequencies differently

•  Choose the right pixels from each image –  Graph-cuts

Interactive Digital Photomontage

Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen

Challenges Find good seams between parts of images so

they can be joined with few visible artifacts Blend along seams to reduce or remove any

artifacts remaining after joining

Seam Objectives Measures suitability of a seam between two image regions

e.g. •  Match colors across seams •  Match colors and color gradients across

seams •  Match colors across seams, but prefer seams

that lie along edges

User Selection

Result of Pasting

Gradient Smoothing

Remove any remaining artifacts after merging source image parts

View input images as sources of gradient information rather than color.

Using graph cut labeling, form composite whose gradients best match source gradient vectors.

Final Result after smoothing

Extended Depth of Field

Input Images

Source Map

Extended Depth of Field

Relighting

Input Images

Source Map

Relighted Result

Visualizing Movement

Selective Composites

Background reconstruc?on

Religh?ng

Composi?ng and Obstruc?on removal

Fictional Composites

Photo Clip Art

Jean-François Lalonde, Derek Hoeim, Alexei A. Efros, Carsten Rother, John Winn and Antonio Criminisi. Photo Clip Art, ACM Transactions on Graphics (SIGGRAPH 2007),

Photo Clip Art

Street accident

Bridge

Painting

Failure cases

Shadow transfer

Porous objects

Failure cases

Best matching objects

Inverse of previous topic Removing content (as opposed to adding it)

Next…

Seam Carving for Content-Aware Image Resizing Shai Avidan Ariel Shamir

Slides from: Kshitij Padalkar

In One Line…

Method for Content-‐Aware Image Resizing

Different Displays-‐Different Resolutions-‐(Different Images??)

HTML allows automatic layout of the text, but not the images

Solu?ons and Other Work Crop and Scale Saliency Segmentation

Defini?on…

Remove unnoticeable pixels. Energy Function:

� Remove Pixels with least energy �  Sort pixels by Energy, remove smallest pixels (optimum) �  Remove Column with smallest total Energy (column) �  Remove Seam with smallest total Energy (seam)

Vertical Seam: �  an 8-connected path of pixels in the image from

top to bottom, containing one, and only one, pixel in each row of the image

Energy of a Seam

� Minimum Energy Seam

Pixel Removal Criteria Optimal

Pixel Removal Criteria Column

Pixel Removal Criteria Seam

Discrete Image Resizing Aspect Ratio Change

•  Optimal Seam Order

Image Enlarging Content Amplification Object Removal

Aspect Ra?o Change Change the image from (n X m) to (n X m’)

•  (assume m’ < m)

Remove m-‐m’=c seams successively

Seam Carving

Aspect Ra?o Change Change the image from (n X m) to (n X m’)

•  (assume m’ < m)

Remove m-‐m’=c seams successively

Scaling

Op?mal Seam Ordering

Change the image from (n X m) to (n’ X m’) •  (assume m’ < m and n’ < n)

What to do? What is the correct order of seam carving? •  Remove vertical seams first? •  Horizontal seams first? •  Or alternate between the two?

Op?mal Seam Ordering

Transport map •  Matrix of size n X m •  Each element T(r,c) holds the minimal cost needed to obtain an

image of size n−r X m−c

Image Enlarging

�  compute the optimal vertical (horizontal) seam ‘s’ on I and duplicate the pixels of s by averaging them with their left and right neighbors (top and bottom in the horizontal case)

Image Enlarging

� To reduce the width by ‘k’, compute first ‘k’ vertical seams and duplicate them

� All seams duplicated, 50% enlargement. For more do it in steps.

Content Amplifica?on � Scale the image, this will scale everything, content as

well as non-content. � Shrink the scaled-image by seam carving, this will

carve out the non-content part.

Object Removal

� User marks the target object to be removed. � Then seams are removed from the image until all

marked pixels are gone. � To regain the original size, seam insertion is

employed.

Object Removal �  Find the missing shoe…

Object Removal � Object marking to remove unwanted results

Limita?ons

� Too much content � No space for seam to avoid content

Announcements HW2 up later today Readings for next week on web page

compositing & blending

Documents

blending and compositing

interactive computer graphics buffers, compositing and...

s05 compositing & staging

compositing, composing worlds

compositing 3d in photoshop

chromakey compositing

matting & compositing

rendering & compositing

buffers, compositing and blending week 8 david breen...

image blending and compositing - computer...

image blending and...

image compositing and matting

1 effetti impressionistici daniele marini. 2 tecniche...

cs 480/680 computer graphics compositing and blending dr....

image blending and compositing

image blending + image...

image blending and compositing 15-463: computational...

digital compositing

image compositing and blending

blending and compositing today