Download - Compositing & Blending
Compositing & Blending
Tamara Berg Computational Photography
Many slides from: Alexei Efros & Derek Hoiem
Image Compositing & Blending
© NASA
This Class How do I put an object from one image into another?
Slide from: Derek Hoiem
Image Compositing
Recent news
Original
“Enhanced” Version
http://www.guardian.co.uk/world/2010/sep/16/mubarak-doctored-red-carpet-picture
Slide from: Derek Hoiem
Compositing Procedure 1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
Composite by David Dewey
2. Blend them into the composite (in the right order)
Three methods
1. Cut and paste
2. Laplacian pyramid blending
3. Poisson blending
Slide from: Derek Hoiem
Method 1: Cut and Paste
Slide from: Derek Hoiem
Method 1: Cut and Paste Method: • Segment using intelligent scissors • Paste foreground pixels onto target region
Slide from: Derek Hoiem
Method 1: Cut and Paste
Problems: • Small segmentation errors noticeable • Pixels are too blocky • Won’t work for semi-transparent materials
Slide from: Derek Hoiem
Feathering Near object boundary pixel values come partly from foreground and partly from background
Slide from: Derek Hoiem
Method 1: Cut and Paste (with feathering)
Slide from: Derek Hoiem
Alpha compositing
+
Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem
Alpha compositing with feathering
Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem
Proper blending is key
Proper blending is key
Alpha Blending / Feathering
0 1
0 1
+
= Iblend = αIleft + (1-α)Iright
Setting alpha: simple averaging
Alpha = .5 in overlap region
Setting alpha: center seam
Alpha = logical(dtrans1>dtrans2)
Distance Transform bwdist!
Setting alpha: blurred seam
Alpha = blurred
Distance transform
Setting alpha: center weighting
Alpha = dtrans1 / (dtrans1+dtrans2)
Distance transform
Ghost!
Alpha Blending / Feathering
0 1
0 1
+
= Iblend = αIleft + (1-α)Iright
Affect of Window Size
0
1 left
right 0
1
Large windows -> lots of ghosting
alpha
Affect of Window Size
0
1
0
1
Small window –> seams are too obvious
Good Window Size
0
1
“Optimal” Window: smooth but not ghosted
Medium window -> just right!
What is the Optimal Window? To avoid seams
• window = size of largest prominent feature
What is the Optimal Window? To avoid seams
• window = size of largest prominent feature
To avoid ghosting • window <= 2*size of smallest prominent feature
What is the Optimal Window? To avoid seams
• window = size of largest prominent feature
To avoid ghosting • window <= 2*size of smallest prominent feature
Natural to cast this in the Fourier domain
FFT
Fourier domain blending
FFT
Fourier domain blending
Idea (Burt and Adelson) • Compute Fleft = FFT(Ileft), Fright = FFT(Iright)
FFT
Fourier domain blending
Idea (Burt and Adelson) • Compute Fleft = FFT(Ileft), Fright = FFT(Iright) • Decompose Fourier image into octaves (bands)
– Fleft = Fleft1 + Fleft
2 + …
FFT
Fourier domain blending
Idea (Burt and Adelson) • Compute Fleft = FFT(Ileft), Fright = FFT(Iright) • Decompose Fourier image into octaves (bands)
– Fleft = Fleft1 + Fleft
2 + … • Feather corresponding octaves Fleft
i with Frighti
FFT
Fourier domain blending
Idea (Burt and Adelson) • Compute Fleft = FFT(Ileft), Fright = FFT(Iright) • Decompose Fourier image into octaves (bands)
– Fleft = Fleft1 + Fleft
2 + … • Feather corresponding octaves Fleft
i with Frighti
• Sum feathered octave images in frequency domain
FFT
Fourier domain blending
Idea (Burt and Adelson) • Compute Fleft = FFT(Ileft), Fright = FFT(Iright) • Decompose Fourier image into octaves (bands)
– Fleft = Fleft1 + Fleft
2 + … • Feather corresponding octaves Fleft
i with Frighti
• Sum feathered octave images in frequency domain • Compute IFFT to get final blended result.
FFT
Fourier domain blending
Idea (Burt and Adelson) • Compute Fleft = FFT(Ileft), Fright = FFT(Iright) • Decompose Fourier image into octaves (bands)
– Fleft = Fleft1 + Fleft
2 + … • Feather corresponding octaves Fleft
i with Frighti
• Sum feathered octave images in frequency domain • Compute IFFT to get final blended result.
FFT
Actually better implemented in the spatial domain
Image Pyramids
Known as a Gaussian Pyramid [Burt and Adelson, 1983] • In computer graphics, a mip map [Williams, 1983]
- Reminder
A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose
Figure from David Forsyth
Gaussian Pyramid
Laplacian Pyramid
Lowpass Images
Laplacian Pyramid
Bandpass Images
Lowpass Images
Laplacian Pyramid
Lowpass Images
Laplacian Pyramid
Lowpass Images
Laplacian Pyramid
Lowpass Images
Bandpass Images
Pyramid Blending
0
1
0
1
0
1
Left pyramid Right pyramid blend
• At low frequencies, blend slowly • At high frequencies, blend quickly
Method 2: Laplacian Pyramid Blending
Implementation: 1. Build Laplacian pyramids for each image 2. Build a Gaussian pyramid of region mask 3. Blend each level of pyramid using region mask
from the same level
4. Collapse the pyramid to get the final blended image
)1(2112iiiii RLRLL −⋅+⋅=
Burt and Adelson 1983
Region mask at level i of Gaussian pyramid
Image 1 at level i of Laplacian pyramid
laplacian level
4
laplacian level
2
laplacian level
0
left pyramid right pyramid blended pyramid
Pyramid Blending
Blending Regions
Horror Photo
© david dmartin (Boston College)
Season Blending (St. Petersburg)
Season Blending (St. Petersburg)
Simplification: Two-band Blending Brown & Lowe, 2003
• Only use two bands: high freq. and low freq. • Blends low freq. smoothly • Blend high freq. with no smoothing: use binary alpha
Low frequency (λ > 2 pixels)
High frequency (λ < 2 pixels)
2-band Blending
Linear Blending
2-band Blending
Practice Quiz Poisson Image Editing: 1) “Solving the Poisson equation computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions” In blending: What does function refer to? What is an example prescribed vector field?
2) How might you implement texture flattening using Poisson techniques? Photomontage: 3) What are the 2 main techniques used in this paper?
4) What is an image objective? Name 3 possible objectives. Seam Carving: 5) What is the definition of a seam?
6) What algorithm would you use to efficiently find the optimal seam for removal in an image? Describe generally how to implement this.
Break
Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background
Perez et al. 2003
Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background
Perez et al. 2003
HW 2!
What’s with Poisson?
Solving Poisson – computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions
What’s that mean?
Given a guidance field (e.g. gradient field from source image, or from a composite image)
Calculate the image that best fits this vector field
In general no image exists whose gradient exactly matches input. Best-fit image in least squares sense can be calculated by solving Poisson equation.
Perez et al. 2003
HW2: Gradient Domain Editing
General concept: Solve for pixels of new image that satisfy constraints on the gradient and the intensity • Constraints can be from one image (for filtering) or more (for
blending)
HW2: Poisson Blending
A good blend should preserve gradients of source region without changing the background
Treat pixels as variables to be solved
• Minimize squared difference between gradients of foreground region and gradients of target region
• Keep background pixels constant
Perez et al. 2003
Example
Source: Evan Wallace
Gradient Visualization
Source: Evan Wallace
+ Specify object region
Other results
Perez et al. 2003
What do we lose?
Perez et al. 2003
Foreground color changes Background pixels in target region are replaced
Blending with Mixed Gradients Use foreground or background gradient with
larger magnitude as the guiding gradient
Perez et al. 2003
Concealment
Transparent Objects
NPR
Things to remember
Three ways to blend/composite 1. Alpha compositing
– Need nice cut (intelligent scissors) – Should feather
2. Laplacian pyramid blending – Smooth blending at low frequencies, sharp at high
frequencies – Usually used for stitching
3. Gradient domain editing – Also called Poisson Editing – Explicit control over what to preserve – Changes foreground color (for better or worse) – Applicable for many things besides blending
Don’t blend, CUT!
So far we only tried to blend between two images. What about finding an optimal seam?
Moving objects become ghosts
Davis, 1998 Segment the mosaic
• Single source image per segment • Pick a good seam (cut) so that when you merge the
segments is looks ok.
- Details about this next week!
Kwatra et al, 2003
Putting it all together Compositing images
• Have a clever blending function – Feathering – Center-weighted – blend different frequencies differently
• Choose the right pixels from each image – Graph-cuts
Interactive Digital Photomontage
Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen
Challenges Find good seams between parts of images so
they can be joined with few visible artifacts Blend along seams to reduce or remove any
artifacts remaining after joining
Seam Objectives Measures suitability of a seam between two image regions
e.g. • Match colors across seams • Match colors and color gradients across
seams • Match colors across seams, but prefer seams
that lie along edges
User Selection
Result of Pasting
Gradient Smoothing
Remove any remaining artifacts after merging source image parts
View input images as sources of gradient information rather than color.
Using graph cut labeling, form composite whose gradients best match source gradient vectors.
Final Result after smoothing
Extended Depth of Field
Input Images
Source Map
Extended Depth of Field
Relighting
Input Images
Source Map
Relighted Result
Visualizing Movement
Visualizing Movement
Selective Composites
Selective Composites
Selective Composites
Selective Composites
Background reconstruc?on
Background reconstruc?on
Religh?ng
Composi?ng and Obstruc?on removal
Fictional Composites
Photo Clip Art
Jean-François Lalonde, Derek Hoeim, Alexei A. Efros, Carsten Rother, John Winn and Antonio Criminisi. Photo Clip Art, ACM Transactions on Graphics (SIGGRAPH 2007),
Photo Clip Art
Photo Clip Art
Street accident
Bridge
Painting
Alley
Failure cases
Shadow transfer
Porous objects
Failure cases
Best matching objects
Inverse of previous topic Removing content (as opposed to adding it)
Next…
Seam Carving for Content-Aware Image Resizing Shai Avidan Ariel Shamir
Slides from: Kshitij Padalkar
In One Line…
Method for Content-‐Aware Image Resizing
Need
Different Displays-‐Different Resolutions-‐(Different Images??)
HTML allows automatic layout of the text, but not the images
Solu?ons and Other Work Crop and Scale Saliency Segmentation
Defini?on…
Remove unnoticeable pixels. Energy Function:
� Remove Pixels with least energy � Sort pixels by Energy, remove smallest pixels (optimum) � Remove Column with smallest total Energy (column) � Remove Seam with smallest total Energy (seam)
Seam
Vertical Seam: � an 8-connected path of pixels in the image from
top to bottom, containing one, and only one, pixel in each row of the image
Seam
Energy of a Seam
� Minimum Energy Seam
Pixel Removal Criteria Optimal
Pixel Removal Criteria Column
Pixel Removal Criteria Seam
Discrete Image Resizing Aspect Ratio Change
• Optimal Seam Order
Image Enlarging Content Amplification Object Removal
Aspect Ra?o Change Change the image from (n X m) to (n X m’)
• (assume m’ < m)
Remove m-‐m’=c seams successively
Seam Carving
Aspect Ra?o Change Change the image from (n X m) to (n X m’)
• (assume m’ < m)
Remove m-‐m’=c seams successively
Scaling
Op?mal Seam Ordering
Change the image from (n X m) to (n’ X m’) • (assume m’ < m and n’ < n)
What to do? What is the correct order of seam carving? • Remove vertical seams first? • Horizontal seams first? • Or alternate between the two?
Op?mal Seam Ordering
Transport map • Matrix of size n X m • Each element T(r,c) holds the minimal cost needed to obtain an
image of size n−r X m−c
Discrete Image Resizing Aspect Ratio Change
• Optimal Seam Order
Image Enlarging Content Amplification Object Removal
Image Enlarging
� compute the optimal vertical (horizontal) seam ‘s’ on I and duplicate the pixels of s by averaging them with their left and right neighbors (top and bottom in the horizontal case)
Image Enlarging
� To reduce the width by ‘k’, compute first ‘k’ vertical seams and duplicate them
� All seams duplicated, 50% enlargement. For more do it in steps.
Discrete Image Resizing Aspect Ratio Change
• Optimal Seam Order
Image Enlarging Content Amplification Object Removal
Content Amplifica?on � Scale the image, this will scale everything, content as
well as non-content. � Shrink the scaled-image by seam carving, this will
carve out the non-content part.
Discrete Image Resizing Aspect Ratio Change
• Optimal Seam Order
Image Enlarging Content Amplification Object Removal
Object Removal
� User marks the target object to be removed. � Then seams are removed from the image until all
marked pixels are gone. � To regain the original size, seam insertion is
employed.
Object Removal � Find the missing shoe…
Object Removal � Object marking to remove unwanted results
Limita?ons
� Too much content � No space for seam to avoid content
Announcements HW2 up later today Readings for next week on web page