Download - More image filtering
![Page 1: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/1.jpg)
More image filtering
15-463, 15-663, 15-862Computational Photography
Fall 2017, Lecture 4http://graphics.cs.cmu.edu/courses/15-463
![Page 2: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/2.jpg)
Course announcements
• Any questions about Homework 1?- How many of you have read/started/finished the homework?
• Make sure to take the Doodle about rescheduling the September 27th lecture!- Link available on Piazza.- Currently 10 responses.
![Page 3: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Template matching.
• Morphological filters.
• Rank filters.
• Adaptive thresholding.
• Bilateral filtering.
• Non-local means.
![Page 4: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted directly from:
• Kris Kitani (15-463, Fall 2016).
Inspiration and some examples also came from:
• James Hays (Georgia Tech).
• Bernd Girod (Stanford).
![Page 5: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/5.jpg)
Template matching
![Page 6: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/6.jpg)
Reminder from last time
How do we detect an edge?
![Page 7: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/7.jpg)
Reminder from last time
How do we detect an edge?• We filter with something that looks like an edge.
original
horizontal edge filter
vertical edge filter
1 0 -1
1
0
-1*
*
We can think of linear filtering as a way to evaluate how similar an image is locally to some template.
![Page 8: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/8.jpg)
Find this template
How do we detect the template in he following image?
![Page 9: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/9.jpg)
Find this template
How do we detect the template in he following image?
Solution 1: Filter the image using the template as filter kernel.
filter
image
output What will the output look like?
![Page 10: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/10.jpg)
Find this template
How do we detect the template in he following image?
Solution 1: Filter the image using the template as filter kernel.
filter
image
output
What went wrong?
![Page 11: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/11.jpg)
Find this template
How do we detect the template in he following image?
Solution 1: Filter the image using the template as filter kernel.
filter
image
output
Increases for higher local intensities.
![Page 12: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/12.jpg)
Find this template
How do we detect the template in he following image?
Solution 2: Filter the image using a zero-mean template.
filter
image
output What will the output look like?
template mean
![Page 13: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/13.jpg)
Find this template
How do we detect the template in he following image?
Solution 2: Filter the image using a zero-mean template.
filter
image
output
template mean
True detection
False detections
output
thresholding
What went wrong?
![Page 14: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/14.jpg)
Find this template
How do we detect the template in he following image?
Solution 2: Filter the image using a zero-mean template.
filter
image
output
template mean
output
Not robust to high-contrast areas
![Page 15: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/15.jpg)
Find this template
How do we detect the template in he following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output What will the output look like?
![Page 16: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/16.jpg)
Find this template
How do we detect the template in he following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output True detection
1-output
thresholding
What could go wrong?
![Page 17: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/17.jpg)
Find this template
How do we detect the template in he following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output
1-output
Not robust to local intensity changes
![Page 18: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/18.jpg)
Find this template
How do we detect the template in he following image?
Observations so far:
• subtracting mean deals with brightness bias
• dividing by standard deviation removes contrast bias
Can we combine the two effects?
![Page 19: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/19.jpg)
Find this template
How do we detect the template in he following image?
Solution 4: Normalized cross-correlation (NCC).
filter
image
output
template mean
local patch mean
What will the output look like?
![Page 20: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/20.jpg)
Find this template
How do we detect the template in he following image?
Solution 4: Normalized cross-correlation (NCC).
True detections
1-output
thresholding
![Page 21: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/21.jpg)
Find this template
How do we detect the template in he following image?
Solution 4: Normalized cross-correlation (NCC).
True detections
1-output
thresholding
![Page 22: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/22.jpg)
What is the best method?
It depends on whether you care about speed or invariance.
• Zero-mean: Fastest, very sensitive to local intensity.
• Sum of squared differences: Medium speed, sensitive to intensity offsets.
• Normalized cross-correlation: Slowest, invariant to contrast and brightness.
![Page 23: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/23.jpg)
Reminder: two types of image transformations
changes pixel values changes pixel locations
Filtering Warping
![Page 24: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/24.jpg)
Effects of image warping
How well does patch-based template matching do under warping?
![Page 25: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/25.jpg)
Effects of image warping
How well does patch-based template matching do under warping?• Not at all.
scaling shearing rotation reflectionoriginal
can handle can’t handle
How would you handle these cases?
![Page 26: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/26.jpg)
Applications of template matching
http://davidwalsh.name/face-detection-jquery
Face detection
http://hugin.sourceforge.net/tech/
Alignment
https://www.cim.mcgill.ca/sre/projects/fingertip/
Fingertip detectionCounting
http://en.wikipedia.org/wiki/File:Neubauer_improved_with_cells.jpg
ASCII art
http://fr.wikipedia.org/wiki/Art_ASCII
“Every computer vision problem can be described as a registration problem.”
Homework 4
Light fields
![Page 27: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/27.jpg)
Morphological filtering
![Page 28: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/28.jpg)
Theme for the rest of this lecture
Last time we discussed filtering operations that are both:
• linear
• shift-invariant
This time we will see filters where we remove one or both of these properties.
![Page 29: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/29.jpg)
Processing binary images
Binary images are quite common:• segmentation• template matching• text• thresholding
Mathematical morphology: • set-theoretic study of binary image processing• well-studied field with rich history
Generalizes to:• grayscale image filtering• distance transforms• diffusion operations
![Page 30: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/30.jpg)
Representation of binary images
Foreground or object pixels:• intensity value 1 (white)
Background pixels:• intensity value 0 (black)
![Page 31: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/31.jpg)
Some logic preliminaries
Basic logic operationsImage A Image B
How do you create these images as logical combinations of A and B?
![Page 32: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/32.jpg)
Some logic preliminaries
Basic logic operationsImage A Image B
How do you create these images as logical combinations of A and B?
NOT(A)
![Page 33: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/33.jpg)
Some logic preliminaries
Basic logic operationsImage A Image B
How do you create these images as logical combinations of A and B?
NOT(A) AND(A,B)
![Page 34: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/34.jpg)
Some logic preliminaries
Basic logic operationsImage A Image B
How do you create these images as logical combinations of A and B?
NOT(A) AND(A,B) OR(A,B)
![Page 35: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/35.jpg)
Some logic preliminaries
Basic logic operationsImage A Image B
How do you create these images as logical combinations of A and B?
NOT(A) AND(A,B) OR(A,B) XOR(A,B)
![Page 36: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/36.jpg)
Some logic preliminaries
Basic logic operationsImage A Image B
How do you create these images as logical combinations of A and B?
NOT(A) AND(A,B) OR(A,B) XOR(A,B) AND(NOT(A), B)
Notation: B-A
![Page 37: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/37.jpg)
Structuring element
Basically the binary equivalent of a kernel• specifies a neighborhood around a binary pixel
5x5 square crosss
For each structuring element, we can specify a corresponding windowing operator:
structuring element
![Page 38: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/38.jpg)
Basic morphological filters
Dilation: expand a binary image based on some structuring element
=What does the output look like?
![Page 39: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/39.jpg)
Basic morphological filters
Dilation: expand a binary image based on some structuring element
=
![Page 40: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/40.jpg)
Performing dilation
Shift structuring element to every pixel, then compute the OR operator in the neighborhood defined by the structuring element
![Page 41: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/41.jpg)
Basic morphological filters
=What does the output look like?
Erosion: shrink a binary image based on some structuring element
![Page 42: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/42.jpg)
Basic morphological filters
Erosion: shrink a binary image based on some structuring element
=
![Page 43: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/43.jpg)
Example
dilation with 3 x 3 dilation with 7 x 7
erosion with 3 x 3 erosion with 7 x 7
original
![Page 44: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/44.jpg)
Example
30 x 30 square 70 x 70 square
diam = 30 circle diam = 70 circle
original
Erosion with structuring elements of different shapes
![Page 45: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/45.jpg)
Template matching using morphological filters
binary fence image
How to detect the gaps in the fence?
![Page 46: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/46.jpg)
Template matching using morphological filters
binary fence image erosion with 150 x 150 cross
![Page 47: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/47.jpg)
Template matching using morphological filters
How to detect all instances of the letter “e”?
binarized text
![Page 48: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/48.jpg)
Template matching using morphological filters
binarized text erosion with structuring element
![Page 49: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/49.jpg)
Edge detection using morphological filters
original dilated - eroded
dilated - original original - eroded
![Page 50: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/50.jpg)
Set-theoretic interpretation
Dilation: Minkowski set addition Erosion: Minkowski set subtraction
structuring element
![Page 51: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/51.jpg)
Which of the following is true?
Assume we always use the same structuring element.
• First eroding and then dilating an image produces the same result as first dilating and then eroding the image.
• Eroding and then dilating an image returns the original image.
![Page 52: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/52.jpg)
Which of the following is true?
Assume we always use the same structuring element.
• First eroding and then dilating an image produces the same result as first dilating and then eroding the image.
• Eroding and then dilating an image returns the original image.
Nope.
Nope.
“Dual” morphological operations generally neither commute nor are inverses of each other.
![Page 53: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/53.jpg)
More morphological filters
Closing: first dilate then erode image
Opening: first erode then dilate image
Majority: replace pixel with majority value in neighborhood
![Page 54: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/54.jpg)
Denoising using majority operation
![Page 55: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/55.jpg)
Opening and closing
original
original
opening
closing
erosion
![Page 56: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/56.jpg)
Small hole closing
closingdilationoriginal
![Page 57: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/57.jpg)
Are morphological filters:
Shift-invariant?
Linear?
![Page 58: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/58.jpg)
Are morphological filters:
Shift-invariant?
Linear?
• No.
• Yes.
We can prove that morphological filters are equivalent generalized forms of convolution, where maximum (supremum) replaces summation, and additions replace products:
![Page 59: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/59.jpg)
How to generalize morphological filters to grayscale images?
![Page 60: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/60.jpg)
How to generalize morphological filters to grayscale images?
General theory based on image level sets:
• Separate image into multiple binary images, by thresholding at each possible intensity level (“level sets”).
• Apply morphological filter to each level set image.
• Combine results using maximum across level set images.
We will see one simple instance of this.
![Page 61: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/61.jpg)
Rank filters
![Page 62: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/62.jpg)
Replacing logical operators
Can you think of a function of the binary pixel values in an image neighborhood that produces the same result as the logical OR operator?
![Page 63: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/63.jpg)
Replacing logical operators
Dilation:
Erosion:
Majority:
Replace OR with MAX
Replace AND with ?
Replace MAJ with ?
![Page 64: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/64.jpg)
Replacing logical operators
Dilation:
Erosion:
Majority:
Replace OR with MAX
Replace AND with MIN
Replace MAJ with ?
![Page 65: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/65.jpg)
Replacing logical operators
Dilation:
Erosion:
Majority:
Replace OR with MAX
Replace AND with MIN
Replace MAJ with MEDIAN
Given these replacements, how would you generalize these filters to grayscale images?
![Page 66: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/66.jpg)
Rank filters
Grayscale dilation Max filtering
Grayscale erosion Min filtering
Grayscale majority Median filtering
• Are these filters linear, shift invariant, neither, or both?
• How would you generalize opening and closing to grayscale images?
![Page 67: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/67.jpg)
Min and max filtering example
original dilation (max filtering) erosion (min filtering)
![Page 68: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/68.jpg)
Effect of structuring element
original
20-degree line
disk
2 horizontal lines
diamond
9 points
![Page 69: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/69.jpg)
Morphological edge detection
original dilation - erosion thresholded result
![Page 70: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/70.jpg)
Standard “salt and pepper” noise example
(a) (b) (c) (d)
Salt and Pepper noise Original Median filter Gaussian filter
Denoising
Which is which?
![Page 71: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/71.jpg)
More realistic denoising
3x3 median filteringsalt and pepper noiseoriginal 7x7 median filtering
![Page 72: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/72.jpg)
Original
Removing annoying artifacts
Median filtering
![Page 73: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/73.jpg)
Cartoonization
How would you create this effect?
![Page 74: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/74.jpg)
Cartoonization
edges from median blurred image median blurred image
+ =
animated effect
Note: image cartoonization and abstraction are very active research areas.
![Page 75: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/75.jpg)
Adaptive thresholding
![Page 76: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/76.jpg)
How would you turn this into a bright binary image?
![Page 77: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/77.jpg)
Single-value thresholding
inp
ut
bin
ariz
ed
threshold
0
1
intensity
x
intensity
x
intensity
x
0
1
intensity
x
What is the problem here?
![Page 78: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/78.jpg)
Single-value thresholding
inp
ut
bin
ariz
ed
threshold
0
1
intensity
x
intensity
x
intensity
x
0
1
intensity
x
We can’t get both dips with
single-value thresholding
How would you do thresholding here?
![Page 79: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/79.jpg)
Single-value thresholding
inp
ut
bin
ariz
ed
threshold
0
1
intensity
x
intensity
x
intensity
x
0
1
intensity
x
Adapt threshold to local values
Can you think of a way to implement this using filtering?
![Page 80: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/80.jpg)
Adaptive thresholding
g[x, y] = 1, f[x, y] > threshold
0, otherwiseGlobal thresholding:
g[x, y] = 1, f[x, y] > mean(W(x, y))
0, otherwise
Adaptive thresholding using mean filtering:
g[x, y] = 1, f[x, y] > median(W(x, y))
0, otherwise
Adaptive thresholding using median filtering:
Median: greater than 50%You can use any other percentile
When using rank filters, this is a generalized version of morphological operations.
![Page 81: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/81.jpg)
Examples
global thresholding adaptive thresholdingoriginal
![Page 82: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/82.jpg)
Examples
adaptive thresholdingoriginal
![Page 83: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/83.jpg)
Bilateral filtering
![Page 84: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/84.jpg)
Fixing Gaussian blur
How to smooth out the details in an image without losing the important edges?
![Page 85: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/85.jpg)
The problem with Gaussian filtering
input
Gaussian kernel
*
*
*
output
Why is the output so blurry?
![Page 86: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/86.jpg)
The problem with Gaussian filtering
input
Gaussian kernel
*
*
*
output
Blur kernel averages across edges
![Page 87: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/87.jpg)
The bilateral filtering solution
input
bilateral filter kernel
*
*
*
output
Do not blur if there is an edge! How does it do that?
![Page 88: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/88.jpg)
Bilateral filtering vs Gaussian filtering
Spatial weightingNormalization factor Intensity range weighting
Does it matter how
far the pixel position
is?
if it’s nearby it looks like meand
![Page 89: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/89.jpg)
Bilateral filtering vs Gaussian filtering
Which is which?
![Page 90: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/90.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filtering
![Page 91: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/91.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filteringSpatial weighting: favor nearby pixels
![Page 92: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/92.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filteringSpatial weighting: favor nearby pixels
Intensity range weighting: favor similar pixels
![Page 93: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/93.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filteringSpatial weighting: favor nearby pixels
Normalization factor
Intensity range weighting: favor similar pixels
![Page 94: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/94.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filtering
Smooths everything nearby (even edges)Only depends on spatial distance
Smooths ‘close’ pixels in space and intensityDepends on spatial and intensity distance
![Page 95: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/95.jpg)
Bilateral filtering visualization
Output Bilateral Filter Input
Spatial range Intensity range
![Page 96: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/96.jpg)
Exploring the bilateral filter parameter space
input
ss = 2
ss = 6
ss = 18
sr = 0.1 sr = 0.25sr =
(Gaussian blur)
![Page 97: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/97.jpg)
Does the bilateral filter respect all edges?
input
bilateral filter kernel
*
*
*
output
![Page 98: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/98.jpg)
Does the bilateral filter respect all edges?
input
bilateral filter kernel
*
*
output
Bilateral filter crosses (and blurs) thin edges.
![Page 99: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/99.jpg)
Denoising
noisy input bilateral filtering median filtering
![Page 100: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/100.jpg)
Tone mapping
original bilateral filtering simple gamma correction
![Page 101: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/101.jpg)
Photo retouching
original digital pore removal (aka bilateral filtering)
![Page 102: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/102.jpg)
Before
![Page 103: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/103.jpg)
After
![Page 104: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/104.jpg)
Close-up comparison
original digital pore removal (aka bilateral filtering)
![Page 105: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/105.jpg)
Is the bilateral filter:
Shift-invariant?
Linear?
![Page 106: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/106.jpg)
Is the bilateral filter:
Shift-invariant?
Linear?
• No.
• No.
Bilateral filtering cannot be implemented as convolution. This makes naïve implementation very computationally expensive.
Efficient algorithms for bilateral filtering are an active research area.
![Page 107: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/107.jpg)
Non-local means
![Page 108: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/108.jpg)
Redundancy in natural images
![Page 109: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/109.jpg)
Non-local means
𝑤(𝑝, 𝑞)
𝑤(𝑝, 𝑟)𝑤(𝑝, 𝑠)
𝑥 𝑖 =1
𝐶𝑖
𝑗
𝑦(𝑗) 𝑒−𝑆𝑆𝐷 y 𝑁𝑖 −y 𝑁𝑗
2𝜎2
𝑤 𝑖, 𝑗
No need to stop at neighborhood. Instead search everywhere in the image.
![Page 110: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/110.jpg)
Non-local means vs bilateral filtering
Non-local means filtering
Bilateral filtering
Spatial weighting: favor nearby pixels
Intensity range weighting: favor similar pixels (patches in case of non-local means)
![Page 111: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/111.jpg)
Everything put together
Gaussian filtering
Bilateral filtering
Smooths everything nearby (even edges)Only depends on spatial distance
Smooths ‘close’ pixels in space and intensityDepends on spatial and intensity distance
Non-local means
Smooths similar patches no matter how far awayOnly depends on intensity distance
![Page 112: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/112.jpg)
Denoising example
bilateral filteringGaussian filteringnoisy input non-local means
![Page 113: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/113.jpg)
Very general forms of “structural” filtering
We will see more in later lectures.
![Page 114: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/114.jpg)
Is non-local means:
Shift-invariant?
Linear?
![Page 115: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/115.jpg)
Is non-local means:
Shift-invariant?
Linear?
• No.
• No.
Non-local means is not a convolution, and is generally very very challenging to implement efficiently.
Efficient algorithms for non-local means are an active research area.
![Page 116: More image filtering](https://reader031.vdocuments.site/reader031/viewer/2022020620/61e312e48a285637a468d99e/html5/thumbnails/116.jpg)
References
Basic reading:• Szeliski textbook, Sections 3.2 and 8.1
Additional reading:• Serra, “Image Analysis and Mathematical Morphology,” Academic Press 1983.
standard reference book on mathematical morphology, also available in course formhttp://cmm.ensmp.fr/~serra/cours/index.htm
• Paris et al., “A Gentle Introduction to the Bilateral Filter and Its Applications,” SIGGRAPH 2007-08, CVPR 2008short course on the bilateral filter, including discussion of fast implementationshttps://people.csail.mit.edu/sparis/bf_course/
• Xu et al., “Image Smoothing via L0 Gradient Minimization,” SIGGRAPH 2011one of many works on image abstraction and cartoonization, with a good related work section
• Buades et al., “Nonlocal Image and Movie Denoising,” IJCV 2008the journal version of the original non-local means paper
• Felzenszwalb and Huttenlocher, “Distance Transforms of Sampled Functions,” ToC 2012discusses how to compute distance transforms and skeletons using morhology