introduction and course...
TRANSCRIPT
![Page 1: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/1.jpg)
Deconvolution
15-463, 15-663, 15-862Computational Photography
Fall 2017, Lecture 17http://graphics.cs.cmu.edu/courses/15-463
![Page 2: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/2.jpg)
Course announcements
• Homework 4 is out.- Due October 26th.- There was another typo in HW4, download new version.- Drop by Yannis’ office to pick up cameras any time.
• Homework 5 will be out on Thursday.- You will need cameras for that one as well, so keep the ones you picked up for HW4.
• Project ideas were due on Piazza on Friday 20th.- Responded to most of you.- Some still need to post their ideas.
• Project proposals are due on Monday 30th.
![Page 3: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Telecentric lenses.
• Sources of blur.
• Deconvolution.
• Blind deconvolution.
![Page 4: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted from:
• Fredo Durand (MIT).• Gordon Wetzstein (Stanford).
![Page 5: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/5.jpg)
Why are our images blurry?
![Page 6: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/6.jpg)
Why are our images blurry?
• Lens imperfections.
• Camera shake.
• Scene motion.
• Depth defocus.
![Page 7: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/7.jpg)
Lens imperfections
object distance D focus distance D’
• Ideal lens: An point maps to a point at a certain plane.
![Page 8: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/8.jpg)
Lens imperfections
object distance D focus distance D’
• Ideal lens: An point maps to a point at a certain plane.• Real lens: A point maps to a circle that has non-zero minimum radius among all planes.
What is the effect of this on the images we capture?
![Page 9: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/9.jpg)
Lens imperfections
object distance D focus distance D’
• Ideal lens: An point maps to a point at a certain plane.• Real lens: A point maps to a circle that has non-zero minimum radius among all planes.
Shift-invariant blur.
blur kernel
![Page 10: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/10.jpg)
Lens imperfectionsWhat causes lens imperfections?
![Page 11: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/11.jpg)
Lens imperfectionsWhat causes lens imperfections?• Aberrations.
• Diffraction.
large aperture
small aperture
![Page 12: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/12.jpg)
Lens as an optical low-pass filter
object distance D focus distance D’
Point spread function (PSF): The blur kernel of a lens.• “Diffraction-limited” PSF: No aberrations, only diffraction. Determined by aperture shape.
diffraction-limited PSF of a circular
aperture
blur kernel
![Page 13: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/13.jpg)
Lens as an optical low-pass filter
object distance D focus distance D’
Point spread function (PSF): The blur kernel of a lens.• “Diffraction-limited” PSF: No aberrations, only diffraction. Determined by aperture shape.
Optical transfer function (OTF): The Fourier transform of the PSF. Equal to aperture shape.
diffraction-limited PSF of a circular
aperture
blur kernel
diffraction-limited OTF of a circular
aperture
![Page 14: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/14.jpg)
Lens as an optical low-pass filter
image from a perfect lens
*
imperfect lens PSF
=
image from imperfect lens
x * c = b
![Page 15: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/15.jpg)
Lens as an optical low-pass filter
image from a perfect lens
*
imperfect lens PSF
=
image from imperfect lens
x * c = b
If we know c and b, can we recover x?
![Page 16: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/16.jpg)
Deconvolution
x * c = bIf we know c and b, can we recover x?
![Page 17: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/17.jpg)
Deconvolution
x * c = bReminder: convolution is multiplication in Fourier domain:
F(x) . F(c) = F(b)If we know c and b, can we recover x?
![Page 18: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/18.jpg)
Deconvolution
x * c = b
After division, just do inverse Fourier transform:
Reminder: convolution is multiplication in Fourier domain:
F(x) . F(c) = F(b)Deconvolution is division in Fourier domain:
F(xest) = F(c) \ F(b)
xest = F-1 ( F(c) \ F(b) )
![Page 19: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/19.jpg)
Deconvolution
Any problems with this approach?
![Page 20: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/20.jpg)
Deconvolution
Any problems with this approach?
• The OTF (Fourier of PSF) is a low-pass filter
b = c * x + n
• The measured signal includes noise
noise term
zeros at high frequencies
![Page 21: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/21.jpg)
Deconvolution
• When we divide by zero, we amplify the high frequency noise
• The OTF (Fourier of PSF) is a low-pass filter
b = c * x + n
• The measured signal includes noise
noise term
zeros at high frequencies
![Page 22: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/22.jpg)
Naïve deconvolution
* =
b * c-1 = xest
-1
Even tiny noise can make the results awful.• Example for Gaussian of σ = 0.05
![Page 23: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/23.jpg)
Wiener Deconvolution
amplitude-dependent damping factor
Apply inverse kernel and do not divide by zero:
|F(c)|2
xest = F-1 ( ⋅ ) |F(c)|2 + 1/SNR(ω)
F(b)
F(c)
• Derived as solution to maximum-likelihood problem under Gaussian noise assumption• Requires noise of signal-to-noise ratio at each frequency
SNR(ω) =mean signal at ω
noise std at ω
![Page 24: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/24.jpg)
Deconvolution comparisons
naïve deconvolution Wiener deconvolution
![Page 25: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/25.jpg)
Deconvolution comparisons
σ = 0.01 σ = 0.05 σ = 0.01
![Page 26: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/26.jpg)
Wiener Deconvolution
amplitude-dependent damping factor
Apply inverse kernel and do not divide by zero:
|F(c)|2
xest = F-1 ( ⋅ ) |F(c)|2 + 1/SNR(ω)
F(b)
F(c)
• Derived as solution to maximum-likelihood problem under Gaussian noise assumption• Requires noise of signal-to-noise ratio at each frequency
SNR(ω) =mean signal at ω
noise std at ω
![Page 27: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/27.jpg)
Natural image and noise spectraNatural images tend to have spectrum that scales as 1 / ω2
• This is a natural image statistic
![Page 28: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/28.jpg)
Natural image and noise spectraNatural images tend to have spectrum that scales as 1 / ω2
• This is a natural image statistic
Noise tends to have flat spectrum, σ(ω) = constant• We call this white noise
What is the SNR?
![Page 29: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/29.jpg)
Natural image and noise spectraNatural images tend to have spectrum that scales as 1 / ω2
• This is a natural image statistic
Noise tends to have flat spectrum, σ(ω) = constant• We call this white noise
Therefore, we have that: SNR(ω) = 1 / ω2
![Page 30: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/30.jpg)
Wiener Deconvolution
amplitude-dependent damping factor
Apply inverse kernel and do not divide by zero:
|F(c)|2
xest = F-1 ( ⋅ )|F(c)|2 + 1/SNR(ω)
F(b)
F(c)
• Derived as solution to maximum-likelihood problem under Gaussian noise assumption• Requires noise of signal-to-noise ratio at each frequency
SNR(ω) =1
ω2
![Page 31: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/31.jpg)
Wiener Deconvolution
gradient regularization
For natural images and white noise, it can be re-written as the minimization problem
• What does this look like?• How can it be solved?
minx ‖b – c ∗ x‖2 + ‖∇x‖2
![Page 32: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/32.jpg)
Deconvolution comparisons
blurry input gradient regularizationnaive deconvolution original
![Page 33: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/33.jpg)
Deconvolution comparisons
blurry input gradient regularizationnaive deconvolution original
![Page 34: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/34.jpg)
… and a proof-of-concept demonstration
noisy input gradient regularizationnaive deconvolution
![Page 35: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/35.jpg)
Can we do better than that?
![Page 36: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/36.jpg)
Can we do better than that?
Use different gradient regularizations:
minx ‖b – c ∗ x‖2 + ‖∇x‖2
minx ‖b – c ∗ x‖2 + ‖∇x‖1
minx ‖b – c ∗ x‖2 + ‖∇x‖0.8
• L2 gradient regularization (Tikhonov regularization, same as Wiener deconvolution)
• L1 gradient regularization (sparsity regularization, same as total variation)
• Ln<1 gradient regularization (fractional regularization)
All of these are motivated by natural image statistics. Active research area.
How do we solve
for these?
![Page 37: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/37.jpg)
Comparison of gradient regularizations
inputsquared gradient
regularizationfractional gradient
regularization
![Page 38: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/38.jpg)
High quality images using cheap lenses
[Heide et al., “High-Quality Computational Imaging Through Simple Lenses,” TOG 2013]
![Page 39: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/39.jpg)
Deconvolution
* =
x * c = b
If we know b and c, can we recover x?
?
How do we measure this?
![Page 40: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/40.jpg)
PSF calibration
Take a photo of a point source
Image of PSF
Image with sharp lens Image with cheap lens
![Page 41: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/41.jpg)
Deconvolution
* =
x * c = b
If we know b and c, can we recover x?
?
![Page 42: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/42.jpg)
Blind deconvolution
* =
x * c = b
If we know b, can we recover x and c?
? ?
![Page 43: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/43.jpg)
Camera shake
![Page 44: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/44.jpg)
Camera shake as a filter
image from static camera
*
PSF from camera motion
=
image from shaky camera
x * c = b
If we know b, can we recover x and c?
![Page 45: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/45.jpg)
Multiple possible solutions
How do we detect this
one?
![Page 46: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/46.jpg)
Use prior information
Among all the possible pairs of images and blur kernels, select the ones where:
• The image “looks like” a natural image.
• The kernel “looks like” a motion PSF.
![Page 47: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/47.jpg)
Use prior information
Among all the possible pairs of images and blur kernels, select the ones where:
• The image “looks like” a natural image.
• The kernel “looks like” a motion PSF.
![Page 48: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/48.jpg)
Shake kernel statisticsGradients in natural images follow a characteristic “heavy-tail” distribution.
sharp natural image
blurry natural image
![Page 49: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/49.jpg)
Shake kernel statisticsGradients in natural images follow a characteristic “heavy-tail” distribution.
sharp natural image
blurry natural image
Can be approximated by ‖∇x‖0.8
![Page 50: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/50.jpg)
Use prior information
Among all the possible pairs of images and blur kernels, select the ones where:
• The image “looks like” a natural image.
• The kernel “looks like” a motion PSF.
Gradients in natural images follow a characteristic “heavy-tail” distribution.
Shake kernels are very sparse, have continuous contours, and are always positive
How do we use this information for blind deconvolution?
![Page 51: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/51.jpg)
Regularized blind deconvolution
Solve regularized least-squares optimization
minx,b ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖c‖1
What does each term in this summation correspond to?
![Page 52: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/52.jpg)
Regularized blind deconvolution
natural image prior
Solve regularized least-squares optimization
minx,b ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖c‖1
data term shake kernel prior
Note: Solving such optimization problems is complicated (no longer linear least squares).
![Page 53: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/53.jpg)
A demonstration
input deconvolved image and kernel
![Page 54: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/54.jpg)
A demonstration
input deconvolved image and kernel
This image looks worse than the original…
This doesn’t look like a plausible shake kernel…
![Page 55: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/55.jpg)
Regularized blind deconvolution
Solve regularized least-squares optimization
minx,b ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖c‖1
loss function
![Page 56: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/56.jpg)
Regularized blind deconvolution
Solve regularized least-squares optimization
minx,b ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖c‖1
loss functioninverse
loss
pixel intensity
Where in this graph is the solution we find?
![Page 57: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/57.jpg)
Regularized blind deconvolution
Solve regularized least-squares optimization
minx,b ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖c‖1
loss functioninverse
loss
pixel intensityoptimal solution
many plausible solutions here
Rather than keep just maximum, do a weighted
average of all solutions
![Page 58: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/58.jpg)
A demonstration
input maximum-only
This image looks worse than the original…
average
![Page 59: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/59.jpg)
More examples
![Page 60: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/60.jpg)
Results on real shaky images
![Page 61: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/61.jpg)
Results on real shaky images
![Page 62: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/62.jpg)
Results on real shaky images
![Page 63: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/63.jpg)
Results on real shaky images
![Page 64: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/64.jpg)
More advanced motion deblurring
[Shah et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH 2008]
![Page 65: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/65.jpg)
Why are our images blurry?
• Lens imperfections.
• Camera shake.
• Scene motion.
• Depth defocus.
Can we solve all of these problems in the same way?
![Page 66: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/66.jpg)
Why are our images blurry?
• Lens imperfections.
• Camera shake.
• Scene motion.
• Depth defocus.
Can we solve all of these problems in the same way?• No, because blur is not always shift invariant.• See next lecture.
![Page 67: Introduction and course overviewgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture17.pdfthe main motion deblurring and blind deconvolution paper we covered in this lecture](https://reader036.vdocuments.site/reader036/viewer/2022071401/60eca2721fcacf286a10ff09/html5/thumbnails/67.jpg)
ReferencesBasic reading:• Szeliski textbook, Sections 3.4.3, 3.4.4, 10.1.4, 10.3.• Fergus et al., “Removing camera shake from a single image,” SIGGRAPH 2006.
the main motion deblurring and blind deconvolution paper we covered in this lecture.
Additional reading:• Heide et al., “High-Quality Computational Imaging Through Simple Lenses,” TOG 2013.
the paper on high-quality imaging using cheap lenses, which also has a great discussion of all matters relating to blurring from lens aberrations and modern deconvolution algorithms.
• Levin, “Blind Motion Deblurring Using Image Statistics,” NIPS 2006.• Levin et al., “Image and depth from a conventional camera with a coded aperture,” SIGGRAPH 2007.• Levin et al., “Understanding and evaluating blind deconvolution algorithms,” CVPR 2009 and PAMI 2011.• Krishnan and Fergus, “Fast Image Deconvolution using Hyper-Laplacian Priors,” NIPS 2009.• Levin et al., “Efficient Marginal Likelihood Optimization in Blind Deconvolution,” CVPR 2011.
a sequence of papers developing the state of the art in blind deconvolution of natural images, including the use Laplacian (sparsity) and hyper-Laplacian priors on gradients, analysis of different loss functions and maximum a-posteriori versus Bayesian estimates, the use of variational inference, and efficient optimization algorithms.
• Minskin and MacKay, “Ensemble Learning for Blind Image Separation and Deconvolution,” AICA 2000.the paper explaining the mathematics of how to compute Bayesian estimators using variational inference.
• Shah et al., “High-quality Motion Deblurring from a Single Image,” SIGGRAPH 2008.a more recent paper on motion deblurring.