multi-scale statistical image models and denoising
TRANSCRIPT
![Page 1: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/1.jpg)
Multi-scale Statistical Image
Models and Denoising
Eero P. Simoncelli
Center for Neural Science, and
Courant Institute of Mathematical Sciences
New York University
http://www.cns.nyu.edu/~eero
![Page 2: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/2.jpg)
Multi-scale roots
Multigrid solvers
for PDEs
Signal/image
processing
Biological vision
models
![Page 3: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/3.jpg)
The “Wavelet revolution”
• Early 1900’s: Haar introduces first orthonormal wavelet
• Late 70’s: Quadrature mirror filters
• Early 80’s: Multi-resolution pyramids
• Late 80’s: Orthonormal wavelets
• 90’s: Return to overcomplete (non-aliased) pyramids,
especially oriented pyramids
• >250,000 articles published in past 2 decades
• Best results in many signal/image processing applications
![Page 4: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/4.jpg)
“Laplacian” pyramid
[Burt & Adelson, ‘81]
![Page 5: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/5.jpg)
Multi-scale gradient basis
• Multi-scale bases: efficient representation
• Derivatives: good for analysis
• Local Taylor expansion of image structures
• Explicit geometry (orientation)
• Combination:
• Explicit incorporation of geometry in basis
• Bridge between PDE / harmonic analysis
approaches
![Page 6: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/6.jpg)
“Steerable”
pyramid
[Simoncelli, Freeman, Heeger, Adelson, ‘91]
![Page 7: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/7.jpg)
Steerable pyramid
• Basis functions are Kth derivative operators, related by
translation/dilation/rotation
• Tight frame (4(K-1)/3 overcomplete)
• Translation-invariance, rotation-invariance
[Freeman & Adelson 1991; Simoncelli et.al., 1992; Simoncelli & Freeman 1995]
![Page 8: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/8.jpg)
Denoising
![Page 9: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/9.jpg)
Pyramid denoising
How do we distinguish
signal from noise?
![Page 10: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/10.jpg)
Bayesian denoising framework
• Signal: x
• Noisy observation: y
• Bayes’ least squares (BLS) solution is
conditional mean:
x̂(y) = IE(x|y)
∝
∫x
x P(y|x) P(x)
![Page 11: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/11.jpg)
Image statistical models
I. (1950’s): Fourier transform + Gaussian marginals
II. (late 80’s/early 90’s): Wavelets + kurtotic marginals
III. (late 90’s - ): Wavelets + adaptive local variance
Substantial increase in model accuracy
(at the cost of increased model complexity)
![Page 12: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/12.jpg)
I. Classical Bayes denoising
If signal is Gaussian, BLS estimator is linear:
den
oise
d(x̂
)
noisy (y)
x̂(y) =σ
2x
σ2x
+ σ2n
· y
![Page 13: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/13.jpg)
Coefficient distributions
Wavelet coefficient value
log(
Pro
babi
lity)
p = 0.46!H/H = 0.0031
Wavelet coefficient valuelo
g(P
roba
bilit
y)
p = 0.58!H/H = 0.0011
Wavelet coefficient value
log(
Pro
babi
lity)
p = 0.48!H/H = 0.0014
Wavelet coefficient value
log(
Pro
babi
lity)
p = 0.59!H/H = 0.0012
Fig. 4. Log histograms of a single wavelet subband of four example images (see Fig. 1 for image description). For eachhistogram, tails are truncated so as to show 99.8% of the distribution. Also shown (dashed lines) are fitted model densitiescorresponding to equation (3). Text indicates the maximum-likelihood value of p used for the fitted model density, andthe relative entropy (Kullback-Leibler divergence) of the model and histogram, as a fraction of the total entropy of thehistogram.
non-Gaussian than others. By the mid 1990s, a numberof authors had developed methods of optimizing a ba-sis of filters in order to to maximize the non-Gaussianityof the responses [e.g., 36, 4]. Often these methods oper-ate by optimizing a higher-order statistic such as kurto-sis (the fourth moment divided by the squared variance).The resulting basis sets contain oriented filters of differentsizes with frequency bandwidths of roughly one octave.Figure 5 shows an example basis set, obtained by opti-mizing kurtosis of the marginal responses to an ensembleof 12 × 12 pixel blocks drawn from a large ensemble ofnatural images. In parallel with these statistical develop-ments, authors from a variety of communities were devel-oping multi-scale orthonormal bases for signal and imageanalysis, now generically known as “wavelets” (see chap-ter 4.2 in this volume). These provide a good approxima-tion to optimized bases such as that shown in Fig. 5.
Once we’ve transformed the image to a multi-scalewavelet representation, what statistical model can we useto characterize the the coefficients? The statistical moti-vation for the choice of basis came from the shape of themarginals, and thus it would seem natural to assume thatthe coefficients within a subband are independent andidentically distributed. With this assumption, the modelis completely determined by the marginal statistics of thecoefficients, which can be examined empirically as in theexamples of Fig. 4. For natural images, these histogramsare surprisingly well described by a two-parameter gen-eralized Gaussian (also known as a stretched, or generalizedexponential) distribution [e.g., 31, 47, 34]:
Pc(c; s, p) =exp(−|c/s|p)
Z(s, p), (3)
where the normalization constant is Z(s, p) = 2 spΓ( 1
p ).An exponent of p = 2 corresponds to a Gaussian den-sity, and p = 1 corresponds to the Laplacian density. In
Fig. 5. Example basis functions derived by optimizing amarginal kurtosis criterion [see 35].
5
P (x) ∝ exp−|x/s|p
[Mallat, ‘89; Simoncelli&Adelson ‘96; Mouline&Liu ‘99; etc]
Well-fit by a generalized Gaussian:
![Page 14: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/14.jpg)
II. Bayesian coring
• Assume marginal distribution:
• Then Bayes estimator is generally nonlinear:
P (x) ∝ exp−|x/s|p
p = 2.0 p = 1.0 p = 0.5
[Simoncelli & Adelson, ‘96]
![Page 15: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/15.jpg)
Joint statistics
• Large-magnitude values are found at neighboring
positions, orientations, and scales.
[Simoncelli, ‘97; Buccigrossi & Simoncelli, ‘97]
![Page 16: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/16.jpg)
Joint statistics
[Simoncelli, ‘97; Buccigrossi & Simoncelli, ‘97]
-40 0 40 50
0.2
0.6
1
-40 0 40
0.2
0.6
1
0
-40
0
40
-40 40
![Page 17: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/17.jpg)
Joint GSM model
Model generalized neighborhood of coefficients as a Gaus-sian Scale Mixture (GSM) [Andrews & Mallows ’74]:
!x =√
z !u, where
- z and !u are independent
- !x|z is Gaussian, with covariancezCu
- marginals are always leptokur-totic
[Wainwright & Simoncelli, ’99]
IPAM, 9/04 16
![Page 18: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/18.jpg)
Simulation
!!" " !"#"
"
#"!
!!" " !"#"
"
#"!
Image data GSM simulation
[Wainwright & Simoncelli, ‘99]
![Page 19: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/19.jpg)
III. Joint Bayes denoising
IE(x|!y) =∫
dz P(z|!y) IE(x|!y, z)
=∫
dz P(z|!y)
zCu(zCu + Cw)−1!y
ctr
where
P(z|!y) =P(!y|z) P(z)
P!y, P(!y|z) =
exp(−!yT (zCu + Cw)−1!y/2)√
(2π)N |zCu + Cw|
Numerical computation of solution is reasonably efficient ifone jointly diagonalizes Cu and Cw ...
[Portilla, Strela, Wainwright, Simoncelli, ’03]
IPAM, 9/04 20
![Page 20: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/20.jpg)
Example joint estimator
!!"
"
!"
!#"
"
#"
!!"
"
!"
$%&'()*%+,,-$%&'()./0+$1
+'1&2/1+3)*%+,,-
[Portilla, Wainwright, Strela, Simoncelli, ‘03;
see also: Sendur & Selesnick, ‘02]
![Page 21: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/21.jpg)
Denoising Simulation: Face
noisy(4.8)
I-linear(10.61)
II-marginal(11.98)
III-GSMnbd: 5 × 5 + p
(13.60)
- Semi-blind (all parameters estimated except for σw).- All methods use same steerable pyramid decomposition.- SNR (in dB) shown in parentheses.
Snowbird, 4/04 23
joint
![Page 22: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/22.jpg)
OriginalNoisy
(22.1 dB)
Matlab’s
wiener2
(28 dB)
BLS-GSM
(30.5 dB)
![Page 23: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/23.jpg)
OriginalNoisy
(8.1 dB)
UndecWvlt
HardThresh
(19.0 dB)
BLS-GSM
(21.2 dB)
![Page 24: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/24.jpg)
Real sensor noise
400 ISO denoised
![Page 25: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/25.jpg)
Comparison to other methods
!" #" $" %" &"!$'&
!$
!#'&
!#
!!'&
!!
!"'&
"
"'&
()*+,-*.)/-0123
,456748+91:;<=:>6965#8*>?6<
Relative PSNR
improvement as a
function of noise level
(averaged over three
images):
- squares: Joint model
- diamonds: soft thresholding, optimized threshold [Donoho, '95]
- circles: MatLab wiener2, optimized neighborhood [Lee, '80]
![Page 26: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/26.jpg)
Pyramid denoising
How do we distinguish
signal from noise?
![Page 27: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/27.jpg)
“Steerable”
pyramid
[Simoncelli, Freeman, Heeger, Adelson, ‘91]
![Page 28: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/28.jpg)
orientation
orientation
magnitude
[Hammond & Simoncelli, 2005; cf. Oppenheim & Lim 1981]
![Page 29: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/29.jpg)
Importance of local orientation
Randomized orientation Randomized magnitude
Two-band, 6-level steerable pyramid
[with David Hammond]
![Page 30: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/30.jpg)
Reconstruction from orientation
• Alternating projections onto convex sets
• Resilient to quantization
• Highly redundant, across both spatial position and scale
Quantized to 2 bits
[with David Hammond]
Original
![Page 31: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/31.jpg)
Spatial redundancy
• Relative orientation histograms, at different locations
• See also: Geisler, Elder
[with Patrik Hoyer & Shani Offen]
x??
y
y
x
![Page 32: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/32.jpg)
Scale redundancy
[with Clementine Marcovici]
![Page 33: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/33.jpg)
Conclusions
• Multiresolution pyramids changed the
world of image processing
• Statistical modeling can provide refinement
and optimization of intuitive solutions:
- Wiener
- Coring
- Locally adaptive variances
- Locally adaptive orientation
![Page 34: Multi-scale Statistical Image Models and Denoising](https://reader031.vdocuments.site/reader031/viewer/2022012518/61926325225896191c711066/html5/thumbnails/34.jpg)
Cast
• Local GSM model: Martin Wainwright, Javier Portilla
• Denoising: Javier Portilla, Martin Wainwright, Vasily
Strela, Martin Raphan
• GSM tree model: Martin Wainwright, Alan Willsky
• Local orientation: David Hammond, Patrik Hoyer,
Clementine Marcovici
• Local phase: Zhou Wang
• Texture representation/synthesis: Javier Portilla
• Compression: Robert Buccigrossi