noise, image reconstruction with noise · • noise caused by the ﬂuctuation when the incident...

Noise, Image Reconstruction with Noise

Gordon WetzsteinStanford University

EE367/CS448I: Computational Imaging and Displaystanford.edu/class/ee367

Lecture 10

What’s a Pixel?

source: Molecular Expressions

photon to electron converterà photoelectric effect!

wikipedia

What’s a Pixel?

source: Molecular Expressions

• microlens: focus light on photodiode

• color filter: select color channel

• quantum efficiency: ~50%

• fill factor: fraction of surface area

used for light gathering• photon-to-charge conversion and

ADC includes noise!

ISO (“film speed”)sensitivity of sensor

to light –

digital gain

bobatkins.com

Noise• noise is (usually) bad!

• many sources of noise: heat, electronics, amplifier gain, photon to

electron conversion, pixel defects, read, …

• let’s start with something simple: fixed pattern noise

Fixed Pattern Noise

• dead or “hot” pixels, dust, …

• remove with dark frame calibration:

on RAW image!not JPEG (nonlinear)

Emil Martinec

I =Icaptured − IdarkIwhite − Idark

Noise• other than that, different noise follows different statistical

distributions, these two are crucial:

• Gaussian

• Poisson

Gaussian Noise• thermal, read, amplifier

• additive, signal-independent, zero-mean

+ =

Gaussian Noise

• with i.i.d. Gaussian noise:(independent and identically-distributed)

additive Gaussian noise η ∼ N 0,σ 2( )b = x +η x ∼ N x,0( )

Gaussian Noise


η ∼ N 0,σ 2( )b = x +η x ∼ N x,0( )

b ∼ N x,σ 2( )

p b | x,σ( ) = 12πσ 2

e−b−x( )22σ 2

Gaussian Noise


η ∼ N 0,σ 2( )b = x +η x ∼ N x,0( )

b ∼ N x,σ 2( )

p b | x,σ( ) = 12πσ 2

e−b−x( )22σ 2

• Bayes’ rule:

Gaussian Noise - MAP


η ∼ N 0,σ 2( )b = x +η x ∼ N x,0( )

b ∼ N x,σ 2( )

p b | x,σ( ) = 12πσ 2

e−b−x( )22σ 2

• Bayes’ rule:• maximum-a-posteriori estimation:

Gaussian Noise – MAP “Flat” Prior

p x( ) = 1

• trivial solution (maximum likelihood, not useful in practice):

Gaussian Noise – MAP Self Similarity Prior

• Gaussian “denoisers” like non-local means and other self-similarity priors actually solve this problem:

General Self Similarity Prior

• generic proximal operator for function f(x):

• proximal operator for some image prior


• can use self-similarity as general image prior (not just for denoising)



• can use self-similarity as general image prior (not just for denoising)


σ 2 = λ /σ

σ = λ /σ(h parameter in most NLM

implementations is std. dev.)

Image Reconstruction with Gaussian Noise


η ∼ N 0,σ 2( )b = Ax +η x ∼ N Ax,0( )



η ∼ N 0,σ 2( )b = Ax +η x ∼ N Ax,0( )

b ∼ N Ax,σ 2( )

p b | x,σ( ) = 12πσ 2

e−b−Ax( )22σ 2



η ∼ N 0,σ 2( )b = Ax +η x ∼ N Ax,0( )

b ∼ N Ax,σ 2( )

p b | x,σ( ) = 12πσ 2

e−b−Ax( )22σ 2

• Bayes’ rule:• maximum-a-posteriori estimation:

• regularized least squares (use ADMM)

Scientific Sensors

• e.g., Andor iXon Ultra 897: cooled to -100° C• scientific CMOS & CCD

• reduce pretty much all noise, except for photon noise

What is Photon (or Shot) Noise?

• noise caused by the fluctuation when the incident light is converted to charge (detection)

• also observed in the light emitted by a source, i.e. due to particle

nature of light (emission)

• same for re-emission à cascading Poisson processes are also

described by a Poisson process [Teich and Saleh 1998]

Photon Noise

• noise is signal dependent!

• for N measured photo-electrons

• standard deviation is , variance is

• mean is

• Poisson distribution

σ = N σ 2 = NN

f (k;N ) = N ke−N

k!

Photon Noise - SNR

N photons: 2N photons:

nonlinear!

σ = N

σ = 2 N

signal-to-noise ratio

SNR = NN

N

SNR in dB

1,000 10,000

N photons: 2N photons:

nonlinear!wikipedia

σ = N

σ = 2 N

signal-to-noise ratio

SNR = NN

Photon Noise - SNR

Maximum Likelihood Solution for Poisson Noise

• image formation:

• probability of measurement i:

• joint probability of all M measurements (use notation trick ):

• log-likelihood function:

• gradient:



• Richardson-Lucy algorithm: iterative approach to ML estimation for Poisson noise

• simple idea:

1. at solution, gradient will be zero

2. when converged, further iterations will not change, i.e.

• equate gradient to zero

• rearrange so that 1 is on one side of equation

Richardson-Lucy Algorithm

= 0

• equate gradient to zero

• rearrange so that 1 is on one side of equation

• set equal to


= 0


• for any multiplicative update rule scheme: • start with positive initial guess (e.g. random values)

• apply iteration scheme

• future updates are guaranteed to remain positive

• always get smaller residual

• RL multiplicative update rules:

Richardson-Lucy Algorithm - DeconvolutionBlurry & Noisy Measurement RL Deconvolution


• what went wrong?

• Poisson deconvolution is a tough problem, without priors it’s pretty

much hopeless

• let’s try to incorporate the one prior we have learned: total variation

• log-likelihood function:

• gradient:

Richardson-Lucy Algorithm + TV

• gradient of anisotropic TV term

• this is “dirty”: possible division by 0!


• follow the same logic as RL, RL+TV multiplicative update rules:

• 2 major problems & solution “hacks”:

1. still possible division by 0 when gradient is zero

2. multiplicative update may become negative!


à set fraction to 0 if gradient is 0

à only work with (very) small values for lambda

(A Dirty but Easy Approach to) Richardson-Lucy with a TV Prior

RL

RL+T

V, la

mbd

a=0.

005

RL+T

V, la

mbd

a=0.

025

Mea

sure

men

ts

Log Residual Mean Squared Error

Signal-to-Noise Ratio (SNR)

SNR = mean pixel valuestandard deviation of pixel value

= µσ

signal

noise

= PQetPQet + Dt + Nr

2

P = incident photon flux (photons/pixel/sec)Qe = quantum efficiencyt = eposure time (sec)D = dark current (electroncs/pixel/sec), including hot pixelsNr = read noise (rms electrons/pixel), including fixed pattern noise

Signal-to-Noise Ratio (SNR)

SNR = mean pixel valuestandard deviation of pixel value

= µσ

signal

noise

= PQetPQet + Dt + Nr

2

P = incident photon flux (photons/pixel/sec)Qe = quantum efficiencyt = eposure time (sec)D = dark current (electroncs/pixel/sec), including hot pixelsNr = read noise (rms electrons/pixel), including fixed pattern noise

signal-dependentsignal-independent

Next: Compressive Imaging

• single pixel camera• compressive hyperspectral imaging

• compressive light field imaging

Wakin et al. 2006

Mar

wah

et a

l., 2

013

References and Further Reading• https://en.wikipedia.org/wiki/Shot_noise• L. Lucy, 1974 “An iterative technique for the rectification of observed distributions”. Astron. J. 79, 745–754.

• W. Richardson, 1972 “Bayesian-based iterative method of image restoration” J. Opt. Soc. Am. 62, 1, 55–59.• N. Dey, L. Blanc-Feraud, C. Zimmer, Z. Kam, J. Olivo-Marin, J. Zerubia, 2004 “A deconvolution method for confocal microscopy with total

variation regularization”, In IEEE Symposium on Biomedical Imaging: Nano to Macro, 1223–1226

• M. Teich, E. Saleh, 1998, “Cascaded stochastic processes in optics”, Traitement du Signal 15(6)

• please also read the lecture notes, especially for the “clean” ADMM derivation for solving the maximum likelihood estimation of Poisson

reconstruction with TV prior!

noise, image reconstruction with noise · • noise caused by the ﬂuctuation when the incident...

Documents