video denoising via bayesian modeling of patchesvideo denoising via bayesian modeling of patches...

Video denoising via Bayesian modeling of patches

Pablo Arias

joint work with Thibaud Ehret and Jean-Michel Morel

CMLA, ENS Paris-Saclay

IHP workshop: Statistical modeling for shapes and imaging

Paris - 14/03/2019

Why are we still researching in denoising

Denoising is mainly for people (i.e. not as much as a pre-processing for a CV task)

I Photography

I Smartphone cameras

I Film industry

I Night vision cameras

I Surveillance cameras

I Medical imaging

I Astronomy

A lot of research has been devoted to still image denoising,

but there is still room for improvement in video denoising.

2

Recursive vs. non-recursive methods

· · · · · ·︸︷︷︸

denoise

frame

· · ·

In this talk: video denoising via Bayesian estimation

of patches, non-recursive and recursive approaches.

3

Recursive vs. non-recursive methods

· · · · · ·︸︷︷︸denoise

frame

· · ·︷︸︸︷

In this talk: video denoising via Bayesian estimation

of patches, non-recursive and recursive approaches.

4

State of the art

Non-recursiveI Non-local means [Buades et al.’05],

[Liu, Freeman ’10]

I K-SVD [Protter, Elad ’07]

I V-BM3D [Dabov et al.’07]

I Graph regularization [Ghoniem et al.

’10]

I BM4D, V-BM4D [Maggioni et al.’12]

I SPTWO [Buades et al.’17]

I VNLB [Arias, Morel’18]

I VIDOSAT [Wen et al.’19]

I SALT [Wen et al.’19]

I VNLnet [Davy et al.’19]

Recursive

I Bilateral + Kalman filter [Zuo et al.’13]

I Bilateral + Kalman filter [Pfleger et

al.’17]

I Recursive NL-means [Ali, Hardie’17]

I Gaussian-Laplacian fusion [Ehmann et

al.’18]

Non-recursive/mixed approaches good results at high computational cost.

Recursive methods: real-time performance but worse results.

5

State of the art

Non-recursiveI Non-local means [Buades et al.’05],

[Liu, Freeman ’10]

I K-SVD [Protter, Elad ’07]

I V-BM3D [Dabov et al.’07]

I Graph regularization [Ghoniem et al.

’10]

I BM4D, V-BM4D [Maggioni et al.’12]

I SPTWO [Buades et al.’17]

I VNLB [Arias, Morel’18]

I VIDOSAT [Wen et al.’19]

I SALT [Wen et al.’19]

I VNLnet [Davy et al.’19]

Recursive

I Bilateral + Kalman filter [Zuo et al.’13]

I Bilateral + Kalman filter [Pfleger et

al.’17]

I Recursive NL-means [Ali, Hardie’17]

I Gaussian-Laplacian fusion [Ehmann et

al.’18]

Non-recursive/mixed approaches good results at high computational cost.

Recursive methods: real-time performance but worse results.

6

Contents

Non-recursive methods

Recursive method I

Recursive method II

Empirical comparison

7

Contents


Recursive method I

Recursive method II


8

Attention! Strange diagrams ahead

9

Video denoising framework

Extract n

similar patchesWiener filtering

Aggregate

by averaging

Estimate model

parameters

[ q1, . . . , qn]

[ q1 , . . . , qn ]

[ p1, . . . , pn]

set as guide

10


Extract n


Aggregate

by averaging

Estimate model

parameters

[ q1, . . . , qn]

[ q1 , . . . , qn ]

[ p1, . . . , pn]

set as guide

11


Extract n


Aggregate

by averaging

Estimate model

parameters

[ q1, . . . , qn]

[ q1 , . . . , qn ]

[ p1, . . . , pn]

set as guide

12


Extract n


Aggregate

by averaging

Estimate model

parameters

[ q1, . . . , qn]

[ g1 , . . . , gn ]

[ p1, . . . , pn]

set as guide

set as guide

13

VNLB: Bayesian patch estimation with Gaussian prior

p1, . . . ,pn : n “similar” clean patches from u

q1, . . . , qn : n “similar” noisy patches from v

Assumption: patches are IID samples of a Gaussian distribution

P(pi) = N (pi |µ, C)

P(qi|pi) = N (qi |pi, σ2I)

For each noisy patch qi estimate pi as the MAP/MMSE, given by the Wiener filter:

pi = µ+ C(C + σ2I)−1(qi − µ)

14

VNLB: Bayesian patch estimation with Gaussian prior

p1, . . . ,pn : n “similar” clean patches from u

q1, . . . , qn : n “similar” noisy patches from v

Assumption: patches are IID samples of a Gaussian distribution

P(pi) = N (pi |µ, C)

P(qi|pi) = N (qi |pi, σ2I)

For each noisy patch qi estimate pi as the MAP/MMSE, given by the Wiener filter:

pi = µ+ C(C + σ2I)−1(qi − µ)

15


Extract n


Aggregate

by averaging

Estimate model

parameters

[ q1, . . . , qn]

[ q1 , . . . , qn ]

[ p1, . . . , pn]

16

Estimating the prior parameters

How to estimate µ and C?

STEP 2: Use the guide patches g1, . . . , gn ∼ N(µ, C):

µ =1n

n∑i=1

gi C =1n

n∑i=1

(gi − µ)(gi − µ)T

STEP 1: Use the noisy pathes q1, . . . , qn ∼ N(µ, C + σ2I):

µ =1n

n∑i=1

qi C ≈ Cq − σ2I =1n

n∑i=1

(qi − µ)(qi − µ)T − σ2I

17

Estimating the prior parameters

We use the following hard-thresholding of the eigenvalues of the sample covariance

matrix of the noisy patches:

λHi = Hτ (ξi − σ2) =

{ξi − σ2 if ξi > τσ2

0 if ξi < τσ2.

Based on the results of Debashis Paul (2007) on the aymptotic properties for the

eigenvalues and eigenvectors of the noisy sample covariance matrix.

18

Examples of local Gaussian models

Groups of 200 similar 9× 9× 4 patches, in a 37× 37× 5 search region.

Nearest neighbors 1 to 5.



19

VLNB: Examples of local Gaussian models

20 nearest neighbors (noisy).

sample mean and first 19 principal directions.

corresponding MAP estimates.

20

Patch search and aggregation

I 3D rectangular patches, typical sizes are 10× 10× 2.

I Motion compensated search region: a square tracing the motion trajectory of the

target patch. Motion estimation using optical flow (TV-L1 [Zach et al. ’07]).

I Search region size: 21× 21× 9

I Two iterations over the whole video (as in BM3D [Dabov et al. ’07]).

21

Contents


Recursive method I

Recursive method II


22

Motivation

ut = recursive-method(ft, ut−1)

Design constraints:

I reduce computational cost

I reduce memory cost (Mt = O(1))

Appealing properties:

I natural mechanism to enforce temporal consistency

I can aggregate an arbitrary number of past frames

23

Motivation

ut,Mt = recursive-method(ft,Mt−1)

Design constraints:

I reduce computational cost

I reduce memory cost (Mt = O(1))

Appealing properties:

I natural mechanism to enforce temporal consistency

I can aggregate an arbitrary number of past frames

24

Dynamic Gaussian patch model

p =

p0 p1 p2 p3 p4

“Static” model:

{p ∼ N (µ, C),

q ∼ N (p, σ2I).

We assume a Gaussian linear dynamic model for a patch trajectory pt:p0 = µ0 +w0, w0 ∼ N (0, P0),

pt = pt−1 +wt, wt ∼ N (0,Wt),

qt = pt + rt, rt ∼ N (0, σ2I).

25


p =

p0 p1p2

p3p4

“Static” model:

{p ∼ N (µ, C),

q ∼ N (p, σ2I).

We assume a Gaussian linear dynamic model for a patch trajectory pt:p0 = µ0 +w0, w0 ∼ N (0, P0),

pt = pt−1 +wt, wt ∼ N (0,Wt),

qt = pt + rt, rt ∼ N (0, σ2I).

26


p =

p0 p1p2p3

p4

Full patch model:

{p ∼ N (µK , Q

−1K ),

q ∼ N (p, σ2I).

µK =

µ0

µ0

µ0

µ0

µ0

, QK =

P−10 +W−1

1 −W−11

−W−11 W−1

1 +W−12 −W−1

2

−W−12 W−1

2 +W−13 −W−1

3

−W−13 W−1

3 +W−14 −W−1

4

−W−14 W−1

5

27

Recursive Bayesian patch estimation: filtering

q =

p =

q10

p2 = E{p2|q2, q1, q0)}p2 = F(q2, p1, P1,W2)}

Kalman filter: recursive computation of P(pt|qt, . . . , q1) ∼ N(pt, Pt

)

pt = (I −Kt)pt−1 +Ktqt state mean

Pt = (I −Kt)(Pt−1 +Wt)(I −Kt)T + σ2K2t state covariance

Kt = (Pt−1 +Wt)(Pt−1 +Wt + σ2I)−1 Kalman gain

28


q =

p =

q10

p2 = F(q2, p1, P1,W2)}p2 = F(q2, p1, P1,W2)}


)




29


q =

p =

q10

p2 = F(q2, p1, P1,W2)}


)




30


q =

p =

q10

p2 = F(q2, p1, P1,W2)}

pt = E{pt|qt, ..., q0)}


)




31

Recursive Bayesian patch estimation: smoothing

q =

p =

q10

p2 = F(q2, p1, P1,W2)}

pt = E{pt|qT , ..., q0)}

Rauch-Tung-Streibel smoother: back-recursion for P(pt|qT , . . . , q1) ∼ N(pt, Pt

)

pt = (I − St)pt + Stpt+1 state mean

Pt = Pt + St(Pt+1 − Pt −Wt+1)St state covariance

St = Pt(Pt +Wt+1)−1 smoothing gain

32

Parameter estimation

The only model parameters that need to be estimated are the state transition

covariances Wt associated to the group.

E{(qt − qt−1)(qt − qt−1)T } = Wt + 2σ2I.

We assume that similar patches are iid realizations of the same dynamic model:

pt,i = pt−1,i +wt,i with wt,i ∼ N (0,Wt) (1)

qt,i = pt,i + rt,i with rt,i ∼ N (0, σ2I). (2)

We estimate Wt via the sample covariance matrix of the innovations:

Wt = βWt−1 + (1− β)

(1n

n∑i=1

(qt,i − qt−1,i)(qt,i − qt−1,i)T − 2σ2I

)+

.

A forgetting factor β ∈ [0, 1] is introduced to increase temporal stability.

33

Forward NL-Kalman filter: frame 1

f1

Extract n


Aggregate

by averaging

Estimate model

parameters

u1

Patch groups 1

FOFt−1→ t

{q1,i}

{q1,i } W

1

{ p1,i }

34

Forward NL-Kalman filter: frame 1

f1

Extract n


Aggregate

by averaging

Estimate model

parameters

u1

Patch groups 1

FOFt−1→ t

{q1,i}

{q1,i } { p

1,i }, P1

W1

W1

coordinates

{ p1,i }

35

Forward NL-Kalman filter: frame t

ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

Patch groups

t − 1 Patch groups t

FOFt−1→ t

coor

dina

test−

1

36


ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

Patch groups


FOFt−1→ t

coor

dina

test−

1

coordinates t

37


ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

Patch groups


FOFt−1→ t

{qt,i }{q

t−1,i }

coor

dina

test−

1

Wt−1

38


ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

Patch groups


FOFt−1→ t

{qt,i }{q

t−1,i }Wt

coor

dina

test−

1

Wt−1

Wt

39


ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

Patch groups


FOFt−1→ t

{qt,i}

{qt,i }{q

t−1,i }

{ pt,i }, P

t

Wt

coor

dina

test−

1

Wt−1

{ pt−1,i }, Pt−1

40


ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

ut

Patch groups


FOFt−1→ t

{qt,i}

{qt,i }{q

t−1,i }

{ pt,i }, P

t

Wt

coor

dina

test−

1

Wt−1

{pt,i}

{ pt−1,i }, Pt−1

41

FNLK: Managing patch trajectories

I Occlusions: Terminate patch trajectories if an occlusion is detected.

I Dis-occlusions: Create groups for parts not covered by existing groups.

Patch groups t

For each of these groups, we store

I coordinates of the patches in the group

I estimated clean patches pt,i

I covariance of estimated patches Pt

I transition covariance matrix Wt

42

Contents


Recursive method I

Recursive method II


43

Dropping the patch groups memory in FNLK

ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

ut

Patch groups


FOFt−1→ t

{qt,i}

{qt,i }{q

t−1,i }Wt { p

t,i }, Pt {

pt,i}

{ pt−1,i }, Pt−1

coor

dina

test−

1

Wt−1

44

Dropping the patch groups memory in FNLK

ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

ut

Patch groups


FOFt−1→ t

{qt,i}

{qt,i }{q

t−1,i }Wt

{ pt,i }

{ pt,i }, P

t {pt,i}

{ pt−1,i }, Pt−1

coor

dina

test−

1

Wt−1

45

Recursive Bayesian patch estimation w/out covariances

pt−1 pt

qt Given pt−1, Pt−1, at time t we havept−1 ∼ N ( pt−1, Pt−1 ),

pt = pt−1 +wt, wt ∼ N (0,Wt),

qt = pt + rt rt ∼ N (0, σ2I).

Kalman filter: Recursive computation of P(pt|qt, . . . , q1) ∼ N(pt, Pt

)




46

Recursive Bayesian patch estimation w/out covariances

pt−1 pt

qt If we don’t have pt−1, Pt−1 we introduce them as parameterspt−1 ∼ N (µt−1, Ct−1 ),

pt = pt−1 +wt, wt ∼ N (0,Wt),

qt = pt + rt rt ∼ N (0, σ2I).

We have the following posterior P(pt|qt) ∼ N(pt, Pt

)

pt = (I −Kt)µt−1 +Ktqt state mean

Pt = (I −Kt)(Ct−1 +Wt)(I −Kt)T + σ2K2t state covariance

Kt = (Ct−1 +Wt)(Ct−1 +Wt + σ2I)−1 “Kalman” gain

47

Parameter estimation for the memoryless model

We assume that similar patches are iid realizations of the same dynamic model:pt−1,i ∼ N (µt−1, Ct−1)

pt,i = pt−1,i +wt,i wt,i ∼ N (0,Wt)

qt,i = pt,i + rt,i rt,i ∼ N (0, σ2I).

µt−1 =1m

m∑i=1

pt−1,i

Ct−1 =1m

m∑i=1

(pt−1,i − µt−1,i)(pt−1,i − µt−1,i)T

Wt =1n

n∑i=1

(qt,i − pt−1,i)(qt,i − pt−1,i)T − σ2.

NOTE: We introduced m to control the spatial averaging in µt−1. Typically m < n.

48

Additional simplification: work in DCT domain

Assumption: Wt = UDiag(νt)UT where U is the DCT basis

Then:

1. Pt = UDiag(νt)UT

2. The Kalman recursion separates in d scalar filters on each DCT component:

49

Backward NL-Kalman filter: frame t

ut−1/ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

ut

Patch groups t

FOFt−1→ t

{qt,i , pt−1,i }

{qt,i , p

t−1,i }

{qt−1,i }Wt

µ t−

1, Ct−

1

{ pt,i }

50


ut−1/ft−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

ut

Patch groups t

BOFt→ t−1

{qt,i , pt−1,i }

{qt,i , p

t−1,i }

{qt−1,i }Wt

µ t−

1, Ct−

1

{ pt,i }

51


uwt−1/fwt−1 ft

Extract n

similar patches

Kalman

filtering

Aggregate

by averaging

Estimate model

parameters

ut

Patch groups t

OFt→ t−1

{qt,i , pt−1,i }

{qt,i , p

t−1,i }

{qt−1,i }Wt

µ t−

1, Ct−

1

{ pt,i }

52

Backward NL-Kalman filtering and smoothingAlgorithm 1: Recursive video filteringinput : Noisy video f , noise level σ

output: Denoised video u

1 for t = 1 . . . T do

2 vbt ← compute-optical-flow(ft, ut−1, σ)

3 uwt−1 ← warp-bicubic(ut−1, vbt)

4 gt ← bwd-nlkalman-filter(ft, uwt−1, σ)

5 ut ← bwd-nlkalman-filter(ft, uwt−1, gt, σ)

Algorithm 2: Recursive video smoothinginput : Noisy video f , noise level σ

output: Denoised video u

1 for t = 1 . . . T do

2 vft ← compute-optical-flow(ut, ut+1, σ)

3 uwt+1 ← warp-bicubic(ut+1, vbt)

4 ut ← bwd-nlkalman-smoother(ut, uwt−1, σ)

53

Three approaches based on Gaussian models of patches

VNLB

I Fixed 3D patch size

I No distinction between space and time

I Does not require OF

I Two iterations

FNLK

I Patch with arbitrary duration

I Processing organized by patch groups

I A lot of house-keeping

I Sensitive to OF

I Costly in memory

BNLK

I Patch with arbitrary duration (kind of)

I Very cheap in memory

I Processing by raster order

I DCT domain (for the moment)

I Two iterations

I Simple smoother

I Sensitive to OF

54

Contents


Recursive method I

Recursive method II


55

Quantitative denoising results (PSNR)

FNLK

BNLK1

BNLK2

BNLK2+S

V-BM3D-np

V-BM4D-mp

BM4D-OF

SPTWO

VNLnet

VNLB

38.27

36.20

38.17

38.69

38.24

38.89

39.54

39.56

40.21

40.46

34.59

32.19

34.59

35.26

34.68

35.10

36.19

35.99

36.47

36.81

30.28

28.20

31.19

31.95

31.11

31.40

32.81

30.93

32.51

32.96

Average PSNR over 7 grayscale sequences 960× 540× 100.

σ = 10 σ = 20 σ = 40

56

Quantitative denoising results (SSIM)

FNLK

BNLK1

BNLK2

BNLK2+S

V-BM3D-np

V-BM4D-mp

BM4D-OF

SPTWO

VNLnet

VNLB

.9575

.9278

.9570

.9609

.9599

.9534

.9671

.9675

.9732

.9731

.9164

.8447

.9163

.9251

.9100

.9169

.9371

.9368

.9414

.9428

.8245

.8460

.8648

.8360

.8432

.8836

.7901

.8752

.8856

Average SSIM over 7 grayscale sequences 960× 540× 100.

σ = 10 σ = 20 σ = 40

57

Conclusions and future work

I Current state-of-the-art in video denoising: either good results or fast results.

I Presented two recursive approaches bridging the gap between costly good methods

and fast methods.

I They integrate information accross longer time ranges and are allow to recover

many more details.

I Still very sensitive to optical flow.

Ongoing work

I Joint denoising and optical flow computation.

I Implement BNLK in an adaptive basis (instead of DCT).

I Fixed lag smoothers.

I Multiscale versions.

58

Thank you!

Reproducibility! Code and results:

I Non-recursive results:

http://dev.ipol.im/˜pariasm/video_denoising_models/

I VNLB: http://github.com/pariasm/vnlb

I SPTWO: https://doi.org/10.5201/ipol.2018.224

I BM4D-OF: https://github.com/pariasm/vbm3d

I BNLK: http://github.com/pariasm/bwd-nlkalman

I FNLK: http://github.com/tehret/nlkalman

59

http://dev.ipol.im/~pariasm/video_denoising_models/

http://github.com/pariasm/vnlb

https://doi.org/10.5201/ipol.2018.224

https://github.com/pariasm/vbm3d

http://github.com/pariasm/bwd-nlkalman

http://github.com/tehret/nlkalman

References I[Dabov’07] K. Dabov, A. Foi, V. Katkovnik, K. Egiazarian. Image denoising by sparse 3Dtransform-domain collaborative filtering. IEEE Trans. on Image Processing, 16, 2007.

[Mairal’09] J. Mairal, F. Bach, J. Ponce, G. Sapiro and A. Zisserman, Non-local sparse models forimage restoration. CVPR 2009.

[Ji et al.’10] H. Ji, C. Liu, Z. Shen, Y. Xu, Robust video denoising using low-rank matrixcompletion. CVPR 2010.

[He et al.’11] Y. He, T. Gan, W. Chen, H. Wang, Adaptive denoising by Singular ValueDecomposition. IEEE Signal Processing Letters, 18(4), 2011.

[Zoran’11] D. Zoran and Y. Weiss, From learning models of natural image patches to wholeimage restoration, ICCV 2011.

[Wang et al.’12] Wang S., Zhang L., Liang Y., Nonlocal Spectral Prior Model for Low-LevelVision. ACCV 2012.

[Yu et al.’12] G. Yu, G. Sapiro and S. Mallat, Solving Inverse Problems With Piecewise LinearEstimators: From Gaussian Mixture Models to Structured Sparsity, IEEE TIP, 21(5), 2012.

60

References II

[Dong et al.’13] Dong, W., Shi, G., Li, X., Nonlocal Image Restoration With Bilateral VarianceEstimation: A Low-Rank Approach, IEEE TIP, 22(2), 2013.

[Lebrun’13] M. Lebrun, A. Buades and J.M. Morel. A Nonlocal Bayesian image denoisingalgorithm. SIAM Journal on Imaging Sciences, 6, 2013.

[Gu et al.’14] S. Gu, L. Zhang, W. Zuo, X. Feng, Weighted Nuclear Norm Minimization withApplication to Image Denoising, CVPR 2014.

[Guo et al.’16] Q. Guo, C. Zhang, Y. Zhang and H. Liu, An Efficient SVD-Based Method forImage Denoising, IEEE Trans. on Circuits and Systems for Video Tech., 26(5), 2016.

[Badri et al.’16] H. Badri, H. Yahia and D. Aboutajdine, Low-Rankness Transfer for RealisticDenoising, in IEEE TIP, 25(12), 2016.

[Xie et al.’16] Y. Xie, S. Gu, Y. Liu, W. Zuo, W. Zhang and L. Zhang, Weighted Schatten p-NormMinimization for Image Denoising and Background Subtraction,” in IEEE TIP, 25(10), 2016.

61

Empirical Wiener filters in high dimensions

ML estimator of the inverse covariance matrix Qy = C−1y results from the following

convex SDP:max log detQy − tr(QyCy)

subject to 0 ≺ Qy � σ−2Id.

Although there are efficient algorithms for solving such an SDP, they are still

prohibitive for the present application.

62

Convergence of the sample covariance matrix

Spiked covariance model: Samples x1, . . . ,xn ∼ N (0, C). We observe

yi ∼ N (xi, σ2I).

C = U Diag(λ1, . . . , λm, 0, . . . , 0)UT .

Theorem (Paul 2007). Suppose that d/n→ γ ∈ (0, 1) as n→∞. Let ξi be the ith

eigenvector of the sample covariance matrix Cy . Then ξi converges almost surely to

ξi →

σ2(1 +√γ)2 if λi 6√γσ2,

f(λi) := (λi + σ2)(

1 +γσ2

λi

)if λi >

√γσ2.

We can estimate λi as λSi (ξi) =

{0 if ξi 6 σ2(1 +√γ)2,

f−1(ξi) if ξi > σ2(1 +√γ)2.

63

Convergence of the sample covariance matrix

Spiked covariance model: Samples x1, . . . ,xn ∼ N (0, C). We observe

yi ∼ N (xi, σ2I).

C = U Diag(λ1, . . . , λm, 0, . . . , 0)UT .

Theorem (Paul 2007). Suppose that d/n→ γ ∈ (0, 1) as n→∞. Let ξi be the ith

eigenvector of the sample covariance matrix Cy . Then ξi converges almost surely to

ξi →

σ2(1 +√γ)2 if λi 6√γσ2,

f(λi) := (λi + σ2)(

1 +γσ2

λi

)if λi >

√γσ2.

We can estimate λi as λSi (ξi) =

{0 if ξi 6 σ2(1 +√γ)2,

f−1(ξi) if ξi > σ2(1 +√γ)2.

64

Estimators for the eigenvalues of Cx

0 50 100 150 200 250 300−50

0

50

100

150

200

250

ξ

λ

λS(ξ)

ξ −σ2(1 +γ)

65

Simulations

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

0 10 20 30 40 50 600

5

10

15

20

25

30

λ = λλ = ξ − σ2

λ = λS(ξ)√

γσ2

66

Variance threshold

We propose an additional estimator by hard thresholding the difference ξ − σ2. The

value of the threshold is given by a parameter τ :

λHi = Hτ (ξi − σ2) =

{ξ − σ2 if ξ > τσ2

0 if ξ < τσ2.

67

Denoising performance of both estimators

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

0 1 2 3 4 5 6 7 81

1.5

2

2.5

3

3.5

4

τ

MSE/MMSE

λ = λλ = ξ − σ2

λ = λS(ξ)λ = Hτ (ξ − σ2)

Normalized MSE obtained by estimating the n data samples from their noisy

observations using empirical Wiener filters with different estimators for the a priori

variances λ.

68

video denoising via bayesian modeling of patchesvideo denoising via bayesian modeling of patches...

Documents