Milanfar et al. EE Dept, UCSC 1

“Locally Adaptive Patch-based Image and Video Restoration”

Session I: Today (Mon) 10:30 – 1:00

Session II: Wed

Same Time, Same Room

Thank you for participating inand contributing to our mini-

symposium on

Milanfar et al. EE Dept, UCSC 2

Local Adaptivity + Patch-Based Approaches

• State of the Art Performance A Convergence of Ideas Extremely Popular

Milanfar et al. EE Dept, UCSC 3

Patchy the Pirate

Patch-based methods have become so popular in fact ….

Milanfar et al. EE Dept, UCSC 4

Multi-dimensional Kernel Regression for Video Processing

and Reconstruction

*Joint work with Hiro Takeda (UCSC), Mattan Protter and Michael Elad (Technion),

Peter van Beek (Sharp Labs of America)

SIAM Imaging Science Meeting, July 7, 2008

Peyman Milanfar*EE Department

University of California, Santa Cruz

Milanfar et al. EE Dept, UCSC 5

Outline

• Background and Motivation

• Classic Kernel Regression

• Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel

(AIMS)

• Motion-Aligned Steering Kernel (MASK)

• Conclusions

Milanfar et al. EE Dept, UCSC 6

Summary

• Motivation:– Existing methods make strong assumptions

about signal and noise models.– Develop “universal”, robust methods based on

adaptive nonparametric statistics

• Goal:– Develop the adaptive Kernel Regression

framework for a wide class of problems, including video processing; producing algorithms competitive with state of the art.

Milanfar et al. EE Dept, UCSC 7

Outline

• Background and Motivation

• Classic Kernel Regression

• Data-Adaptive Regression• Adaptive Implicit-Motion Steering Kernel

(AIMS)

• Motion-Aligned Steering Kernel (MASK)

• Conclusions

Milanfar et al. EE Dept, UCSC 8

Kernel Regression Framework

• The data model

A sample

The regression function

Zero-mean, i.i.d noise (No other assump.)

The number of samplesThe sampling position

• The specific form of

may remain unspecified for now.

Milanfar et al. EE Dept, UCSC 9

• The data model

• Local representation (N-terms Taylor expansion)

• Note– With a polynomial basis, we only need to estimate the first unknown,

– Other localized representations are also possible, and may be advantageous.

Local Approximation in KR

Unknowns

Milanfar et al. EE Dept, UCSC 10

Optimization Problem• We have a local representation with respect to each sample:

• MinimizationN+1 terms

This term give the estimated

pixel value at x.

The regression order

The choice of the kernel function is

open, e.g. Gaussian.

Milanfar et al. EE Dept, UCSC 11

Locally Linear Estimator

• The optimization yields a pointwise estimator:

• The bias and variance are related to the regression order and the smoothing parameter:– Large N small bias and large variance

– Large h large bias and small variance

The weighted linear

combinations of the given data

Equivalent kernel

function

Kernelfunction

The smoothingparameter

The regressionorder

Milanfar et al. EE Dept, UCSC 12

Outline

• Background and Motivation

• Classic Kernel Regression

• Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel

(AIMS)

• Motion-Aligned Steering Kernel (MASK)

• Conclusions

Milanfar et al. EE Dept, UCSC 13

(2D) Data-Adaptive Kernels

Classic kernel Data-adapted kernel

• Take not only spatial distances, but also radiometric distances (pixel value differences) into account

• Data-adaptive kernel function

• Yields locally non-linear estimators

Milanfar et al. EE Dept, UCSC 14

Simplest Case: Bilateral Kernels

= .

= .

= .

Low noise case

Spatial kernel

Radiometric kernel

Milanfar et al. EE Dept, UCSC 16

Better: Steering Kernel MethodLocal dominant

orientation estimate based

on local gradient covariance

H. Takeda, S. Farsiu, P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”,IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349-366, February 2007.

Milanfar et al. EE Dept, UCSC 17

Steering Kernel

• Kernel adapted to locally dominant structure

• The steering matrices scale, elongate, and rotate the kernel footprints locally.

Local dominant orientation estimation

Steering matrix

Elongate Rotate Scale

Milanfar et al. EE Dept, UCSC 19

Steering Kernel (Low Noise)

• Kernel weights and footprints:

Low noise case

FootprintsWeightsSteering kernel

as a function of xi with x held fixed

Steering kernel as a function of x with xi and

Hi held fixed

Milanfar et al. EE Dept, UCSC 20

Steering Kernel (High Noise)

High noise case

FootprintsWeightsSteering kernel

as a function of xi with x held fixed

Steering kernel as a function of x with xi and

Hi held fixed

• Steering approach provides stable weights even in the presence of significant noise.

• Kernel weights and footprints:

Milanfar et al. EE Dept, UCSC 21

Some Related (0th-order) Methods

• Non-Local Means (NLM)– A. Buades, B. Coll, and J. M. Morel. “A review of image denoising algorithms, with a new one.” Multiscale

Modeling & Simulation, 4(2):490-530, 2005.

• Optimal Spatial Adaptation (OSA)– C. Kervrann, J. Boulanger “Optimal spatial adaptation for patch-based image denoising.” IEEE Trans. on

Image Processing, 15(10):2866-2878, Oct 2006.

SKR

2 4 6 8 10 12 14 16 18 20

2

4

6

8

10

12

14

16

18

20

2 4 6 8 10 12 14 16 18 20

2

4

6

8

10

12

14

16

18

20

NLM OSA

Milanfar et al. EE Dept, UCSC 22

Adaptive Kernels for Interpolation

• When there are missing pixels:– We cannot have the radiometric distance.

– Using a “pilot” estimate, fill the missing pixels:

• Classic kernel regression

• Cubic or bilinear interpolation ??

?

?

?

?

?

?

?

??

??

?

?

?

Milanfar et al. EE Dept, UCSC 23

Outline

• Background and Motivation

• Classic Kernel Regression

• Data-Adaptive Regression • Regression in 3-D

• Adaptive Implicit-Motion Steering Kernel (AIMS)• Motion-Aligned Steering Kernel (MASK)

• Conclusions

Milanfar et al. EE Dept, UCSC 24

Kernel Regression in 3-D

• Setup is similar to 2-D, but…..

• Data samples come from various (nearby) frames

• Signal “structure” is now in 3-D

• We can perform– Denoising – Spatial Interpolation– Frame rate upconversion– Space-time super-resolution

Spatial gradients

Temporalgradients

Milanfar et al. EE Dept, UCSC 25

Kernel Regression in 3-D Cont.

• Two ways to proceed

– Adaptive Implicit-Motion Steering Kernel (AIMS)

• Roughly warp the data to “neutralize” large motions • Implicitly capture sub-pixel motions in 3-D Kernel

– Motion-Aligned Steering Kernel (MASK)

• Estimate motion with subpixel accuracy• Accurately warp the kernel (instead of the data)

Milanfar et al. EE Dept, UCSC 26

Outline

• Background and Motivation

• Classic Kernel Regression

• Data-Adaptive Regression • Regression in 3-D

• Adaptive Implicit-Motion Steering Kernel (AIMS)• Motion-Aligned Steering Kernel (MASK)

• Conclusions

Milanfar et al. EE Dept, UCSC 27

AIMS Kernel in 3-D

• Steering kernel visualization examples

A plane structure Steering kernel weights Isosurface

A tube structure

Milanfar et al. EE Dept, UCSC 28

AIMS Motion Compensation

• Large displacements make orientation estimation difficult.

• By neutralizing the large displacement, the steering kernel can effectively spread again.

Small motions Large motions

The local kernel effectively spread

along the local motion trajectory.

The local kernel effectively spread

along the local motion trajectory.

Shiftdown

Shiftup

The local kernel after motion

compensation.

Important: The compensation does not require subpixel accurate motion estimation, nor does it require interpolation

Milanfar et al. EE Dept, UCSC 29

AIMS Contains Implicit Motion“Small” motion vector

Optical flow equation

Assuming the patch moves

with approximate uniformity

Homogeneous Optical Flow Vector

(Eigenvalues of C)

Space-time gradientsof roughly compensated data

Milanfar et al. EE Dept, UCSC 30

AIMS Summary

• AIMS is a two-tiered approach.

1. Neutralize whole-pixel motions.

2. 3-D SKR with implicit subpixel motion information

Steering matrices estimated from the

motion compensated data in 3-D.

Milanfar et al. EE Dept, UCSC 32

Foreman Example

Lanczos(frame-by-frame upscaling)

AIMS

Factor of 2 upscaling

Input video(QCIF: 144 x 176 x 28)

Milanfar et al. EE Dept, UCSC 33

Spatial Upscaling Example

Input (200 x 200) Upscaled image by AIMS(multi-frame, 5 frames), 400x400

Milanfar et al. EE Dept, UCSC 34

Spatiotemporal Upscaling

Input video(200 x 200 x 20)

Single framesteering kernelregression(400 x 400 x 20)

Spatiotemporalclassic kernel

regression(400 x 400 x 40)

AIMSregression(400 x 400 x 40)

Milanfar et al. EE Dept, UCSC 35

Outline

• Background and Motivation

• Classic Kernel Regression

• Data-Adaptive Regression • Regression in 3-D

• Adaptive Implicit-Motion Steering Kernel (AIMS)• Motion-Aligned Steering Kernel (MASK)

• Conclusions

Milanfar et al. EE Dept, UCSC 36

Motion-Aligned Steering Kernel

• Motion is explicitly estimated to subpixel accuracy

• Kernel weights are aligned with the local motion vectors using warping/shearing

• The warped kernel acts directly on the data– Handles large and/or complex motions

“2-D motion-steered” (spatial) kernel

1-D (temporal) kernel

Accurate, explicit motion estimates

Milanfar et al. EE Dept, UCSC 37

Intuition Behind the MASK

2-D “motion-steered” (spatial) kernel 1-D (temporal) kernel

Milanfar et al. EE Dept, UCSC 38

The Shapes of MASK

• Spreads along spatial orientations and local motion vectors.

Local dataSlices of

MASK kernels

Milanfar et al. EE Dept, UCSC 40

A Comparison of AIMS and MASK

• Spin Calendar video

Input video(200 x 200 x 20)

AIMS(400 x 400 x 40)

MASK(400 x 400 x 40)

Milanfar et al. EE Dept, UCSC 41

A Comparison of AIMS and MASK

• Foreman video

Input video(QCIF: 144 x 176 x 28)

AIMS + BTV deblurring(CIF: 288 x 352 x 28)

MASK + BTV deblurring(CIF: 288 x 352 x 28)

Milanfar et al. EE Dept, UCSC 42

Conclusions

• We extended the 2-D kernel regression framework to 3-D.– Illustrated 2 distinct approaches

• AIMS: Avoids subpixel motion estimation, needs comp. for large motions

• MASK: Needs subpixel motion estimation, deals directly with large motions

– Which is better? Depends on the application.

• The overall 3-D SKR framework is simultaneously well-suited for spatial, temporal, and spatiotemporal– upscaling, denoising, blocking artifact removal, superresolution– not only in video but in general 3-D data sets.

• Future work– Integration of deblurring directly in the 3-D framework– Computational complexity