sroubekf@utia.cas.cz www.utia.cas.cz

Institute of Information Theory and Automation

Academy of Sciences of the Czech Republic Prague

Digital Image Restoration Blur as a chance and not a nuisance

Filip Šroubek

1 CMP 2017

2 CMP 2017

original image

u

Image Degradation Model

3

channel

convolution

acquired image

= z + n

noise

CMP 2017

Energy Minimization

CMP 2017 4

Data term

Energy Minimization

5 CMP 2017

Data term

Blur regularization

• Blur has different shapes • Compact support • Non-negative • Preserve energy

Energy Minimization

6 CMP 2017

Motion blur

Out-of-focus & Abberations

Image regularization

Energy Minimization

7 CMP 2017

Data term

0 0.2 0.4 0.6 0.8 1

-0.5 0 0.5

Statistics of sharp images

CMP 2017 8

Intensity

Gradient

-0.5 0 0.5

0 0.2 0.4 0.6 0.8 1

Statistics of blurred images

CMP 2017 9

Intensity

Gradient

Regularization

CMP 2017 10

• “NO-BLUR” solution: • WHY? • Regularization favors blurred images

Energy Minimization

11 CMP 2017

Regularization favors blur

CMP 2017 12

5 10 15 200.5

1

1.5

Regularization favors blur

CMP 2017 13

5 10 15 200.5

1

1.5

Artificially sparsify images

Ad-hoc steps!

• To avoid “no-blur” solution:

• Artificial sharpening • Remove spikes • Adjusting priors on the fly • Hierarchical approach

CMP 2017 14

Chan TIP98 Shan SigGraph08 Cho SigGraph09 Xu ECCV09, CVPR13 Almeida TIP10 Krishnan CVPR11 Zhong CVPR13 Sun ICCP13 Michaeli ECCV14 Perrone PAMI15 Pan CVPR16

Artificial Sharpening

15

Shock filter Blur prediction

h-step

Deconvolution u-step

Blurred image

Cho et al., SIGGRAPH 2009

CMP 2017

Remove Spiky Objects

16

Xu et al., ECCV 2010

CMP 2017

Remove Spiky Objects

CMP 2017 17

Mask out small objects

Xu et al., ECCV 2010

Remove Spiky Objects

CMP 2017 18

Reconstructed image with small objects removed

Xu et al., ECCV 2010

Hierarchical Deconvolution

19

Scale N-1

Scale N-2

Scale N

image blur

CMP 2017

Bayesian Paradigm

a posteriori distribution unknown

likelihood given by our problem

a priori distribution our prior knowledge

20 CMP 2017

Blind deconvolution with MAP

21 CMP 2017

Bayesian Paradigm revisited

• Marginalize the posterior

• Maximize the marginalized prob.

• Then maximize the posterior

CMP 2017 22

CMP 2017 23

-0.4-0.6

-0.8-1

-1.21

0.5

0

-0.5

-1

0.5

0.6

0.7

0.8

0.9

1

no-blur solution

correct solution

How to marginalize?

• If Gaussian distributions analytic solution exists in the form of Gaussian distribution

• If not (our case) approximation • Laplace approximation • Factorization with Variational Bayes

CMP 2017 24

Variational Bayes

• Factorization of the posterior

and then marginalization is trivial. • Every factor q depends on moments of other

variables => must be solved iteratively.

CMP 2017 25

Miskin AICA00 Fergus SIGGRAPH06 Tzikas TIP09 Whyte IJCV12 Levin PAMI11 Babacan ECCV12 Wipf JMLR14

Marginalization in blind deconvolution

26 CMP 2017

Limitations of BD

• Gaussian noise

CMP 2017 27

original

SNR=20dB SNR=50dB

Limitations of BD

• Model discrepancies

CMP 2017 28

original

7% discrepancy

Automatic Relevance Determination (Students’ t-distribution) • Standard data term

replaced by

CMP 2017 29

CMP 2017 30

Space-variant Blur

CMP 2017 31

Camera motion

Optical aberrations

Object motion

Scene depth

Approximation of SV Blur

• Patch-wise convolution • General SV PSF • Locally convolution may not hold

• Parametric model (Blur Basis) • More accurate • Model may not hold

• Conversion to space-invariant • HW design

• Local object motion • Segmentation • Line blur

CMP 2017 32

Joshi CVPR08, Sorel ICIP09, Ji CVPR12,Sun CVPR15

Levin NIPS06, Shan ICCV07, Dai CVPR08, Chakrabarti CVPR10, Kim CVPR14

Whyte CVPR10, Gupta ECCV10, Hirsch ICCV11, Zhang NIPS13

Levin SIGGRAPH08, Ben-Ezra PAMI04, Tai PAMI10, Wavefront coding

Fast Moving Objects

• FMO moves over a distance exceeding its size within exposure time

CMP 2017 33

D.R.,J.K.,F.S.,L.N.,J.M. arXiv:1611.07889

Formation model

CMP 2017 34

Alpha matting

FMO Appearance Estimation

CMP 2017 35

FMO Localization

• The pipeline consists of three stages:

CMP 2017 36

1) Explorer 2) Detector 3) Tracker

Input FMO Video

CMP 2017 37

FMO Tracking + Reconstruction

CMP 2017 38

Temporal SR - Table tennis

CMP 2017 39

Temporal SR - Darts

CMP 2017 40

Rotating FMOs

• Simple convolution

replaced by space-variant convolution is a function of trajectory and angular velocity is a 3D object (sphere) • Gradient descent w.r.t • Exhaustive search w.r.t angular velocity

CMP 2017 41

Temporal SR with Rotation

42 CMP 2017

Input video frame Estimated high-speed video

Limitations • Poor collaboration between tracking and restoration

• Estimated blur and FMO appearance improve tracking

• Unstable FMO appearance estimation • Combine multiple video frames

• Track multiple FMOs

CMP 2017 43

Thank You

For Your Attention

44 CMP 2017

original image

u

acquired images

= zk

channel K

channel 2

channel 1

[u * hk]

Multichannel Model

+ nk

noise

46 CMP 2017

Image regularization

term

Blur Regularization

term

Data term

Gaussian noise

L2 norm

Multichannel Model

• Acquisition model:

• Optimization problem

47 CMP 2017

0

Multichannel Blur Regularization

u

h2 * u u h1 * = = z1 z2

48

z1 * * u h1 = * z2 * h2 u * = *

CMP 2017

Camera motion

49 CMP 2017

Degraded images

Reconstructed image

Astronomical images

PSF estimation

50 CMP 2017

original image

u

acquired images

= zk

channel K

channel 2

channel 1

[u * hk]

MC Model with Decimation

+ nk

noise

D

51 CMP 2017

Robustness to misalignment

52

original image PSF degraded image

CMP 2017

Super-resolution

53

Superresolved image (2x) Optical zoom (ground truth)

rough registration

CMP 2017

8 images

SR 3x

SR 2x

original

CMP 2017 54

Input video Super-resolution

Video Super-resolution

55 CMP 2017

Embedded Systems

56 CMP 2017

Acquired blurred image

57 CMP 2017

Blur estimation

58 CMP 2017

Patch-wise Deconvolution

59 CMP 2017

Super-resolution

CMP 2017 60

JPEG SR

Embedded Super-Resolution

Input image

SR from 5 images

SR from 10 images

61 CMP 2017

Patch-wise restoration

62 CMP 2017

63 CMP 2017

Regularization

CMP 2017 64

Adjusting priors

65

• Start with an overestimated noise level and slowly decrease it to the correct level.

• Start with p<<1 and slowly increase it to p=1. CMP 2017