Yu-Wing Tai, Hao Du, Michael S. Brown, Stephen LinCVPR’08

(Longer Version in Revision at IEEE Trans PAMI)

Google Search: Video Deblurring

Spatially Varying Deblur

Image/Video Deblurring using a Hybrid Camera

Project Page (visit): http://www.comp.nus.edu.sg/~yuwing 1 / 25

Image Deblurring: The ProblemGiven a motion blurred image, we want to

recover a sharp image:

Input Desired Output

Point Spread Function (PSF)Motion blur Kernel

2 / 25

Blur kernel is known:

Why this is a difficult problem ?

This is an ill-posed under constrained problem:Different inputs can produce the same (very similar) output after convolution

3 / 25

Blind deconvolution problemBlur kernel is unknown:

4 / 25

Two causes for motion blur

Hand shaking(Camera “ego motion”)

Object motion

Blur is the same

Blur is different

5 / 25

Properties of motion blurHand shaking

PSF is globally the same for the whole imageObservations are the whole imageDeconvolution is a global processRelatively Easy – ``Well studied, some current works

produce very good results’’

Object MotionPSF is varying across the whole imageObservations are only valid for local regionsDeconvolution is a local processHave problem of mixing colorsMight have problem of occlusions and disocclussionsVery Difficult – ``Nothing closed, there is still have

no good solution’’

6 / 25

Related works (Hand shaking) Traditional approaches:

Wiener filter [Wiener, 1949]Richardson and Lucy [Richardson 1972; Lucy 1974]

Recent approaches:Regularization based:

Total variation regularization [Dey et al. 2004]Natural image statistics [Fergus et al. Siggraph 2006]Alpha matte [Jia CVPR 2007]Multiscale regularization [Yuan et al. Siggraph 2008]High-order derivatives of gaussian model [Shan et al. Siggraph 2008]

Auxiliary information:Different exposure [Ben-Ezra and Nayar, CVPR 2003]Flutter shutter [Raskar et al. Siggraph 2006]Coded aperture and sparsity prior [Levin et al. Siggraph 2007]Blurred and noisy pairs [Yuan et al. Siggraph 2007]Two blurred Images [Rav-Acha and Peleg2005; Chen and Tang

CVPR2008]

arg min I,K f(I◦K – B)arg min I,K f(I◦K – B)

arg min I,K f(I◦K – B) + Regularization Termsarg min I,K f(I◦K – B) + Regularization Terms

arg min I,K f(I◦K’ – B’) + Regularization Termsarg min I,K f(I◦K’ – B’) + Regularization Terms

7 / 25

Related works (Object motion)Translational motion

Natural image statistics [Levin, NIPS 2006]Two blurred Images [Cho et al. ICCV 2007]Motion Invariant Photography [Levin et al., Siggraph 2008]

In-plane rotational motion Shan et al. ICCV 2007

Our approach [CVPR 2008]Handle motion blur from both hand shaking and object

movingHandle translational, in-plane/out-of-plane rotational,

zoom-in motion blur in a unified framework

8 / 25

Basic idea [Ben-Erza CVPR’03]Observation: Tradeoff between Resolution and Exposure Time

Motion blur exist in high resolution images.

time

Hi-ResolutionLow Frame-rate

Low-ResolutionHi-Frame-rate

Our goal is to deblur the high resolution images with assistance from low resolution, high frame rate video. 9 / 25

Our Hybrid Camera

Hi-Res: 1024 x768 resolution at 25 fpsLow-Res: 128 x 96 resolution at 100 fps. A beam-splitter is use to align their optical axesDual-video capture synchronized by hardware

Low-Res Camera

High-Res Camera

Beam-splitter

10 / 25

Low-Resolution High Frame-rate

Spatially-varying motion blur kernels can be approximated by motion vector from low resolution video

Observation 1

Motion Blur Kernels KHi-Resolution Low Frame-rate

11 / 25

The deblured image, after down-sampling, should look similar to the low resolution image

Observation 2

Deblurred Hi-Resolution Image Low-Resolution Image

12 / 25

Bayesian ML/MAP model:

I : Deblurred ImageK: Estimated Blur KernelIb: Observed High Resolution Blur ImageIl: Observed Low Resolution Shape Image Sequences

Ko: Observed Blur Kernel from optical flow computation

Our Formulation (Main Algorithm)

13 / 25

Optimization ProcedureGlobal Invariant Kernel (Hand Shaking)

Spatially varying Kernels (Object Moving)

Deconvolution Eq. Low Resolution Reg. Kernel Reg.

14 / 25

Moving object appears sharp in the high-frame-rate low-resolution video

Perform binary moving object segmentation in the low-resolution imagesCompose the binary masks with smoothing to

approximate the alpha matte in the high-resolution image

Moving Object Extraction

Problem with mixing color

15 / 25

ResultsImage Deblurring:

Hand-shaking Motion Blur (Global Motion)In-plane Rotational Motion BlurTranslational MotionZoom-in motion

Video DeblurringMoving box: arbitrary in-plane motionMoving car towards camera: translational + zoom

in motion

16 / 25

Hand Shaking (Motion blur = Global)

Results

Input [Fregus et. al. Siggraph’06] [Ben-Ezra et. al. CVPR’03]

Back Projection Our Result Ground Truth

17 / 25

Rotational Motion (Motion Blur = Spatially-varying)

Results

Input [Shan et. al. ICCV’07] [Ben-Ezra et. al. CVPR’03]

Back Projection Our Result Ground Truth

18 / 25

Translational Motion (Motion Blur = Global for object)

Results

Input [Fregus et. al. Siggraph’06] [Ben-Ezra et. al. CVPR’03]

Back Projection Our Result Ground Truth

19 / 25

Zoom-in motion (Motion Blur = Spatially-varying)

Results

Input [Fregus et. al. Siggraph’06] [Ben-Ezra et. al. CVPR’03]

Back Projection Our Result Ground Truth

20 / 25

Results (moving object)

In-plane Rotation 21 / 25[Show video]

Results (moving object)

Out-of-plane Motion (zoom translate) 22 / 25[Show video]

Limitations and DiscussionHigh frequency lost during the convolution

process cannot be recoveredSmall ringing artifacts cannot be removedBasic assumptions:

Constant Illumination during exposureRigid objectsMoving objects are not overlapped

Problems in separating moving objects from moving background

23 / 25

Hybrid camera frameworkExtended to spatially varying motion blurExtended to video

Combined Deconvolution and BackprojectionEffective in reducing ringing artifactsEffective in recovering motion blurred details

Formulated into a Bayesian ML/MAP Solution

Summary of Image/Video Deblurring

24 / 25

Personal Homepage: http://www.comp.nus.edu.sg/~yuwing/

Thank you! (Question/Answers)

25 / 25