fast non-uniform deblurring using constrained camera pose
TRANSCRIPT
Fast Non-uniform Deblurring
Fast Non-uniform Deblurring usingConstrained Camera Pose Subspace
Zhe Hu and Ming-Hsuan Yang
Electrical Engineering and Computer ScienceUniversity of California at Merced
Merced, CA 95343
1 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Non-uniform Image Deblurring
Static scene
Long exposure time/dim light condition
Different blur effects on different pixels
2 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Non-uniform Deblurring Algorithms
Segment-wise Uniform Deblur
[Bardsley et al. Optics06], [Cho et al. ICCV07]
Additional Image Sequence/Hybrid Camera System
[Tai et al. CVPR08], [Joshi et al. SIGGRAPH10]
Single Image Deblurring with a Global Model
[Whyte et al. CVPR10], [Gupta et al. ECCV10], [Harmeling et al.NIPS10], [Hirsch et al. ICCV11]
3 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Problem Formulation
Model the blurred input B as the integration of intermediateimages captured at different poses of a motion trajectory [Whyteet al. CVPR10],
B =
∫f (Hθ, L)w(θ)dθ + n (1)
L: latent image, n: noiseθ: camera pose, Hθ: homography induced by camera posef (Hθ, L): transformed copy of the sharp imagew(θ): exposure time at each pose
4 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Discretized Formulation
B =∑θ∈S
f (Hθ, L)wθ + n =∑θ∈S
wθ(KθL) + n (2)
Kθ: matrix that warps latent image L to the transformed copyS : camera pose space
Optimization Problem
min(L,W )
||(∑θ∈S
wθKθ)L− B||2 + Φ1(L) + Φ2(W ) (3)
Φ1, Φ2: regularization terms on L and W
5 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Main Challenges
Lack of good initialization technique of camera motion
High computational load
CPU execution timeof a 441× 611 image
Whyte et al. CVPR10 about 3 hours
Gupta et al. ECCV10 about 1 hour
Hirsch et al. ICCV11 about 1500 secs
6 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Main Steps
1 Initialization
2 Estimating W by fixing L
3 Recovering L by fixing W
7 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
1. Initialization
Importance of Initialization
Avoid numerous solutions at local minimum
Fast convergence
Our Approach
Motivated by backprojection in image processing:reconstruct 2D signal from its 1D projections
8 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Computed tomography (CT) views. Each sample acquired in aview is equal to the sum of the image values along a ray pointingto that sample.
9 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
A backprojection is formed by duplicating each view through theimage in the direction it was originally acquired.
10 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization
Our Approach
Motivated by backprojection in image processing:reconstruct 2D signal from its 1D projections
Backproject local blur kernels (PSFs) to initialize cameramotion: reconstruct 3D camera motion from 2D blur kernels
11 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blurred image.
Ground-truth camera motion(in-plane rotation only).
x ,y axis: in-plane translationsz axis: in-plane rotation
12 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Selected regions used to estimatelocal blur kernels.
Corresponding estimated blur kernels.
13 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blur kernel to be backprojected.
Backprojected camera motion.
14 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blur kernel to be backprojected.
Backprojected camera motion.
15 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blur kernel to be backprojected.
Backprojected camera motion.
16 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blur kernel to be backprojected.
Backprojected camera motion.
17 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blur kernel to be backprojected.
Backprojected camera motion.
18 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Initialization Using Backprojection
Blur kernel kl : projection of camera motion trajectory,
kl =∑θ
wθp(θ, l) (4)
p(θ, l): how camera pose θ affects pixel l
Backprojection: approximate camera motion
bp(kl , l) =∑i
∑{θ|p(θ,l)=i}
kl(i)Γ(θ) (5)
i : element index of blur kernel klΓ(θ): indicator function, 1 for pose θ and 0 for others
Initialization by backprojection:
W =1
n
∑l
bp(kl , l) (6)
19 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Main Steps
1 Initialization
2 Estimating W by fixing L
3 Recovering L by fixing W
20 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
2. Weight Estimation on Constrained Pose Subspace
Camera motion estimation: weight estimation
minW||∑θ∈S
wθ(KθL)− B||2 + Φ2(W ) (7)
Kθ: the matrix that warps latent image L to the transformed copyL, B: column vectors
21 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Weight Estimation on Constrained Pose Subspace
Consider Kθ
Large matrix mn ×mn (image size m × n)
High computational load if |S | is large
Active Set
Motion trajectory is sparse in the pose space
Find a subset A ∈ S to cover the poses lie on the motiontrajectory
Estimate weights on A
22 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Weight Estimation on Constrained Pose Subspace
How to determine the active set?
Initialize A by thresholding the backprojected camera motion
At each iteration, discard the poses with small weights andadd in poses by sampling
Sampling based on the previous A using a Gaussiandistribution
23 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Weight Estimation on Constrained Pose Subspace
Active set from last iteration
24 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Weight Estimation on Constrained Pose Subspace
Discard poses with small weights
25 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Weight Estimation on Constrained Pose Subspace
Add in new poses by sampling and form the new active set
26 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Weight Estimation on Constrained Pose Subspace
Original formulation
minW||∑θ∈S
wθ(KθL)− B||2 + Φ2(W ) (8)
Using active set
minW||∑θ∈A
wθ(KθL)− B||2 + Φ2(W ) (9)
Pose numberused per iteration
Execution timeper iteration
warping matrixto be computed
Original Thousands 2 mins ThousandsUsing active set Less than 100 Less than 1 min Around 200 - 300
27 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Main Steps
1 Initialization
2 Estimating W by fixing L
3 Recovering L by fixing W
28 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
3. Recovering Latent Image
Estimating latent image L by fixing weights W as [Gupta et al.ECCV10],
minL||KL− B||2 + Φ1(L) (10)
where K =∑
θ∈A wθKθ is the blur matrix within the active set A
29 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
MATLAB implementation: 600 seconds to process a blurryimage of 441× 611 pixels (3.40 GHz CPU and 16 GB RAM)
10-15 iterations to converge
Results and code are available onhttp://eng.ucmerced.edu/people/zhu/
30 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
Input blurred image
31 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
[Gupta et al. ECCV10]
32 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
[Hirsch et al. ICCV11]
33 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
Our method
34 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
35 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
Input blurred image
36 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
[Harmeling et al. NIPS10]
37 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
[Hirsch et al. ICCV11]
38 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
Our method
39 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Experimental Results
40 / 41
Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions
Concluding Remarks
Propose a algorithm to remove non-uniform blur using one image
Initialize camera pose effectively using backprojection
Estimate pose weights efficiently by restricting solution spacewith the active set
Perform favorably against the state-of-the-art methods
41 / 41