precise multi-frame motion estimation and its...
Post on 21-Feb-2020
15 Views
Preview:
TRANSCRIPT
Milanfar et al. EE Dept, UCSC
Precise Multi-Frame Motion Estimation and Its Applications
Peyman MilanfarEE Department
University of California, Santa Cruzmilanfar@ee.ucsc.edu
Joint work with Dirk Robinson, Michael Elad, Sina Farsiu
Milanfar et al. EE Dept, UCSC
Motivating Application: Resolution Enhancement from Video
The Idea: “Diversity” + Aliasing• Given multiple low-resolution movingmoving images of a
scene (a video), generate a high resolution image (or video).
Milanfar et al. EE Dept, UCSC
Practical Motivation: (Real Video Enhancement)
MotionEstimation
ImageReconstruction
Resolution enhancement x4 from video frames captured by a commercial webcam(3COM Model No.3719)
Milanfar et al. EE Dept, UCSC
Problem: Given a pair of frames,
we want to estimate the translation
Typical Assumptions:–Sampled on a finite grid.–Sampled above Nyquist rate
– Will discuss aliased case later.–Additive white Gaussian Noise
Translational Motion Estimation
),(),(),(),(),(),(
2122211212
21121211
xxevxvxfxxfxxexxfxxf+−−=
+=
=
2
1
vv
v
Milanfar et al. EE Dept, UCSC
Translational Motion Estimation
Optimum Statistical Estimator: Max. Likelihood
– Correlation Methods• Direct Maximization• Phase-Correlation
– Nonlinear Least Squares• Gradient-Based algorithms• Pyramid-Gradient-Based algorithms• Direct Minimization
– Improving to subpixel accuracy• Fits a quadratic about the peak of the
correlation surface.• Gauss-Newton methods, iterated improvement • Iterating over scale: pyramid-based methods
( )∑ −−−2121 ,
221222111
,),(),(min
xxvvxxfvxvxf
∑ −−2121 ,
21222111,
),(),(maxxxvv
xxfvxvxf
Milanfar et al. EE Dept, UCSC
Performance Limits in Image Registration
How close to the limit are typical methods?Image used
At 5 dB
BiasIndependent of underlylingvelocity vector
Milanfar et al. EE Dept, UCSC
Effect of Aliasing
How does aliasing affect the ability to estimate translation between sets of images?
Little aliasing Lot of aliasing
Note “false” motions.
Milanfar et al. EE Dept, UCSC
Performance of Aliased Image Registration
Observations:
• Very little work addressing registration of aliased image.
• Performance bound depends on the motion parameters (not true for non-aliased registration)
• Traditional algorithms designed for non-aliased scenario will fail.
Milanfar et al. EE Dept, UCSC
• Consider a sequence of noisy, translating images over time.
Data and Formulation
{ }kf
noiseSample += )],(*),,([ yxhtyxf kkf
error Translate += −− ),( ,11 kkkk vff
Nf
1f2,1v
2f3,2v
• Image formation model:
Point-spread function
Frame-to-frame motion vectors
Aliasing
Milanfar et al. EE Dept, UCSC
error Translate += −− ),( ,11 kkkk vff
noiseSample += )],(*),,([ yxhtyxf kkf
• Motion Problem: Given the frames, estimate vectors
– Implicit problem: Estimate underlying high resolution image
Registration of Multiple Video Frames
Desiredunknowns
{ }kkv ,1−
Nuisance Parameter
Milanfar et al. EE Dept, UCSC
error Translate += −− ),( ,11 kkkk vff
noiseSample += )],(*),,([ yxhtyxf kkf
• Reconstruction Problem: Given the frames, estimate the high resolution image . (Superresolution)
– Implicit problem: Estimate the motion vectors
Fusion of Multiple Video Frames
NuisanceParameters
Desired unknowns
),,( tyxf
Milanfar et al. EE Dept, UCSC
Registration Information Information
“Correlation”
Reconstruction Information
vvJ - Depends on the set of motions (sampling offsets) and the amount of texture energy in the signal
- Depends only on the set of motionsffJ
( )
=−
ffTfv
fvvvkk fv
JJJJ
J },{ ,1
Fisher Information Matrix (FIM) is partitioned on the motion parameters and the high-res (alias-free) “atlas” image .
{ }kkv ,1−
f
How well can the problem be solved?
Milanfar et al. EE Dept, UCSC
CRB for Aliased Image Registration
Using Schur decomposition, the CRB for aliased image registration is:
With just a pair of aliased images, the FIM becomessingular, hence pairwise registration of aliased images is essentially impossible
{ }( ) ( ) 11,
−−−≥ Tfvfffvvvkjv JJJJCov
Registration Information
Information Loss due to uncertainty about the high resolution image.
Milanfar et al. EE Dept, UCSC
Registering Sets of Images
0 5 10 15 20 25 30 35 40 45 500.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
0.022
Number of Frames (K+1)
Reg
istr
atio
n C
RB
(pi
x/fr
ame)
M = 1M = 2M = 3M = 4M = 5+
CR Bound (per frame) for multi-frame image registration.
No aliasing
More aliasing
Milanfar et al. EE Dept, UCSC
What to do? • Almost all motion estimation algorithms
today deal with the case of only two (consecutive) frames at a time.
• These methods are far from optimal.• Proposal: Use multiple frames
simultaneously, with care!
Pairwise estimation(“Progressive”)
Fixed reference estimation(“Anchored”)
Milanfar et al. EE Dept, UCSC
Constraints on Translational Motion Vectors Across Time
Frame i Frame j Frame k
kjv ,
kiv ,
jiv ,
0,
,,
,,,
=
−=
+=
ii
kjjk
kjjiki
v
vv
vvvLinear set of constraints imply that themotion vectors live in a subspace.
Milanfar et al. EE Dept, UCSC
An Algorithm Thus Motivated
∑ −kj
p
pkjjkvv
jk ,,}{
),(ff Translate minimize
kjjk
kjjiki
vv
vvv
,,
,,,
−=
+=
to subject
Can be any penalty function
• With p=2, and linear constraints, we have a quadratic programming problem.
• Computationally simpler (but suboptimal) is to project any estimated parameters onto the constraint subspace.
• With p=1, we can have a more robust solution.
Milanfar et al. EE Dept, UCSC
Performance
0 10 20 30 40 50 60 7010
-3
10-2
10-1
100
101
SingleProjected
p=2, Constrainedp=1, Constrained
RM
SE
(Pix
els)
SNR (dB)
Milanfar et al. EE Dept, UCSC
Application to Simultaneous Demosaicingand Resolution EnhancementBayer Filtered Motion Sequence
. . .
Single-FrameDemosaicing
OLD
Image fusion Hi-resolutionDemosaicing
NEW
Milanfar et al. EE Dept, UCSC
Example with Real Data
27 Raw CFA Images
• Registering the color-filtered data is non-standard and difficult in practice.
Milanfar et al. EE Dept, UCSC
Nice Algebraic Structure
• S is closed under +• S is associative• S has an identity element : 0 • Every element of S has an inverse
{ }kjvS ,= Satisfies
Set of all pairwise motions S between frames is a GROUP
0,
,,
,,,
=
−=
+=
ii
kjjk
kjjiki
v
vv
vvv
Milanfar et al. EE Dept, UCSC
A Few Words on Affine Motion
Frame i Frame j Frame k
),( ,, kiki TM
),( ,, jiji TM ),( ,, kjkj TM
Constraints
Milanfar et al. EE Dept, UCSC
General Comments • The constraints for the affine case are nonlinear.
• But the algebraic structure persists.– Group operation is no longer simple vector addition
• Algebraic structure also for dense optical flow– Here the elements of the algebra (motions) are
defined by nonlinear transformations applied to images.
– Lie Algebra defined by the composition of operators.– If operators are differentiable, then a Lie Group.
Milanfar et al. EE Dept, UCSC
Conclusions
• Accurate motion estimation is a (very) hard problem.
• Registering a pair of aliased frames is an ill-posed problem.
• Using simultaneous image registration and reconstruction is one possible solution.
• Motion estimation with constraints (hard or soft) is another alternative.
• There are many applications.
top related