some problems... lens distortion uncalibrated structure and motion recovery assumes pinhole cameras...
TRANSCRIPT
Some problems...
Lens distortion
Uncalibrated structure and motion recovery assumes pinhole cameras
Real cameras have real lenses
How can we correct distortion, when original calibration is inaccessible?
1. Even small amounts of lens distortion can upset uncalibrated structure from motion
2. A single distortion parameter is enough for mapping and SFX accuracy
3. Including the parameter in the multiview relations changes the 8-point algorithm from
4. You can solve such “Polynomial Eigenvalue Problems”
5. This is as stable as computation of the Fundamental matrix, so you can use it all the time.
Even small amounts of lens
distortion can upset uncalibrated structure from motion—
A map-building problem
(a) Input movie – relatively low distortion(b) Plan view: red is structure, blue is motion
(a) (b)
Effects of Distortion
(a) Input movie – relatively low distortion(b) Recovered plan view, uncorrected distortion
(a) (c)
Does distortion do that?
Distortion of image plane is conflated with focal lengthwhen the camera rotates
[From: Tordoff & Murray, ICPR 2000]
Distortion correction in man-made scenes
Distortion correction in natural scenes
In natural images, distortion introduces correlations in frequency domain
Choose distortion parameters to minimize correlations in bispectrum
Less effective on man-made scenes....
[Farid and Popescu, ICCV 2001]
Distortion correction in multiple images
Multiple views, static scene• Use motion and scene rigidity [Zhang, Stein,
Sawhney, McLauchlan, ...]Advantages:• Applies to man-made or natural scenesDisadvantages:• Iterative solutions|require initial estimates
A single distortion parameter
is accurate enough for map-building and cinema post production—
Modelling lens distortion
x: xeroxednoxious
experimental artifax
p: perfect pinhole
perspective pure
xp p
x
Known Unknown
Single-parameter models
Single-parameter modelling power
Single-parameter model
Radial term onlyAssumes distortion
centre is at centre of image
A one-parameter model suffices
A direct solution for
Look at division model again
>> help polyeig
POLYEIG Polynomial eigenvalue problem.
[X,E] = POLYEIG(A0,A1,..,Ap) solves the polynomial eigenvalue problem
of degree p:
(A0 + lambda*A1 + ... + lambda^p*Ap)*x = 0.
The input is [etc etc...]
>>
A quick matlab session
Algorithm
T his is as stable as
computation of the fundamental matrix, so you can use it all the time—
Performance: Synthetic data
0 0.2 0.4 0.6 0.8 1-0.4
-0.3
-0.2
-0.1
0
Noise (pixels)
Com
pu
ted
• Stable – small errorbars• Biased – not centred on true value
Analogy: Linear ellipse fitting
True
Data
Fitted: 10 trials
Best-fit line
Performance: Synthetic data
Performance: Real sequences
-0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.30
10
20
30
40
50
• 250 pairs• Low distortion• Linear estimate used to initialize nonlinear• Number of inliers changes by [-25..49]
Conclusions
Environment matting
In: magnifying glass moving over background
Out: same magnifying glass, new background
Environment matting: why?
• Learn– light-transport
properties of complex optical elements
• Previously– Ray tracing
geometric models– Calibrated
acquisition
• Here– Acquisition in situ
Image formation model
• Purely 2D-2D– Optical element performs weighted sum of (image of)
background at each pixel
– suffices for many interesting objects
– separate receptive field for each output pixel
– Environment matte is collection of all receptive fields—yes, it’s huge.
Image formation model
Step 1: Computing backgroundInput:
Mosaic:
Clean plate:Point tracks:
Step 2: Computing w...Input:
Computing w(x,y,u,v) at a single (x,y)
Assume wi independent
Composite over new background
A more subtle exampleInput: Two images
Moving cameraPlanar background
- Need priors
Window example
Discussion
• Works well for non-translucent elements– need to develop for diffuse
• Combination assumes independence– ok for large movements: “an edge crosses
the pixel”
• Need to develop for general backgrounds
A Clustering Problem
• Watch a movie, recover the cast list– Run face detector on every frame– Cluster faces
• Problems– Face detector unreliable– Large lighting changes– Changes in expression– Clustering is difficult
A sample sequence
Detected faces
Face positions
Lighting correction
Clustering: pairwise distances
Raw distance
Clustering: pairwise distances
Transform-invariant distance
Clusters: “tangent distance”
Clusters: Bayesian tangent distance
Conclusions
• Extend to feature selection, texton clustering etc
• Remove face detector