Learning Structure-And-Motion-Aware Rolling Shutter CorrectionBingbing Zhuang1, Quoc-Huy Tran2, Pan Ji2, Loong Fah Cheong1, Manmohan Chandraker2,3
1National University of Singapore, 2NEC Labs, 3UC San Diego
Problem: Rolling shutter (RS) cameras are very popular due to its
low-cost advantage. However, they adopt a sequential exposure
mode, causing image distortion when captured during camera
motion.
1. Introduction
2. Contributions
3. Critical configuration: RS two-view SfM in pure translation
4. Structure-and-motion-aware rolling shutter correction
5. ExperimentsProposition. RS two-view geometry for pure translational camera motion is degenerate.
➢ one cannot tell if the two images are captured with a RS or GS camera based on 2D
correspondences only.
➢Even RS is known a priori, there are infinitely many solutions for the per-scanline camera position
and depth.
𝑐1
𝑐2
𝑠1
𝑠2𝑠2
𝑒
𝑒
Confounding of Depth & Translation
𝒔2 − 𝒆 =𝑍
𝑍 − 𝑇𝑖𝑗(𝒔1 − 𝒆)
See paper for a proof!Radiating Pattern
Overview Network architecture
Image resizing to get training data with same size? No, image cropping.
wx rotation vertical resizing
Training data generation
Sequential exposure mode
3D point
Camera motion between scanlines
Our correction
Rolling shutter distortion
❑ RS two-view SfM in pure translation is degenerate.
❑ A single-view structure-and-motion-aware RS correction
method.
Difficulties: removing such distortion exactly needs
recovering both the intra-frame camera motion and 3D
structure.
• Complexity of RS geometry
Two-view approach → [Dai et al. CVPR17], [Zhuang et
al. ICCV17]
• Degeneracy of RS geometry
Baseline:
(1) MH, [Purkait et al. ICCV17]
(2) 2DCNN, [Rengarajan et al. CVPR17]
Input MH 2DCNN Ours
Bas-relief like ambiguity
Result: Synthetic KITTI
Result: Real data
Contributions:
❑ A geometrically meaningful way to synthesize large-scale
training data
❑ Identify a geometric ambiguity that arises for training.
[Albl et al. ECCV16], [Ait-Aider et al. ICCV09]
— Data-driven with CNN to overcome the geometrical complexity
and degeneracy.
GS image
RS image
RS depth
GS depth
Inp
ut
Ou
rsM
H2
DC
NN
Cropping
VS
Resizing
6-DoF motion
Pure rotation
Example image
Input
Ours
MH
2DCNN
SfM
