Download - Super-Resolution Dr. Yossi Rubner [email protected] Many slides from Miki Elad - Technion 1
Example - Video
53 images, ratio 1:42
40 images ratio 1:4
Example – Surveillance
3
Example – Enhance Mosaics
4
5
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution• SR in time
6
D
For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible.
D
Intuition
7
Due to our limited camera resolution, we sample using an insufficient 2D grid
2D
2D
Intuition
8
However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:
2D
2D
Intuition
9
Similarly, by shifting down we get a third image:
2D
2D
Intuition
10
And finally, by shifting down and to the right we get the fourth image:
2D
2D
Intuition
11
It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.
Intuition
12
What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?
Rotation/Scale/Disp.
13
There is no sampling theorem covering this case
Rotation/Scale/Disp.
14
15
3:1 scale-up in each axis using 9 images, with pure global translation between them
A Small Example
Further Complications
• Complicated motion– perspective, local motion, …
• Blur– sampling is not a point operation– Spatially variant blur– Temporally variant blur
• Noise
• Changes in the scene
16
Super-Resolution - Agenda
• The basic idea
• Image formation process• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution• SR in time
17
Image Formation
Scene Noise
HR LR
Can we write these steps as linear operators?
Geometrictransformation
kF
OpticalBlur
kH
Sampling
kD
HRLR kkk FHD 18
Geometric Transformation
• Any appropriate motion model• Every frame has different transformation • Usually found by a separate registration algorithm
Scene Geometrictransformation
kF
19
Geometric Transformation
Can be modeled as a linear operation XkF
kF
X
=
kF XXkF
20
Optical Blur
• Due to the lens PSF and pixel integration• Usually
Geometrictransformation
OpticalBlur
kH
HH k
21
H
PSF PIXEL H* =
22
Optical Blur
Can be modeled as a linear operation XH
H
X
=
XHXH
23
Sampling
• Pixel operation consists of area integration followed by decimation• D is the decimation only• Usually
Optical Blur Sampling
kD
DD k
24
Decimation
Can be modeled as a linear operation XD
D
X
=
X
01
01...
01
01
01 o
o
D
1
XD
25
Super-Resolution - Agenda
• The basic idea• Image formation process
• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution• SR in time
26
Super-Resolution - Model
N
k
nkkkkkk VVXY1
2,0~ ,
NFHD
X
High-Resolution
ImageH
H
Blur
1
N
F =I1
FN
Geometric warp
D
D1
N
Decimation
V1
VN
Additive Noise
Y1
YN
Low-ResolutionExposures
27
Simplified Model
N
k
nkkkk VVXY1
2,0~ ,
NDHF
X
High-Resolution
ImageH
H
Blur
F =I1
FN
Geometric warp
D
D
Decimation
V1
VN
Additive Noise
Y1
YN
Low-ResolutionExposures
28
The Super-Resolution Problem
• Given
Yk – The measured images (noisy, blurry, down-sampled ..)
H – The blur can be extracted from the camera characteristics
D – The decimation is dictated by the required resolution ratio
Fk – The warp can be estimated using motion estimation
n – The noise can be extracted from the camera / image
• RecoverX – HR image
2,0~ , nkkkk VVXY NDHF
29
VX
V
V
V
X
Y
Y
Y
Y
NNNNN
G
FHD
FHD
FHD
2
1
222
111
2
1
The Model as One Equation
1 size of ,
size of
1 size of
22
222
2
MrVX
MrNM
NMY
G
r = resolution factorMXM = size of the framesN = number of frames
r = resolution factor = 4MXM = size of the frames = 1000X1000N = number of frames = 10
=[10M×1]=[10M×16M]=[16M×1] Linear algebra notation is
intended only to develop algorithm 30
SR - Solutions• Maximum Likelihood (ML):
N
kkk
XYXX
1
2minarg DHF
Smoothness constraintregularization
Often ill posed problem!
XAYXXN
kkk
X
1
2 minarg DHF
• Maximum Aposteriori Probability (MAP)
31
ML Reconstruction (LS)
N
kkkML YXX
1
22 DHFMinimize:
0ˆ21
2
N
kkk
TTTk
ML YXX
XDHFDHF
Thus, require:
k
N
k
TTTk
N
kk
TTTk YX
11
ˆ DHFDHFDHF
A B
BA X̂32
LS - Iterative Solution
• Steepest descent
N
kknk
TTTknn YXXX
11
ˆˆˆ DHFDHF
Simulated error
Back projection
All the above operations can be interpreted as operations performed on images.
There is no actual need to use the Matrix-Vector notations as shown here.
33
LS - Iterative Solution
• Steepest descent
N
kknk
TTTknn YXXX
11
ˆˆˆ DHFDHF
nX̂
1ˆ
nX
geometrywrap
convolvewith H
downsample
upsample
convolvewith HT
inversegeometry
wrap
kY
-
-
kF H DTD TH T
kF
For k=1..N
34
Example
Simulated example from Farisu at al. IEEE trans. On Image Processing, 04
HR image Least squaresLR + noiseX4
35
Robust Reconstruction
• Cases of measurements outlier:– Some of the images are irrelevant – Error in motion estimation– Error in the blur function– General model mismatch
36
Robust Reconstruction
N
kkk YXX
1
2 1
DHFMinimize:
N
kknk
TTTknn YXXX
11
ˆsignˆˆ DHFDHF
37
Robust Reconstruction
• Steepest descent
N
kknk
TTTknn YXXX
11
ˆsignˆˆ DHFDHF
sign
For k=1..N
nX̂
1ˆ
nX
geometrywrap
convolvewith H
downsample
upsample
convolvewith HT
inversegeometry
wrap
kY
-
-
kF H DTD TH T
kF
38
Example - Outliers
Simulated example from Farisu at al. IEEE trans. On Image Processing, 04
HR image LR + noiseX4
Least squares
Robust Reconstruction39
20 images, ratio 1:4
L2 norm based
Example – Registration Error
L1 norm based
40
MAP Reconstruction
• Regularization term:
– Tikhonov cost function
– Total variation
– Bilateral filter
XAYXXN
kkkMAP
1
22 DHF
2XXAT
1
XXATV
P
Pl
P
Pm
my
lx
mlB XSSXXA
1 41
Robust Estimation + Regularization
P
Pl
P
Pm
my
lx
mlN
kkk XSSXYXX
11
1
2 DHF Minimize:
P
Pl
P
Pmn
my
lxn
my
lx
ml
N
kknk
TTTknn
XSSXSSI
YXXX
ˆˆsign
ˆsignˆˆ1
1
DHFDHF
42
Robust Estimation + Regularization
1ˆ
nX
geometrywrap
convolvewith H
downsample
upsample
convolvewith HT
inversegeometry
wrap
kY
sign
-
P
Pl
P
Pmn
my
lxn
my
lx
mlN
kknk
TTTknn XSSXSSIYXXX ˆˆsignˆsignˆˆ
11 DHFDHF
nX̂
horizontalshift l
verticalshift m
horizontalshift -l
verticalshift -m
sign
-
-
From Farisu at al. IEEE trans. On Image Processing, 04
- lm
For k=1..N
For l,m=-P..P
43
Example
• 8 frames
• Resolution factor of 4
From Farisu at al. IEEE trans. On Image Processing, 0444
Example
Images from Vigilant Ltd.45
Handling Color in SR
XAYXXN
kkkMAP
1
22 DHF
Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr.
Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images).
In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.
46
20 images, ratio 1:4
SR for Full Color
47
20 images, ratio 1:4
Mosaiced input
Mosaicing and then SR Combined treatment
SR+Demosaicing
48
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution
• Special cases and related problems• Limitations of Super-Resolution• SR in time
49
Special Case – Translational Motion
• In this case H and F commute:
• SR is decomposed into 2 steps1. Find blur HR image from LR images non-iterative2. Deconvolve the result using H iterative
XZVZ
VX
VXY
kk
kk
kkk
HDF
HDF
DHF
TTk
Tk
Tkk HFFHHFHF
50
IntuitionXZVZY kkk HDF
X PSF*X Z=PIXEL*PSF*X
• Using the samples can, at most, reconstruct Z• To recover X, need to deconvolve Z
51
Step I – Find Blurred HR
• L2 For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 01]
• L1 For all frames, copy registered pixels to HR grid and use median [Farisu, 04]
N
kkkML YZZ
1
22 DF Minimize:
52
Solution for L2
N
kkkML YZZ
1
22 DF Minimize:
N
kk
TTk
N
kk
TTk
YP1
1
DF
DFDFR
PZˆR
0
2
Z
ZMLThus, require:
Sum of HR grid
Diagonal, number Of occurrences
per HR grid
53
Step II - Deblur
XAZXX 1
2 H Minimize:
n
n
knT
nn XAX
ZXXX ˆˆ
ˆsignˆˆ1 HH
54
Example
From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated.
64X64 LR 256X256Before deblur
256X256After deblur
55
Related Problems
• Denoising (multiple frames)
• Denoising (single frame)
• Deblurring
• Interpolation – “single-image super-resolution”
2,0~ nVVXY NDH ,
2,0~ nkkk VVXY N ,
2,0~ nVVXY NH ,
2,0~ nVVXY N ,
56
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems
• Limitations of Super-Resolution• SR in time
57
Limiting Factors
• Main factors– SNR– PSF (optical+pixel)– Number of inputs – Pixel size (sampling rate)– Fill factor– Image content
58
Limiting Factors
Registration DeblurringFusion
LR1
…
LRN
HR
SNR # of inputs PSF
Pixel sizeFill factor
Image content
59
SR Limits Analysis
• Noise– Registration noise– Fusion noise
• SR factor– Point-Spread-Function (PSF)
• Optical Transfer Function (OTF)
– Sensor pixel size• Sensor Transfer Function (STF)
60
Registration Noise
• Using Cramer Rao Lower Bound (CRLB)– Lower bounds for shift estimation:
• Better registration accuracy by:– Less noise– Higher derivatives in image– Bigger registration area– Narrow PSF
222
2222
222
22
222
22
var
var
yxyx
yxnreg
yxyx
xn
yxyx
yn
v
u
IIII
II
IIII
I
IIII
I
61
Fusion Noise
• r = super-resolution factor
• N = number of images
22
2nfusion N
r
62
Registration + Fusion Noise
• If N∞ then– Fusion error vanishes– Registration error is equivalent to Gaussian
blur
2
22
222
222
nn
yxyx
yxtotal N
r
IIII
II
63
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution
• SR in time
Work and slides by Michal Irani & Yaron Caspi (ECCV’02)
64
“Classical” Image Super-Resolution
Low-resolution images:
High-resolution image:
Scene:
65
time
time
time
Space-Time Super-Resolution
Super-resolution in space and in time.
time
High space-time resolution sequence: time
Low-resolution imagesvideo sequences:
66
What is Super-Resolution in Time?
Observing events “faster” than frame-rate.
• Handles:
(1) Motion aliasing (2) Motion blur
• Application areas: - sports scenes- scientific imaging- etc...
67
(1) Motion AliasingThe “Wagon wheel” effect: Slow-motion:
time
Continuous signal
time
Sub-sampled in time
time
“Slow motion”68
(2) Motion Blur
69
lnS
lS1
Sh(xh,yh,th)
Space-Time Super-Resolution
x
y t
y
x
t
Blur kernel:
PSF
Exposure time
T
71
Input 1 Input 2
Input 3 Input 4
Example: Motion-Aliasing
25 [frames/sec]72
Input sequence in slow-motion (x3):
75 [frames/sec]
Super-resolutionSuper-resolution in time (x3):
75 [frames/sec]
Example: Motion-Aliasing
73
Output trajectory:
Without estimating motion of the ball!
Output sequence:
(x15 frame-rate)
Deblurring:
Input:
Output:
3 out of 18 low-resolution input sequences (frame overlays; trajectories):
74
Frames at collision:
4 input sequences:
Output frame at collision:
Video 1
Video 3
Video 2
Video 4
Example: Motion-Blur (real)
75