influence of signal-to-noise ratio and point spread function on limits of super-resolution
DESCRIPTION
T.Q. Pham, L.J. van Vliet, and K. Schutte, SPIE vol. 5672 Image Processing: Algorithms and Systems IV Orlando, FL, 2005TRANSCRIPT
Influence of Signal-to-Noise Ratio and Point Spread Function on
Limits of Super-Resolution
Tuan Pham
Quantitative Imaging GroupDelft University of Technology
The Netherlands
Conf. 5672: Image ProcessingAlgorithms and Systems IV
© 2004 Tuan Pham 2
Super-Resolution: an example
4x super-resolution128x128x100 infra-red sequence
© 2004 Tuan Pham 3
Super-Resolution: an example
4x super-resolutionLow resolution
© 2004 Tuan Pham 4
Overview and Goal
Positioninglimit
SNRlimit
Resolvinglimit
Limits of Super-ResolutionGOAL: Derive the given system inputs
No. of inputs SNR PSF
System inputs
© 2004 Tuan Pham 5
Limit of registration
2nσ
• Cramer-Rao Lower Bound for 2D shift: I2(x, y) = I1(x+vx, y+vy) :
2x x y
S S2 2
x y yS S
I I I1( )
I I Inσ
⎡ ⎤⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦
∑ ∑∑ ∑
F v
where , is noise variance, and F is the Fisher Information Matrix:
1 2 211
1 2 222
var( ) I ( )
var( ) I ( )
x n yS
y n xS
v Det
v Det
σ
σ
−
−
≥ =
≥ =
∑
∑
F F
F F
• Optimal registration is achievable by iterative optimization
• CRLB also exists for more complicated motion models:- 2D projective - optic flow
/ , /x yI I x I I y= ∂ ∂ = ∂ ∂
© 2004 Tuan Pham 6
Noise of HR image after fusion
• Total noise = Intensity noise + Noise due to registration error
222 2 2
n I regINμσ σ σ= + ∇
I re gIx
σ σ∂=
∂
x
I
σreg
σI
position error distribution
localsignal
Intensity error distribution
Blurred & mis-registered5x5 box blur, pixel
Noise due to mis-registration0.2regσ = mis-registration → noise
: zoom factor
: # of LR images
: gradient energy
μN
2I∇
© 2004 Tuan Pham 7
• After fusion, the High-Resolution image is still blurry due to:– Sensor integration blur (severe if high fill-factor)– Optical blur (severe if high sampling factor)
The need for deconvolution
On-chip microlens of Sony Super HAD CCD
© 2004 Tuan Pham 8
• Spectrum is cut off beyond fc due to optics → data forever lost
The necessity of aliasing
0 0.5 1 1.5−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
frequency in unit of sampling frequency (f/fs)
freq
uenc
y sp
ectr
a / t
rans
fer
func
tions
OTF (sampling factor = 0.25)STF (fill factor = 1)Original scene spectrumBand−limited spectrumAliased image spectrum
0 0.5 1 1.5 2−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
frequency in unit of sampling frequency (f/fs)
freq
uenc
y sp
ectr
a / t
rans
fer
func
tions
OTF (sampling factor = 1)STF (fill factor = 1)Original scene spectrumBand−limited spectrumSampled image spectrum
Aliasing due to under-sampling (fs < 2fc)
No aliasing at critical sampling (fs = 2fc)
© 2004 Tuan Pham 9
Limit of deconvolution
• Blur = attenuation of HF spectrum
• Deconvolution = amplify HF spectrum:– noise is also amplified → limit the deconvolution
• Deconvolution can only recover:– Spectrum whose signal power > noise power
fusion result after deconvolution simulated at resolution = 0.44
resolution factor = 0.44
recoverable
Not recoverable
PS>PN
© 2004 Tuan Pham 10
SR reconstruction experiment
64x64 LR inputsampling=1/4, fill = 100%
4xHR after fusionBSNR = 20 dB
4xSR after deconvolutionSR factor = 3.4
• Aim: show that the attainable SR factor agrees with the prediction• Experiment:
– Inputs: sufficient shifted LR images of the Pentagon– Output: SR image and a measure of SR factor from edge width
© 2004 Tuan Pham 11
SR reconstruction experiment
64x64 LR inputsampling=1/4, fill = 100%
4xHR after fusionBSNR = 20 dB
4xSR after deconvolutionSR factor = 3.4
• Aim: show that the attainable SR factor agrees with the prediction• Experiment:
– Inputs: sufficient shifted LR images of the Pentagon– Output: SR image and a measure of SR factor from edge width
© 2004 Tuan Pham 12
SR factor at BSNR=20dB
0
1
2
0
0.5
1
0
2
4
6
sampling factor (f
s/2f
c) fill factor
SR
fact
or
0.6
1.0
1.9
3.4
3.2
0
1
2
0
0.5
1
0
2
4
6
sampling factor (f
s/2f
c) fill factor
SR
lim
it
0.6
1.0
1.7
2.5
3.0
Measured SR factor Predicted SR factor
• Consistent results between prediction and measurement:– Attainable SR factor depends mainly on sampling factor (i.e. level of aliasing)
© 2004 Tuan Pham 13
Summary
• Limit of Super-Resolution depends on:– input Signal-to-Noise Ratio
– System’s Point Spread Function and how well it can be estimated
• Procedure for estimating SR factor directly from inputs:– Measure noise variance from LR images
– Derive registration error
– Determine SR factor from the Power Spectrum Density (PS > PN)
2Iσ
2regσ