radiometric self calibration
DESCRIPTION
Radiometric Self Calibration. Tomoo Mitsunaga Shree K. Nayar Hashimoto Signal Processing Lab. Dept. of Computer Science Sony Corporation Columbia University. CVPR Conference Ft. Collins, Colorado June 1999. - PowerPoint PPT PresentationTRANSCRIPT
Radiometric Self CalibrationRadiometric Self Calibration
Tomoo Mitsunaga Shree K. NayarTomoo Mitsunaga Shree K. NayarHashimoto Signal Processing Lab. Hashimoto Signal Processing Lab. Dept. of Computer Dept. of Computer ScienceScience
Sony Corporation Columbia UniversitySony Corporation Columbia University
CVPR ConferenceFt. Collins, Colorado
June 1999
June/1999 CVPR99 2
Problem StatementProblem Statement
How well does the image represent the real world?How well does the image represent the real world?
Image Image MM11
(High exposure)(High exposure)Image Image MM22
(Low exposure)(Low exposure)
• Non-linear responseNon-linear response )( 12 MM )( 12 MM
• Limited dynamic rangeLimited dynamic rangeUsual imaging systems have :Usual imaging systems have :
June/1999 CVPR99 3
Scene Radiance and Image IrradianceScene Radiance and Image Irradiance
Image irradianceImage irradiance : :
RadianceRadiance IrradianceIrradiance
24
2
2cos
4
dkL
h
dLE 2
42
2cos
4
dkL
h
dLE
tEI tEI
td
e
2
2 t
de
2
2
Ideal camera responseIdeal camera response : :
ExposureExposure : :
LLEE
Aperture areaAperture area
ekLI ekLI
June/1999 CVPR99 4
Scene Radiance and Measured Brightness Scene Radiance and Measured Brightness
ImageImageFormationFormation
ImageImageExposureExposure CCDCCD
CameraCameraElectronicsElectronics
DigitizationDigitization
ImageImageFormationFormation
ImageImageExposureExposure FilmFilm
FilmFilmDevelopmentDevelopment ScanningScanning
VideoVideo
PhotoPhoto
SceneSceneradianceradianceLL
ScaledScaledradianceradianceII
MeasuredMeasuredbrightnessbrightness
MM)(
)(1 IfM
MfI
)(
)(1 IfM
MfI
linearlinear
f (M) : The radiometric response function
June/1999 CVPR99 5
Calibration with Reference ObjectsCalibration with Reference Objects
The scene must be controlledThe scene must be controlled• The reflectance of the objects must be knownThe reflectance of the objects must be known• The illumination must be controlledThe illumination must be controlled
June/1999 CVPR99 6
Calibration without Reference ObjectsCalibration without Reference Objects• Differently exposed images from an arbitrary sceneDifferently exposed images from an arbitrary scene• Recover the response function from the imagesRecover the response function from the images• Calibrate the images with the response functionCalibrate the images with the response function
High dynamic range radiance imageHigh dynamic range radiance image
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Response functionResponse functionInput ImagesInput Images
June/1999 CVPR99 7
Previous WorksPrevious Works• Mann and Picard (95) :Mann and Picard (95) :
– Take two images with known exposure ratio Take two images with known exposure ratio RR
– Restrictive model for Restrictive model for ff : :
– Find parameters Find parameters by regression by regression
• Debevec and Malik (97) :Debevec and Malik (97) :– General model for General model for ff : only smoothness constraint : only smoothness constraint
– Take several (say, 10) high quality imagesTake several (say, 10) high quality images
– At precisely measured exposures (shutter speed)At precisely measured exposures (shutter speed)
IM IM
June/1999 CVPR99 8
Obtaining Exposure InformationObtaining Exposure Information
We have only rough estimatesWe have only rough estimates• Mechanical errorMechanical error• Reading error (ex. F-stop number)Reading error (ex. F-stop number)
June/1999 CVPR99 9
Radiometric Self-CalibrationRadiometric Self-Calibration
Works with roughly estimated exposuresWorks with roughly estimated exposures• Inputs :Inputs :
– Differently exposed imagesDifferently exposed images
– Rough estimates of exposure valuesRough estimates of exposure values– ex. F-stop readingex. F-stop reading
• Outputs :Outputs :– Estimated response functionEstimated response function
– Corrected exposure valuesCorrected exposure values
June/1999 CVPR99 10
A Flexible Parametric ModelA Flexible Parametric Model
nN
nnMcMfI
0
)( nN
nnMcMfI
0
)(
High order polynomial model :High order polynomial model :
Parameters to be recovered :Parameters to be recovered :• Coefficients Coefficients ccnn
• Order Order NNff((MM) of some ) of some
popular imaging productspopular imaging products
videovideo
posiposi
neganega
M
f (M)
June/1999 CVPR99 11
Response Function and Exposure RatioResponse Function and Exposure Ratio
Images: Images: qq = 1,2,…. = 1,2,….QQ , Pixels: , Pixels: pp = 1, 2, ….. = 1, 2, …..PP
Exposure ratio:Exposure ratio:)(
)(,
1,
,
1,
,
11,
11,
qp
qp
qp
qp
qpp
qppqq
q
qqq Mf
Mf
I
I
ekL
ekLR
e
eR
)(
)(,
1,
,
1,
,
11,
11,
qp
qp
qp
qp
qpp
qppqq
q
qqq Mf
Mf
I
I
ekL
ekLR
e
eR
1,
0 1,
0 ,
qqN
n
nqpn
N
n
nqpn R
Mc
Mc1,
0 1,
0 ,
qqN
n
nqpn
N
n
nqpn R
Mc
McUsing polynomial model :Using polynomial model :
21
1 1 0 01,1,,
Q
q
P
p
N
n
N
n
nqpnqq
nqpn McRMc
21
1 1 0 01,1,,
Q
q
P
p
N
n
N
n
nqpnqq
nqpn McRMcObjective function : Objective function :
Thus, we obtain ...Thus, we obtain ...
June/1999 CVPR99 12
An Iterative Scheme for OptimizationAn Iterative Scheme for Optimization
OptimizeOptimizefor for ff
OptimizeOptimizefor for ff
OptimizeOptimizefor for RRq,qq,q+1+1
OptimizeOptimizefor for RRq,qq,q+1+1
f f ((ii))
RRq,qq,q+1+1((ii))
Rough estimates Rough estimates RRq,qq,q+1+1
(0)(0)
Rough estimates Rough estimates RRq,qq,q+1+1
(0)(0)
Optimized Optimized f f andand R Rq,qq,q+1+1Optimized Optimized f f andand R Rq,qq,q+1+1
Mff ii )1()( Mff ii )1()(
June/1999 CVPR99 13
Evaluation : Noisy Synthetic ImagesEvaluation : Noisy Synthetic Images
Solid : Computed response functionSolid : Computed response functionDotsDots : Actual response function : Actual response function
MM
f f ((MM))
June/1999 CVPR99 14
Evaluation : Noisy Synthetic Images (cont’d)Evaluation : Noisy Synthetic Images (cont’d)
PercentagePercentage Error inError in
ComputedComputed ResponseResponseFunctionFunction
Trial NumberTrial Number
Maximum Error : 2.7 %Maximum Error : 2.7 %
June/1999 CVPR99 15
• Calibrating by the response functionCalibrating by the response function• Normalizing by corrected exposure valuesNormalizing by corrected exposure values• Averaging with SNR-based weighting Averaging with SNR-based weighting
Computing a High Dynamic Range ImageComputing a High Dynamic Range Image
)(Mf )(Mf 1,pI 1,pI
2,pI 2,pI
QpI ,QpI ,
1,pM 1,pM
2,pM 2,pM
QpM ,QpM ,
1~1e1~1e
Qe~1
Qe~1
2~1e 2~1e
QpI ,
~QpI ,
~
2,
~pI 2,
~pI
Q
q qp
Q
q qpqp
Mw
IMw
1 ,
1 ,,
)(
~)(
Q
q qp
Q
q qpqp
Mw
IMw
1 ,
1 ,,
)(
~)( pI pI
1,
~pI 1,
~pI
)(Mf )(Mf
)(Mf )(Mf
e~e~
)(Mw )(Mw
)(Mf )(Mf
June/1999 CVPR99 16
Results : Low Library (video) Results : Low Library (video)
Captured imagesCaptured images
Calibration chartCalibration chart
Computed responseComputed response functionfunction
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
II
MM
June/1999 CVPR99 17
Results : Low Library (video)Results : Low Library (video)
Captured imagesCaptured images
Computed radiance imageComputed radiance image
June/1999 CVPR99 18
Results : Adobe Room (photograph)Results : Adobe Room (photograph)
II
Captured imagesCaptured images
Computed response functionComputed response function Computed radiance imageComputed radiance image
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1MM
June/1999 CVPR99 19
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Results : Taos Clay Oven (photograph)Results : Taos Clay Oven (photograph)
Captured imagesCaptured images
Computed response functionComputed response function Computed radiance imageComputed radiance image
II
MM
June/1999 CVPR99 20
ConclusionsConclusions
A Practical Radiometric Self-calibration MethodA Practical Radiometric Self-calibration Method• Works withWorks with
– Arbitrary still sceneArbitrary still scene
– Rough estimates of exposureRough estimates of exposure
• Recovers Recovers – Response function of the imaging systemResponse function of the imaging system
– High dynamic range image of the sceneHigh dynamic range image of the scene
Software and DemoSoftware and Demo
http://www.cs.columbia.edu/CAVE/http://www.cs.columbia.edu/CAVE/