a measure for accuracy disparity maps evaluation
DESCRIPTION
An error measure for evaluating disparity maps is presented. It offers advantages over conventional ground-truth based error measures.Cabezas, I.; Padilla, V. & Trujillo, M. (2011), A Measure for Accuracy Disparity Maps Evaluation., in César San Martín & Sang-Woon Kim, ed., 'CIARP' , Springer, , pp. 223-231 .TRANSCRIPT
A Measure for Accuracy
Disparity Maps Evaluation
Ivan Cabezas, Victor Padilla and Maria Trujillo
November 16th 2011
16th Iberoamerican Congress on Pattern Recognition, CIARP 2011, Pucón, Chile
Slide 2
Multimedia and Vision Laboratory
MMV is a research group of the Universidad del Valle in Colombia
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Slide 3
Content
Multimedia and Vision Laboratory
Stereo Vision
Disparity
Disparity Maps Evaluation
Error Measures
Problem Statement
Illustration of the BMP
The Sigma Z Error Measure
Illustration of the SZE
Impact on Evaluation Results
Final Remarks
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Slide 4
Stereo Vision
The stereo vision problem is to recover the 3D structure of a scene using
two or more images
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
3D Model
Stereo Images
Disparity Map
Left Right
Correspondence Algorithm
Reconstruction Algorithm
Camera System
3D World
2D Images
Inverse Problem
Optics Problem
Yang Q. et al., Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling, IEEE PAMI 2009
Slide 5
Disparity
Disparity is the distance between corresponding points
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Trucco, E. and Verri A., Introductory Techniques for 3D Computer Vision, Prentice Hall 1998
Slide 6
Disparity Maps Evaluation
The evaluation of stereo correspondence algorithms is based on obtained
disparity maps
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Scharstein, D. and Szeliski, R., A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, IJCV 2002
Scharstein, D. and Szeliski, R., High-accuracy Stereo Depth Maps using Structured Light, CVPR 2003
Mei, X., et al., On Building an Accurate Stereo Matching System on Graphics Hardware, GPUCV 2011
Cabezas, I. and Trujillo M., A Non-Linear Quantitative Evaluation Approach for Disparity Estimation, VISAPP 2011
Ground-truth Map Error Criteria
nonocc all disc
Slide 7
Error Measures
The selection of an error measure has an impact on evaluation results
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Van der Mark, W., Gavrila, D., Real-time Dense Stereo for Intelligent Vehicles IEEE Trans. on Intelligent Transportation Systems, 2006
Slide 8
Problem Statement
Conventional error measures have different drawbacks:
The use of the mean, which is sensitive to extreme values
Ignoring the inverse relation between depth and disparity
• Disparity errors of the same magnitude may have different impacts on
depth reconstruction
• Small disparities are difficult to estimate and sensitive to errors
In particular, the BMP measure, which is widely used:
It ignores the error magnitude
It is sensitive to the error threshold selection
It measures the quantity of errors
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Slide 9
Illustration of the BMP
A low percentage of disparity estimation errors does not imply an
accurate depth recovering
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Scharstein, D. and Szeliski, R., A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, IJCV 2002
Scharstein, D. and Szeliski, R., High-accuracy Stereo Depth Maps using Structured Light, CVPR 2003
Slide 10
The Sigma Z Error Measure
We propose the Sigma Z Error (SZE) measure
The proposed measure has the following properties:
It is inherently related to depth reconstruction in a
stereo system
It is based on the inverse relation between depth and
disparity
It considers the magnitude of the estimation error
It is threshold free
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Slide 11
Illustration of the SZE
The SZE measures the impact of disparity estimation errors in terms of
distance along the Z axis of the stereo system
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
ADCensus Cones ADCensus Teddy RDP Teddy RDP Cones
Mei, X., et al., On Building an Accurate Stereo Matching System on Graphics Hardware, GPUCV 2011
Sun, X., et al., Stereo Matching with Reliable Disparity Propagation, 3DIMPVT 2011
Slide 12
Impact on Evaluation Results
Two different evaluation models were used in the experimental validation
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
Cabezas, I. and Trujillo M., A Non-Linear Quantitative Evaluation Approach for Disparity Estimation, VISAPP 2011
Scharstein, D. and Szeliski, R., http://vision.middlebury.edu/stereo/, 2011
Slide 13
Final Remarks
The SZE offers advantages over conventional error
measures such as the BMP since it considers the inverse
relation between depth and disparity
A better judging of algorithms behaviour is obtained using
the SZE
The SZE is suited to evaluate disparity estimations in
different application domains, such as: robotics, unmanned
navigation, and automatic inspection, among others
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011
A Measure for Accuracy
Disparity Maps Evaluation
Ivan Cabezas, Victor Padilla and Maria Trujillo
November 16th 2011
16th Iberoamerican Congress on Pattern Recognition, CIARP 2011, Pucón, Chile