parametric measures to estimate and predict performance of identification techniques amos y. johnson...
TRANSCRIPT
![Page 1: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/1.jpg)
Parametric measures to estimate and predict performance of identification techniques
Amos Y. Johnson & Aaron Bobick
STATISTICAL METHODS FOR COMPUTATIONAL EXPERIMENTS
IN VISUAL PROCESSING & COMPUTER VISIONNIPS 2002
![Page 2: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/2.jpg)
Setup – for example
Given a particular human identification technique
![Page 3: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/3.jpg)
Setup – for example
Given a particular human identification technique This technique measures 1 feature (q) from n individuals
n321 q ... q q qx
- 1D Feature Space -
![Page 4: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/4.jpg)
Setup – for example
Given a particular human identification technique This technique measures 1 feature (q) from n individuals Measure the feature again
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
![Page 5: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/5.jpg)
Setup – for example
Given a particular human identification technique This technique measures 1 feature (q) from n individuals Measure the feature again
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
![Page 6: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/6.jpg)
Setup – for example
Given a particular human identification technique This technique measures 1 feature (q) from n individuals Measure the feature again
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
For template
Target
![Page 7: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/7.jpg)
Setup – for example
Given a particular human identification technique This technique measures 1 feature (q) from n individuals Measure the feature again
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
For template
Target Imposters
![Page 8: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/8.jpg)
Question
For a given human identification technique, how should identification performance be evaluated?
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
For template
Target Imposters
![Page 9: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/9.jpg)
Possible ways to evaluate performance
For a given classification threshold, compute False accept rate (FAR) of impostors Correct accept rate (HIT) of genuine targets
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
For template
Target Imposters
![Page 10: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/10.jpg)
Possible ways to evaluate performance
For various classification thresholds, plot Multiple FAR and HIT rates (ROC curve)
![Page 11: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/11.jpg)
Possible ways to evaluate performance
For various classification thresholds, plot Multiple FAR and HIT rates (ROC curve) Compute area under a ROC curve (AUROC)
Probability of correctclassification
![Page 12: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/12.jpg)
Possible ways to evaluate performance
For various classification thresholds, plot Multiple FAR and HIT rates (ROC curve) Compute 1 - area under a ROC curve (1 -AUROC)
Probability of incorrectclassification
![Page 13: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/13.jpg)
Problem Database size
If the database is not of sufficient size, then results may not estimate or predict performance on a larger population of people.
1 - AUROC
![Page 14: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/14.jpg)
Our Goal
To estimate and predict identification performance with a small number subjects
1 - AUROC
![Page 15: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/15.jpg)
Our Solution
Derive two parametric measures Expected Confusion (EC) Transformed Expected-Confusion (EC*)
![Page 16: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/16.jpg)
Our Solution
Derive two parametric measures Expected Confusion (EC) Transformed Expected-Confusion (EC*)
Probability that an imposter’s feature vector is withinthe measurement variation of a target’s template
![Page 17: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/17.jpg)
Our Solution
Derive two parametric measures Expected Confusion (EC) Transformed Expected-Confusion (EC*)
Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector
![Page 18: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/18.jpg)
Our Solution
Derive two parametric measures Expected Confusion (EC) Transformed Expected-Confusion (EC*)
EC* = 1 - AUROC
![Page 19: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/19.jpg)
Expected Confusion
Probability that an imposter’s feature vector is within the measurement variation of a target’s template
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
For template
Target Imposters
![Page 20: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/20.jpg)
Expected Confusion - Uniform
The templates of the n individuals, are from an uniform density Pp(x) = 1/n
- 1D Feature Space -
n321 q ... q q qx
P(x)
1/nPp(x)
![Page 21: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/21.jpg)
Expected Confusion - Uniform
The measurement variation of a template is also uniform Pi(x) = 1/m
- 1D Feature Space -
n321 q ... q q qx
P(x)
1/nPp(x)
1/mPi(x)
![Page 22: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/22.jpg)
Expected Confusion - Uniform
The probability that an imposter’s feature vector is within the measurement variation of template q3 is the area of overlap
True if m << n
- 1D Feature Space -
n321 q ... q q qx
P(x)
1/nPp(x)
1/mPi(x)
n
mA )q( 3
![Page 23: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/23.jpg)
Expected Confusion - Uniform
The probability that an imposter’s feature vector is within the measurement variation of any template q
True if m << n
n321 q ... q q qx
P(x)
1/nPp(x)
1/mPi(x)
n
mA )q( 3
n
mdqPAEC p
)q()q(
![Page 24: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/24.jpg)
Following the same analysis, for the multidimensional Gaussian case
Expected Confusion - Gaussian
),()q( ppp Np : Population density
),q()( ii Nxp : Measurement variation
![Page 25: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/25.jpg)
Expected Confusion - Gaussian
Following the same analysis, for the multidimensional Gaussian case True if the measurement variation is significantly less then the population variation
2/1
2/1
||
||EC
p
i
Probability that an imposter’s feature vector is within the measurement variation of a target’s template
![Page 26: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/26.jpg)
Expected Confusion - Gaussian
Relationship to other metrics Mutual Information
The negative natural log of the EC is the mutual information of two Gaussian densities
)|ln(|)|ln(|)||
||ln( 2/12/1
2/1
2/1
ipp
i
![Page 27: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/27.jpg)
Transformed Expected-Confusion
Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector
- 1D Feature Space -
n321 q ... q q qx
''''
n321q ... q q q
Gallery
Probe
For template
Target Imposters
![Page 28: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/28.jpg)
Transformed Expected-Confusion
First: We find the probability that a target’s feature vector is some distance k away from its template
n321 q ... q q qx
''''
n321q ... q q q
For template
Target Imposters
k
)( dkkpqt
![Page 29: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/29.jpg)
n321 q ... q q qx
''''
n321q ... q q q
For template
Target Imposters
k
)( dkkpqt
Transformed Expected-Confusion
Second: We find the probability that an imposter’s feature vector is less than or equal to that distance k
k
qim dvvp
0
)(
![Page 30: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/30.jpg)
n321 q ... q q qx
''''
n321q ... q q q
Target Imposters
k
Transformed Expected-Confusion
Therefore: The probability that an imposter’s feature is closer to the target’s template, than the target’s feature (for a distance k) is
dkdqdvvpkpqpk
qim
qtp
00
)()()(
![Page 31: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/31.jpg)
n321 q ... q q qx
''''
n321q ... q q q
Target Imposters
k
Transformed Expected-Confusion
Therefore: The probability that an imposter’s feature is closer to the target’s template, than the target’s feature (for any distance k) is
dkdqdvvpkpqpk
qim
qtp
00
)()()(
![Page 32: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/32.jpg)
x
''''
n321q ... q q q
Transformed Expected-Confusion
Therefore: The expected value of this probability over all target’s templates is
dkdqdvvpkpqpk
qim
qtp
00
)()()(
n321 q ... q q q
![Page 33: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/33.jpg)
Transformed Expected-Confusion
Next: Replace the density of the distance between a target’s feature-vectors and its template q
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
)(kpt
![Page 34: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/34.jpg)
Transformed Expected-Confusion
Answer: Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
![Page 35: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/35.jpg)
Transformed Expected-Confusion
This probability can be shown to be one minus the area under a ROC curve
Following the analysis of Green and Swets (1966)
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
![Page 36: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/36.jpg)
Transformed Expected-Confusion
Integrate: With these assumptions
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
![Page 37: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/37.jpg)
Transformed Expected-Confusion
Integrate: With these assumptions
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
),;( ppqN
![Page 38: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/38.jpg)
Transformed Expected-Confusion
Integrate: With these assumptions
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
),;( ppqN
2
2
2)12/(
2/)2(
)2/(2i
k
ddi
d
ed
k
![Page 39: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/39.jpg)
Transformed Expected-Confusion
Integrate: With these assumptions
),;( ppqN d
dp kVqp )(
2
2
2)12/(
2/)2(
)2/(2i
k
ddi
d
ed
k
dkdqdvvpkpqpECk
qim
qtp
00
)()()(*
![Page 40: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/40.jpg)
Transformed Expected-Confusion
Integrate: Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector
dkdqdvvpkpqpECk
qim
qtp
00
)()()(* ECd
dVd
d
)2/()2(
)!1(2/
![Page 41: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/41.jpg)
Transformed Expected-Confusion
Compare: EC* with 1 - AUROC
EC* = 1 - AUROC
![Page 42: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/42.jpg)
Conclusion
Derive two parametric measures Expected Confusion
(EC) Transformed
Expected-Confusion (EC*)
Probability that an imposter’s feature vector is closer
to a target’s template, than the target’s feature vector
![Page 43: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/43.jpg)
Conclusion
Derive two parametric measures Expected Confusion
(EC) Transformed
Expected-Confusion (EC*)
Probability that an imposter’s feature vector is within
the measurement variation of a target’s template
Probability that an imposter’s feature vector is closer
to a target’s template, than the target’s feature vector
![Page 44: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/44.jpg)
Conclusion
Derive two parametric measures Expected Confusion
(EC) Transformed
Expected-Confusion (EC*)
Probability that an imposter’s feature vector is within
the measurement variation of a target’s template
Probability that an imposter’s feature vector is closer
to a target’s template, than the target’s feature vector
![Page 45: Parametric measures to estimate and predict performance of identification techniques Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL](https://reader036.vdocuments.site/reader036/viewer/2022062619/5517e250550346cb568b457c/html5/thumbnails/45.jpg)
Future Work
Developing a mathematical model of the cumulative match characteristic (CMC) curve Benefit: To predict how the CMC curve
changes as more subjects are added