METHODS & DESIGNS

A nonparametric procedure for evaluation of true and false positives

60 70OBSERVED AREA

experimental condition would yield notjust a single observed relation between onetrue- and one false-positive rate, but ratherthe whole family of relations. This familydefines the operating characteristicbetween true and false positives. With theentire operating characteristic available,there is little need to make strongassumptions about the underlyingtheoretical distributions, since measurescan be obtained directly in terms of theactually obtained operating characteristic.For example, one useful summary measureof performance is the area under theoperating characteristic. Green (1964) hasshown that this measure predicts theproportion of correct responses in atwo-interval forced-ehoice experiment. TheGreen equivalence relation does not requirestrong parametric assumptions aboutunderlying theoretical distributions. Theequivalence relation has been verifiedexperimentally in recognition memory byGreen and Moses (1966) and in auditorydetection tests by Emmerich (1968).

But, in most cases, each experimentalcondition yields only a single relationbetween true and false positives. Howmight we then proceed? We can takeadvantage of a nonparametric approach tothe measure of sensitivity by estimating the

,,,,

7

1,/' .~o

,,'5 .8. A'~' , 7

" J5.: .5a: " .5 .6 J .8 .9 1.0~ 0 P(TPI

/l. 0 .2 4 .6 .8 10PROP FALSE POSITiVES

80 90 100MEASURE, A in %

area under a simplified operatingcharacteristic that includes the given point.In the upper left insert of Fig. 1, a singleperformance is represented at point (x.y),where the x-coordinate defines the falsepositive rate and the y-coordinate definesthe true positive rate. The line between(x,y) and (1,1) represents the performanceanticipated if the S performed at level (x.y)on some given fraction of trials and if hearbitrarily responded positively on theremaining fraction of trials, regardless ofthe experimental conditions. The linebetween (x,y) and (0,0) represents theperformance anticipated if the S performedat level (x,y) on some fraction of trials andif he artibrarily responded negatively onthe remaining fraction of trials. Allperformances below these two lines areclearly inferior to that represented by(x,y). A similar argument can be developedfor the area marked "S," whereperformance is superior to (x,y). The areasmarked "A" are ambiguous in thatperformance may be either inferior orsuperior to that represented by (x,y).Pollack and Norman (1964) suggested thatwe formalize our ignorance by splitting upthe ambiguous area to achieve the measureofEq.l:

I

.6 .8 1.0POSITIVES

4

~ 30

c- .« 2_I-«wuzw0::wu,u,0«

-Iw0::«

-2

-350

A procedure is presented for theevaluation of single-pointestimates of trueand false positives without strongunderlying parametric assumptions. Themethod is based upon the area operatingcharacteristic and the Green area rule.Estimates of sampling error are alsoavailable. The procedure is extended to astrong one-parameter relation between trueand false positives.

In a wide range of psychological studies,it is useful to obtain a summary measure ofperformance in terms of the relationbetween true positives and false positives.For example, in a recognition memorystudy, the S may respond "old" to itemsthat have been previously presented ("truepositives"), but also may respond "old" toitems that have not been previouslypresented ("false positives"). The samemethodology can be extended to thedetection of "signals," to the identificationof emotional signs, or to any phenomenonwhere true and false positives can beidentified.

The most common summary treatmentof the true and false positives is in terms ofthe strong assumptions of the theory ofsignal detectability, namely, theassumptions of Thurstone's Case V ofunderlying normal distribution curves ofequal variance. With a given true-positiverate and a given false-positive rate, one caninfer the distance between two overlappingnormal distributions of equal variance. But,from the single-point estimate, we have noknowledge of whether or not theassumption of underlying normaldistributions is valid.

In the best of all possible worlds, each

Fig. 1. Green's area measure for areaoperating characteristics for specified levelsof A'. The A' isdeflned with respect to theupper left insert. Area operatingcharacteristics for specified levels of the A'measure are shown in the right insert. Alsodrawn, in the lowest right insert, is therelation between A' and the proportion oftrue positives for (x.y) conditions on thenegative diagonal, i.e., whereP(TP) + P(FP) = 1.0.

IRWIN POLLACK,1 Mental HealthResearch Institute, UNIVERSITY OFMICHIGAN, Ann Arbor, Michigan 48104

Behav.Res. Meth. & Instru., 1970, Vol. 2 (4) 155

This suggestion leads to an operatingcharacteristic that is symmetrical about thenegative diagonal, i.e., about a line between(1,1) and (0.5,0.5). The A' measurewill becalled the point estimate for area.

Grier (in press) has recently providedEq. 2 for calculation of A' based upon asingle true and false positive rate:

Fig. 2. The relation between k, theexponent of the ratio operatingcharacteristic, and d~, a measure derivedfrom the theory of signal detectability (leftordinate). The heavy solid line representsA' "; 'the area under the ratio operatingcharacteristic, and scaled on the insideright coordinate. The heavy dashed linerepresen ts N (A' '~);. the equivalentnormalized value of A" ,;'and scaled on theoutside right coordinate. Ratio operatingcharacteristics for specified levels of k areshown in the upper right insert.

where x and yare the false and truepositive rates, respectively.

By means of Eq. 2, a range of false andtrue positive rates can be found thatsatisfies a constant A'. We shall term theresultant relationship between falsepositives and true positives as an areaoperating characteristic. All poin ts on thearea operating characteristic yield the samepoint estimate for a given area measure.Examples of area operating characteristicsare shown in the lower right insert ofFig. 1. The smaller insert describes therelation between A' and the proportion oftrue positives for (x.y) conditions whichfall on the negative diagonal between (1,1)and (0.5,0.5).

What can we say about the areaoperating characteristic? In what mayappear to the reader (and at times to the

(3)

•Sample ratio operating characteristicsare presented in the upper right insert ofFig. 2. Note the strong asymmetry of thefunctions in contrast with those of thelower right insert of Fig. 1.

The area under the ratio operatingcharacteristic, A"'; is simply I/(k + I).The area measure is presented, on the innerright ordinate, as a function of theexponent of Eq. 3. Also presented inFig. 2, on the left ordinate, is d~, a measurederived from the intersection of the ratiooperating characteristic and the negativediagonal and then transformed intostandard or z-scores, based upon thenormal probability curve. Better agreementbetween the two measures is obtainedwhen the area measure is also transformedto equivalent z-score form, as representedby the extreme right ordinate, N(A" ').The suggestions previously directed towardthe sampling error of the area measure ofthe area operating characteristic, A", mayalso be directed toward the area measure ofthe ratio operating characteristic, A" ':' .

REFERENCESEGAN, J. P., GREENBERG, G. Z., &

SCHULMAN, A. I. Operating characteristics,signal detectability, and the method of freeresponse. Journal of the Acoustical Society ofAmerica, 1961, 33. 993-1007.

EMMERICH, D. S. Receiver-operatingcharacteristics determined under severalinteraural conditions of listening. Journal ofthe Acoustical Society of America, 1968, 43,298-307.

GREEN, D. M. General prediction relating yes-noand forced choice results. Journal of theAcoustical Society of America, 1964, 36,1042(A).

GREEN, D. M., & MOSES, F. L. On theequivalence of two recognition measures ofshort-term memory. Psychological Bulletin,1966,66, 228-234.

GRIER, J. B. Nonparametric indices forsensitivity and bias: Computing formulas.Psychological Bulletin, in press.

POLLACK, I., & HSIEH, R. Sampling variabilityof the area under the ROC-curve and of d' e­Psychological Bulletin, 1969, 71, 161-173.

POLLACK, 1.,& NORMAN, D. A. Anonparametric analysis of recognitionexperiments. Psychonomic Science. 1964, 1,125-126.

NOTE1. This research was supported in part by

Grant GB 6148 of the National ScienceFoundation. Mr. Robert Hsieh provided theprogram for Fig. L Although Eq, 2 was requiredfor the earlier development of area operatingcharacteristics, its explicit statement is due toJ. B. Grier.

power relation between true positives andfalse positives. This relation will be calledthe ratio operating characteristic.

P(TP) = [P(FP)] k where 0';;;; k ,;;;; I

(2)

.90 12

1.0 1,5

author) as an infinite regress, we can.- (z.oo) inquire about the area subtended by the

area operating characteristic.An IBM 360 computer was programmed

to yield the main results of Fig. I. In thissimulation, for each value of A', thex-coordinate was varied in steps of .01; they-coordinate yielded by Eq.2 wasobtained; and the resulting area, A", underthe area operating characteristic wascomputed by the trapezoidal rule. Theabscissa of Fig. I is A', the point estimatefor the area measure. The ordinate is thedifference between A" and A'. A" is thearea subtended by the area operatingcharacteristic for all points associated witha constant A'. The difference between thetwo measures is 0 at chance (0.50) and atperfect performance (1.00). The differenceis negative, i.e., larger Ar than Air stores areobtained for A' values to 0.78 and ispositive thereafter. The maximumdifferences are -.028 at A' = '0.66 and+0.035 at A' =0.91.

Where are we now? We started with anexperimental condition that yielded asingle estimate of true positives and falsepositives. We combined these two measuresinto the A' measure of Eq. 2. The area, A",subtended by the area operatingcharacteristic, for defined conditions of theA' measure is obtained from Fig. I.Finally, we can obtain a measure of thesampling error of A", since the samplingerror of the area measure is dependentprimarily upon its own mean value and isrelatively independent of the form of theunderlying distributions (pollack & Hsieh,1969, Fig. 7),

In partial summary, a nonparametricprocedure for the evaluation of a givenpoint of false and true positives, whichdoes not make strong parametricassumptions of specific underlyingdistributions, is:

(1) From Eq.2, due to Grier (in press),determine A'. This defines a single point onthe area operating characteristic.

(2) From Fig. I, determine A". Thisdefines the area under the associated areaoperating characteristic.

(3) From the reference listed in theprevious paragraph, determine the samplingerror of A"..

Finally, consider the application of thearea operating characteristic to one specificstrong relation between true and falsepositives. A simple one-parametricdescription of operating characteristics hasbeen provided by Egan, Greenberg, andSchulman (1961). This relationship wasfound to be an excellent descriptor ofsituations without defined trials, as in"vigilance" studies. Eq. 3 describes their

.70 .6

.60 .3

.2 46 8 1,0 80 .9PROP FALSE POSITIVES

02 0.'EXPONENT Of

A' =(y - xXI + Y- x) + ~4y(l - x)

,:I-:::;a;I! 15u

'"I-l!l

10

0.5

156 Behav. Res. Meth. & Instru., 1970, Vol. 2 (4)