judgments of learning and recall

8/12/2019 Judgments of Learning and Recall

1/19

How Many Dimensions Underlie Judgments of Learning and Recall?Evidence From State-Trace Methodology

Yoonhee Jang and Thomas O. NelsonUniversity of Maryland

The authors used state-trace methodology to investigate whether a single dimension (e.g., strength) issufficient to account for recall and judgments of learning (JOLs) or whether multiple dimensions (e.g.,intrinsic and extrinsic factors) are needed. The authors separately manipulated the independent variablesof intrinsic and extrinsic cues, determining their state traces for recall and JOLs. In contrast to thesupposition that intrinsic cues have similar effects on both recall and JOLs whereas extrinsic cues affectJOLs less strongly than recall (i.e., 2 dimensions underlying recall and JOLs), the authors found repeatedsupport for the sufficiency of a single dimension for both recall and JOLs (not only immediate JOLs butalso delayed JOLs) across a variety of intrinsic and extrinsic cues.

Keywords: metacognition, judgments of learning (JOLs), state-trace analysis, intrinsic and extrinsic cues,single-dimensional versus multidimensional theories of JOLs

This research concerns a kind of metacognitive monitoringknown as judgments of learning (JOLs), which are judgmentsthat occur during or after acquisition and are predictions aboutfuture test performance on recently studied items (Nelson &Narens, 1994, p. 16). JOLs are one of the most frequentlyinvestigated self-monitoring judgments and have been investigatedacross diverse areas of psychology (reviewed in Schwartz, 1994).

In spite of the large amount of research on JOLs, there is littleconsensus about what kind of theoretical structure underliesthem. In this research, we take an initial step toward under-

standing this structure by asking whether a single dimensionprovides a sufficient account of JOLs or whether multipledimensions are needed. The answer to this question will facil-itate development of a parsimonious and consistent theory of JOLs and, more generally, of metacognitive monitoring. Al-though this particular question has not been addressed previ-ously, several theories have been proposed that bear on thisissue.

Single-Dimensional Theories of JOLs

Direct-access (i.e., trace-access) theory proposes that peoplemonitor memory content, assessing the magnitude along someunderlying single dimension (e.g., memory strength in Busey,Tunnicliff, Loftus, & Loftus, 2000; Loftus, Oberg, & Dillon,2004). According to direct-access theory, a strong correspon-dence is expected between recall and JOLs because recall andJOLs are affected by the same underlying factor. The observedcorrelation between JOLs and recall performance may be lessthan unity because of noise, limited degrees of fineness in theJOLs, and other stochastic issues that can be accounted for byprobability theory.

Other ideas for a single dimension underlying JOLs includeease of processing (e.g., Begg, Duft, Lalonde, Melnick, &Sanvito, 1989) or retrieval fluency (e.g., Benjamin, Bjork, &Schwartz, 1998), although it has not been claimed specificallythat these mechanisms are solely responsible for JOLs. Instead,the claim is that one of these mechanisms may underlie bothrecall and JOLs, but this will happen only to the extent thatrecall relies on this mechanism (e.g., retrieval fluency maysometimes mislead people into giving inappropriately highJOLs when it has a greater effect on JOL magnitude than onrecall; Benjamin et al., 1998).

Multidimensional Theories of JOLs: Intrinsic andExtrinsic Cues From the Cue-Utilization Framework

Koriats (1997) cue-utilization framework proposes that peo-ple evaluate several different cues that are differentially pre-dictive of subsequent recall and that JOLs are based on heuris-tics that attempt to forecast the likelihood of recall. Forinstance, research by Begg et al. (1989; replicated and extendedby Wixted, 1992) found that although JOL magnitude wasgreater for high-frequency words than for low-frequency words,

Yoonhee Jang and Thomas O. Nelson, Department of Psychology,University of Maryland.

This research was partially supported by Cognition and StudentLearning (CASL) Research Program Grant R305H030283 from theInstitute of Education Sciences of the U.S. Department of Education.We thank Donald Bamber, Morris Goldsmith, David Huber, ThomasWallsten, and Michael Dougherty for their many valuable commentsand Geoffrey Loftus for useful information about his recent studies.Yoonhee Jang is deeply indebted to Thomas O. Nelson, who passedaway January 14, 2005, for his careful guidance throughout her grad-uate career.

Correspondence concerning this article should be addressed to YoonheeJang, Department of Psychology, University of Maryland, College Park,MD 20742. E-mail: [email protected]

Journal of Experimental Psychology: General Copyright 2005 by the American Psychological Association2005, Vol. 134, No. 3, 308326 0096-3445/05/$12.00 DOI: 10.1037/0096-3445.134.3.308

308


2/19

subsequent recognition performance was greater for low-frequency words than for high-frequency words. This crossoverinteraction 1 was interpreted as establishing that more than onefactor is required to explain both JOLs and memory perfor-mance. However, recent research by Benjamin (2003) revealedthat JOLs correctly predict both recognition (better perfor-

mance for low-frequency words) and recall (better performancefor high-frequency words), although that did not occur in theinitial study/test sequence, suggesting that the single-dimensional account is contingent on metacognitive knowledgeacquisition.

The cue-utilization framework assumes that intrinsic cues in-volve characteristics of the study items that are perceived todisclose the items a priori ease or difficulty of learning (Koriat,1997, p. 350), whereas extrinsic cues are factors that pertaineither to the conditions of learning or to the encoding operationsapplied by the learner (p. 350). For instance, Koriat (1997)proposed item difficulty and item relatedness as prototypical ex-amples of intrinsic cues and number of study presentations andstudy duration as prototypical examples of extrinsic cues. On thebasis of scale-dependent interactions (which are potentially prob-lematic, as discussed below), Koriat speculated that intrinsic cueshave approximately the same effects on recall as they have onJOLs, whereas extrinsic cues have greater effects on recall than onJOLs, which he interpreted as requiring a multidimensional struc-ture underlying recall and JOLs.

The Present Approach and Formulation of the Problem

By separately manipulating intrinsic and extrinsic cues, wetested a single-dimensional account of JOLs and recall versus amultidimensional account. Because it is often assumed that recog-

nition involves more than one process (e.g., memorability vs.discrimination), we considered only recall in the present study,focusing our question on whether a single factor underlies bothJOLs and memorability.

We conceptualize the relation between JOLs and recall in thefollowing way. If JOLs arise from only a single dimension, thenindependent variables can affect only that one underlying dimen-sion (perhaps with noise and/or bias). However, if JOLs arise frommultiple underlying dimensions, then the independent variablescan differentially affect those underlying dimensions.

For instance, Figure 1 illustrates the single-dimensional modelin which the two independent variables (e.g., item difficulty andnumber of study presentations) are assumed to affect a singledimension of the memory representation (D1), which determinesboth JOLs and recall: JOLs f(D1), and recall g(D1), where f and g are positive monotonic functions, with f not necessarilyidentical to g. Thus, both JOL magnitude and recall are assumed tobe monotonic functions of the same single dimension of thememory representation, D1.

In contrast to this single-dimensional account, one possiblemultidimensional model is illustrated in Figure 2, which is basedon the hypothesis that intrinsic cues have the same effects on bothrecall and JOLs whereas extrinsic cues affect recall more strongly

1 Crossover interactions are important because they allow conclusions of an underlying multidimensional structure, in contrast to converging/diverg-ing interactions, which do not require an assumption beyond a single-dimensional structure (e.g., Dunn & Kirsner, 1988). Converging/diverginginteractions (i.e., scale-dependent interactions) also have problems of meaningfulness (Krantz & Tversky, 1971; Loftus, 1978; Townsend &Ashby, 1984), as elaborated in the General Discussion section.

Figure 1. A single-dimensional model for the relation between two independent variables (i.e., item difficultyas an intrinsic cue and number of study presentations as an extrinsic cue) and two dependent variables (i.e., judgments of learning [JOLs] and recall). The single underlying dimension is referred to as D1, and both f andg are monotonic functions with f not necessarily identical to g.

309STATE TRACES FOR JUDGMENTS OF LEARNING AND RECALL


3/19

than JOLs. This multidimensional model (which is only one of many possible multidimensional models) contains two underlyingdimensions (D1 and D2), where D1 is a monotonic function of both item difficulty and number of study presentations, and D2 isa monotonic function of only number of study presentations (i.e.,a two-dimensional model). JOL magnitude is primarily determined

by D1 as in the single-dimensional model. Although JOL magni-tude may also be affected by D2, the effect is small or nonexistent,as indicated by the dotted arrow between D2 and JOLs. Bycontrast, recall is equally affected by both D1 and D2: JOLs f(D1), and recall g(D1, D2), where f and g are positive mono-tonic functions, as described above. Hence, item difficulty exertssimilar effects both on JOLs and on recall, whereas number of study presentations affects JOLs less strongly than recall. Thisparticular two-dimensional model is in accord with the cue-utilization framework.

State-Trace Analysis

We utilized state-trace analysis (originally proposed by Bam-ber, 1979) to test the number of dimensions underlying JOLs andrecall. State-trace analysis is related to conjoint measurementtheory (e.g., this relationship is elaborated on by Loftus et al.,2004) and the logic of additive and multiplicative effects (seeLoftus, 2002). State-trace analysis is achieved by means of ascatter-plot graph showing the covariation of two dependent vari-ables and the manner in which the independent variables affect thedependent variables. A major goal of state-trace analysis is tocompetitively evaluate conceptualizations of single-dimensionaland multidimensional theoretical structures by evaluating curves,referred to as state traces , within such a scatter plot. According toa review by Dunn and James (2003),

State-trace analysis comprises a conceptual framework within whichmodels of the relationships between different dependent variables canbe represented. It also incorporates a method for identifying andtesting these relationships. . . . Thus, if two dependent variables arefunctions of the same latent variable, the resulting state-trace is aone-dimensional curve in two-dimensional state space. (pp. 404405)

The application of state-trace analysis to the present situation isillustrated in Figure 3, which shows predicted outcomes from theFigure 1 single-dimensional model (see Figure 3A, 3B, and 3C) ascompared with predicted outcomes from the Figure 2 two-dimensional model (see Figure 3D, 3E, and 3F). The first twopanels of each row illustrate the separate effects of the independentvariables on the two dependent variables of recall (see Figure 3Aand 3D) and JOL magnitude (see Figure 3B and 3E). In isolation,these first two panels do not qualitatively differentiate between thesingle-dimensional account versus the two-dimensional account of the dependent variables. Instead, this qualitative model compari-son is achieved by combining the two plots into state-trace plots,as shown in the third panel of each row.

As illustrated in Figure 3C, the critical prediction of the single-dimensional model is that both the one-presentation and two-presentation curves lie along a single curve. The location of eachcurve arises from the causal paths shown in Figure 1: (a) from itemdifficulty through D1 to JOLs, and from item difficulty through D1to recall; and (b) from number of study presentations through D1to JOLs, and from number of study presentations through D1 torecall. The critical aspect of the state-trace analysis is highlightedby the two arrows in Figure 3C, which identify the overlappingportion of the one-presentation and two-presentation curves, withthe upper arrow identifying easy items that have been presentedonce and the lower arrow identifying difficult items that have been

Figure 2. A hypothetical two-dimensional model for the relation between two independent variables (i.e., itemdifficulty as an intrinsic cue and number of study presentations as an extrinsic cue) and two dependent variables(i.e., judgments of learning [JOLs] and recall). The two underlying dimensions are referred to as D1 and D2, andf and g are monotonic functions with f not necessarily identical to g. The dotted arrow between D2 and the JOLsindicates that there is little or no effect of D2 on JOLs.

310 JANG AND NELSON


4/19

presented twice. In the single-dimensional model, different condi-tions giving rise to a particular JOL rating must also give rise to aparticular recall performance level (and vice versa), consideringthat any specific value on a dependent variable is obtained onlythrough dimension D1. Therefore, the critical test of the single-dimensional model is the extent to which the two curves fall atopeach other.

This can be formally stated as follows. Assume that C (i, j)represents the joint condition of the ith level of one independentvariable (where i can take on the values of, say, a or b) and the jthlevel of another independent variable (where j can take on thevalues of, say, q or r ). Then, according to the single-dimensionalmodel in Figure 1, the following must be the case: (a) If C (a , r )produces the same JOL as does C (b, q), then C (a , r ) must producethe same value of D1 as did C (b, q); and (b) because C (a , r ) and

C (b, q) have the same value of D1, they must produce the samerecall. In short, the strong prediction of the single-dimensionalmodel is that whenever JOL magnitude is the same for C (a , r ) asfor C (b, q), then recall must be the same for C (a , r ) as for C (b, q).Hence, in Figure 3C, the two curves must be atop each otherthroughout the range in which the two curves have the same valuesof recall (or the same magnitudes of JOLs), as indicated by theportion of the two curves between the two arrows.

By contrast, the data pattern that definitively specifies a multi-dimensional interpretation is one in which the two curves areseparated, such as in Figure 3F. The two-presentation curve couldfall either to the left or to the right of the one-presentation curve,with each pattern indicating the particular connection strengths of the multidimensional structure. For instance, the prediction fromthe cue-utilization framework (from Figure 2) is illustrated in

Figure 3. Predictions of the two models from Figures 1 and 2 (the top three panels show illustrative outcomespredicted by the single-dimensional model, and the bottom three panels show illustrative outcomes predicted bythe two-dimensional model from the cue-utilization framework). Panels A, B, D, and E show traditional data inwhich the two dependent variables (i.e., recall and judgments of learning [JOLs]) are plotted as functions of thetwo independent variables (i.e., item difficulty and one vs. two presentations). Panels C and F show state tracesin which JOL magnitude is plotted against the percentage of correct recall. The data shown between the twoarrows in Panel C are the overlapping portion of the one-presentation curve and the two-presentation curve.



5/19

Figure 3F, wherein the two-presentation curve falls to the right of the one-presentation curve. The location of these two curves arisesfrom the causal paths shown in Figure 2. For this example, JOLsare determined by only D1, and so both of the presentationconditions provide the same value of JOLs (viz., 55%). However,because recall is determined by both D1 and D2, the percentage of

correct recall is higher in the two-presentation condition (e.g.,80%) than in the one-presentation condition (e.g., 60%). Hence,the two-presentation curve in Figure 3F falls to the right of theone-presentation curve (instead of the two curves falling atop eachother as in Figure 3C).

It is important to note that a multidimensional model is capableof producing a single curve, such as would be the case for equallyweighted inputs and outputs to each of the dimensions. However,if such a single-curve pattern is observed repeatedly, across dif-ferent situations, this indicates at a minimum that the multipledimensions are entirely redundant and function as a single dimen-sion. For this reason, a single experiment finding a single curvecannot definitely rule out the possibility of multiple dimensionsthat happen to have conspired to produce a single curve for thatparticular situation. It is for this reason that claims of a singledimension are most effectively made by examining multipleexperiments.

It is also important to note that traditional tests of interactionsbetween independent variables appear to test something similar tothat tested using state-trace analysis, but there are critical differ-ences. Specifically, interactions in an analysis of variance(ANOVA) assume a linear combination of variables, whereasstate-trace analysis allows for nonlinearities in the relationshipbetween the underlying dimensions and the dependent measures.Indeed, it is possible to observe a significant interaction eventhough the state traces lie along a single curve. This fact serves tohighlight the inherent limitations of the general linear model.

Specific Goals of the Present Research

The primary goal of the present research was to assess thenumber of dimensions required for JOLs and recall. Specifically,we wanted to empirically test the predictions of the single-dimensional model to determine whether it is sufficient to accountfor JOLs and recall or, alternatively, to determine whether amultidimensional model is necessary (e.g., the cue-utilizationframeworks version of a two-dimensional model).

A secondary aim of our experiments was to separately examinethis relationship for delayed JOLs as well as for immediate JOLs(which were the only kind of JOLs investigated by Dunlosky &Matvey, 2001, and by Koriat, 1997). Delayed JOLs typicallypredict subsequent recall better than do immediate JOLs (Nelson& Dunlosky, 1991). There is not yet a consensus explanation forthis delayed-JOL effect, which can be explained through a varietyof mechanisms (e.g., Kimball & Metcalfe, 2003; Nelson & Dun-losky, 1992; Spellman & Bjork, 1992; Weaver & Kelemen, 1997).Although cursory consideration of that issue suggests differentmechanisms for immediate and delayed JOLs, some recent studieshave proposed that the delayed-JOL effect may instead correspondto different settings within a single underlying mechanism (e.g.,Nelson, Narens, & Dunlosky, 2004). Nelson et al. (2004) reportedthat JOL accuracy differences can be explained by the differentialbreakdown in the number of dyads comprising immediate and

delayed JOL accuracy. Most of the dyads for immediate JOLsconsist primarily of items that can be both recalled at the time of the JOL, whereas most of the dyads for delayed JOLs consistprimarily of one item that can be recalled and one item that cannotbe recalled at the time of the JOL. In other words, for immediateJOLs, discrimination between the two items is relatively difficult

(i.e., discrimination between recalled items), whereas for delayedJOLs, discrimination between the two items is relatively easy (i.e.,discrimination between a recalled item vs. a nonrecalled item). Itis important to note that this explanation appeals only to a naturalincrease in variability of memorability as a function of delay andis consistent with a single-dimensional interpretation of JOLs andrecall. Thus, we wanted to evaluate the state-trace plots for bothimmediate and delayed JOLs. This procedure allowed additionalopportunitiesnot only from immediate JOLs but also from de-layed JOLsfor the single-dimensional model to fail and for thenecessity of a multidimensional model to be confirmed through theobservation of separate state traces.

Overview of the ExperimentsAll experiments used paired study and cued recall testing, with

JOLs given in response to the cue word. Separate experimentsmanipulated the intrinsic cue of item difficulty or item relatednessand the extrinsic cue of number of study presentations or studyduration. One intrinsic cue and one extrinsic cue were manipulatedin each experiment, and all experiments included both delayed andimmediate JOLs.

We used a 2 2 2 repeated measures design for eachexperiment in which the three independent variables were anintrinsic cue (viz., easy vs. difficult items for Experiments 1A and1C; related vs. unrelated items for Experiments 1B, 1D, and 2), anextrinsic cue (viz., one vs. two presentations for Experiments 1A,1B, and 2; short vs. long presentation for Experiments 1C and 1D),and timing of JOLs (viz., immediate vs. delayed JOLs). The twodependent variables were the percentage of correct recall and JOLmagnitude. For each experiment, we investigated the assumptionthat the independent variables of intrinsic and extrinsic cues hadsignificant effects on both recall and JOL magnitude. Significanceof these effects is a critical prerequisite for state-trace analysisbecause, otherwise, an observation of a single trace could resultfrom a null effect for one of the independent variables. In all, theexperiments yielded 10 state-trace plots that were used to compet-itively evaluate the predictions from the single-dimensional model(see Figure 3C) versus a multidimensional model (e.g., the two-dimensional model from the cue-utilization framework; see Figure

3F).

Experiments 1A1D

For Experiments 1A and 1C, the manipulated intrinsic cue wasitem difficulty, whereas for Experiments 1B and 1D, it was itemrelatedness. For Experiments 1A and 1B, the manipulated extrinsiccue was number of study presentations, whereas for Experiments1C and 1D, it was study duration. During the study phase, partic-ipants were instructed to learn each pair and to make a JOL aboutthe likelihood that they would subsequently be able to recall thetarget word when the cue word was presented. During the test

312 JANG AND NELSON


6/19


7/19

4A7A, and the corresponding mean for items receiving delayedJOLs is shown in Figures 4D7D. The mean magnitude of JOLsfor each condition for items receiving immediate JOLs is shown inFigures 4B7B, and the corresponding mean for items receivingdelayed JOLs is shown in Figures 4E7E.

As shown in Appendix A and in Figures 4A, 4B, 4D, and4E7A, 7B, 7D, and 7E, the pattern of results was quite consistentacross Experiments 1A1D. Thus, for both recall and JOL mag-nitude, the prerequisite was met that both the effect of intrinsiccues and the effect of extrinsic cues were significant. For bothrecall and JOL magnitude, neither the two-way interaction of intrinsic and extrinsic cues nor the three-way interaction involvingintrinsic and extrinsic cues was statistically significant. The pre-requisite allows for analyses of state traces, as described next.

State traces of JOLs and recall. The outcome of the state-traceanalysis of recall and JOL magnitude for items receiving imme-diate JOLs is shown in Figures 4C7C, and the correspondingoutcome for items receiving delayed JOLs is shown in Figures4F7F. Of primary importance, each of those panels of Figures 4and 5 shows that the two-presentation curve falls atop the one-presentation curve, and each of those of Figures 6 and 7 shows thatthe long-presentation curve falls atop the short-presentation curve(also note that the bidirectional standard errors are quite small).The consistency of this outcome across Experiments 1A1D sug-gests that the extra flexibility of a multidimensional model, such asthe two-dimensional model from the cue-utilization framework, isnot needed, with the results most parsimoniously explainedthrough a single underlying dimension for both recall and JOLs.

Figure 4. Results of Experiment 1A. Panels A and B show the mean percentage of correct recall and the meanmagnitude of judgments of learning (JOLs) for items having immediate JOLs, whereas Panels D and E show thecorresponding data for items having delayed JOLs as a joint function of item difficulty and number of studypresentations. Panel C is the state-trace plot of the mean magnitude of immediate JOLs against the meanpercentage of correct recall, whereas Panel F is the state-trace plot of the mean magnitude of delayed JOLsagainst the mean percentage of correct recall. Each vertical and horizontal hash mark depicts the standard errorof the mean.

314 JANG AND NELSON


8/19

Experiment 2

Experiment 2 was similar to Experiment 1B, in which theintrinsic cue was item relatedness and the extrinsic cue was num-ber of study presentations, except that the instructions during thestudy/JOL phase were changed to encourage participants to inten-tionally use a comparison process when they made JOLs. Supposethat a participant studied a pair comprised of unrelated words (e.g.,bottlecalendar ) and responded with a JOL of 30%. If the partic-ipant subsequently studied a pair of related words (e.g., stove kitchen ), then the rating might increase, say, to 70% because theperson could compare and contrast the degree of the relatedness in

the second pair with that in the first pair (Koriat, 1997). Becauseparticipants might not have used such a comparison process inExperiment 1B (and in Experiments 1A, 1C, and 1D as well), theinstructions in Experiment 2 were constructed to encourage such acomparison process, assuming that such a comparison process mayconstitute a critical aspect of a multidimensional account of JOLs.

Method

Participants. Forty-five volunteers from undergraduate psychologycourses at the University of Maryland received course credit in return fortheir participation in Experiment 2.

Figure 5. Results of Experiment 1B. Panels A and B show the mean percentage of correct recall and the meanmagnitude of judgments of learning (JOLs) for items having immediate JOLs, whereas Panels D and E show thecorresponding data for items having delayed JOLs as a joint function of item relatedness and number of studypresentations. Panel C is the state-trace plot of the mean magnitude of immediate JOLs against the meanpercentage of correct recall, whereas Panel F is the state-trace plot of the mean magnitude of delayed JOLsagainst the mean percentage of correct recall. Each vertical and horizontal hash mark depicts the standard errorof the mean.



9/19

Materials and procedure. The materials and procedure of Experiment2 were identical to those of Experiment 1B except for the instructions

during the study/JOL phase. The participants were informed that themembers of each pair were related for some of the pairs but not for otherpairs, and they were encouraged to make greater JOLs for related itemsthan for unrelated items.

Results and Discussion

Prerequisite: Effects of intrinsic and extrinsic cues on recall and JOL magnitude. The mean percentage of correct recall for eachcondition for items receiving immediate JOLs is shown in Figure8A, and the corresponding mean for items receiving delayed JOLsis shown in Figure 8D. The mean magnitude of JOLs for eachcondition for items receiving immediate JOLs is shown in Figure

8B, and the corresponding mean for items receiving delayed JOLsis shown in Figure 8E.

As shown in Appendix A and in Figure 8A, 8B, 8D, and 8E,there were significant main effects of the intrinsic and extrinsiccues on both recall and JOL magnitude. The two-way interac-tion of intrinsic and extrinsic cues on JOL magnitude was notsignificant (as in Experiments 1A1D), whereas the two-wayinteraction of intrinsic and extrinsic cues on recall was signif-icant. This interaction was revealed as a greater effect of number of study presentations for the case of unrelated items(where recall was intermediate) compared with related items(where recall was closer to ceiling). Presumably, this interac-tion resulted from a ceiling effect, but it is important to notethat, as described next, the state-trace analysis placed these

Figure 6. Results of Experiment 1C. Panels A and B show the mean percentage of correct recall and the meanmagnitude of judgments of learning (JOLs) for items having immediate JOLs, whereas Panels D and E show thecorresponding data for items having delayed JOLs as a joint function of item difficulty and study duration. PanelC is the state-trace plot of the mean magnitude of immediate JOLs against the mean percentage of correct recall,whereas Panel F is the state-trace plot of the mean magnitude of delayed JOLs against the mean percentage of correct recall. Each vertical and horizontal hash mark depicts the standard error of the mean.

316 JANG AND NELSON


10/19

conditions along a single curve. There were no three-wayinteractions for either recall or JOL magnitude. With the pre-requisite met, analyses of state traces are described next.

State traces of JOLs and recall. The outcome of the state-traceanalysis of recall and JOL magnitude for items receiving imme-diate JOLs is shown in Figure 8C, and the corresponding outcomefor items receiving delayed JOLs is shown in Figure 8F. As inFigure 5 of Experiment 1B, each of those panels shows that thetwo-presentation curve falls atop the one-presentation curve. As inExperiments 1A1D, the consistency of the result suggests that amultidimensional account is not needed and that a single-dimensional account is sufficient.

General Discussion

The primary goal of this research was to investigate the structureunderlying recall and JOLs by applying state-trace methodology todetermine whether one underlying dimension is sufficient orwhether multiple underlying dimensions are needed (e.g., the twodimensions proposed in the cue-utilization framework). Across allexperiments investigating immediate and delayed JOLs, all 10state-trace plots of recall and JOL magnitude consistently yieldedstate-trace curves that fell atop each other, as predicted by theassumption that only one dimension underlies both JOL magnitudeand recall. The failure to disconfirm the single-dimensional model

Figure 7. Results of Experiment 1D. Panels A and B show the mean percentage of correct recall and the meanmagnitude of judgments of learning (JOLs) for items having immediate JOLs, whereas Panels D and E show thecorresponding data for items having delayed JOLs as a joint function of item relatedness and study duration.Panel C is the state-trace plot of the mean magnitude of immediate JOLs against the mean percentage of correctrecall, whereas Panel F is the state-trace plot of the mean magnitude of delayed JOLs against the meanpercentage of correct recall. Each vertical and horizontal hash mark depicts the standard error of the mean.



11/19

occurred even when participants in Experiment 2 were instructedto intentionally use a comparison process that should have in-creased differential effects of intrinsic and extrinsic cues on JOLs(per the cue-utilization framework).

At first glance, the finding that the pattern of state traces forimmediate JOLs did not differ from that of state traces for delayedJOLs is surprising because previous researchers speculated that therelation between recall and JOLs might change over time. How-ever, the results of this study are in accord with a formulation thatascribes most of the greater accuracy of delayed JOLs to differentratios of easier versus more difficult discriminations between itemswithout invoking different psychological processes for immediateversus delayed JOLs (Nelson et al., 2004).

A multidimensional model will imitate the single-dimensionalmodel, yielding state-trace curves that fall atop each other, if therelations from the independent variables through each of the di-mensions to the dependent variables are weighted to the samedegree. Manipulating various combinations of the intrinsic andextrinsic cues, our experiments afforded multiple opportunities torule out the single-dimensional model in at least a particular case.Across all experiments, however, the results that consistentlyyielded evidence for the single-dimensional model suggest that themultiple dimensions are unnecessary.

The question of what is the single-dimensional structure thatunderlies JOL magnitude and recall is a topic for future research.Whether the single-dimensional structure is strength, ease of pro-

Figure 8. Results of Experiment 2. Panels A and B show the mean percentage of correct recall and the meanmagnitude of judgments of learning (JOLs) for items having immediate JOLs, whereas Panels D and E show thecorresponding data for items having delayed JOLs as a joint function of item relatedness and number of studypresentations. Panel C is the state-trace plot of the mean magnitude of immediate JOLs against the meanpercentage of correct recall, whereas Panel F is the state-trace plot of the mean magnitude of delayed JOLsagainst the mean percentage of correct recall. Each vertical and horizontal hash mark depicts the standard errorof the mean.

318 JANG AND NELSON


12/19

cessing, retrieval fluency, or something else is an open question.However, research designed to answer that question should gobeyond postulating metaphorical structures and instead operation-alize the various possibilities to distinguish empirically betweenthem.

Although the present research attempted to address the issue of

dimensionality from a theoretically agnostic standpoint, we none-theless make a few remarks about the relation between this re-search and Koriats (1997) cue-utilization framework. We used thesame set of independent variables that Koriat dichotomized intointrinsic versus extrinsic cues. Thus, our conclusion that JOLs arebased on a single-dimensional construct is limited to the set of independent variables manipulated in our experiments accordingto Koriats dichotomy. Although we cannot generalize to othernonexamined independent variables, we can generalize our con-clusions to the independent variables that Koriat viewed as beingprototypical for his intrinsic versus extrinsic distinction. Presum-ably, other variables might suggest a multidimensional structure.In fact, a candidate set of dimensions of JOLs is under consider-ation. Koriat, Bjork, Sheffer, and Bar (2004) recently reportedsome evidence for two underlying dimensions of JOLs. Theyshowed that JOLs are insensitive to retention interval relative torecall, suggesting a distinction between experience-based andtheory-based JOLs. They attributed the indifference of JOLs toretention interval to the predominant dependence on subjectiveexperience (i.e., experience-based JOLs). Although further empir-ical research is needed to fully understand how the theory-basedknowledge functions and can be combined with the experience-based knowledge as Koriat et al. suggested, it should be empha-sized that this dual-basis view serves as one potential multidimen-sional model.

To explore the difference between our conclusions and those of Koriat (1997), we conducted another ANOVA for each experi-

ment, treating the contrast of recall and JOLs (labeled measureby Koriat) as a repeated variable. The complete 2 2 2 (i.e.,measure, extrinsic cue, and intrinsic cue) ANOVAs of all experi-ments are reported in Appendix B. According to the cue-utilizationframework, there should be an interaction of measure and extrinsiccue, whereas there should be little or no interaction of measure andintrinsic cue. Specifically, the interaction of measure and extrinsiccue should yield the pattern of results indicating that recall is muchhigher than JOL magnitude in the strong level of extrinsic cues(i.e., two presentations in Experiments 1A, 1B, and 2; long pre-sentation in Experiments 1C and 1D). Note that because Koriatsexperiments examined only immediate JOLs, the comparison be-tween our results and those of Koriat should be limited to only ourimmediate JOL conditions (although the patterns of results in theimmediate and delayed JOL conditions of this study are similar).The ANOVAs showed that, first, the interaction of measure andextrinsic cue was significant in all experiments except for Exper-iment 1C. The discounted effect of extrinsic cues on JOLs foritems of two presentations (i.e., underconfidence, as shown inTable B2) was found in Experiments 1B and 2, which is consistentwith the hypothesis of the cue-utilization framework. This patternof interaction, however, was not found in any of the other exper-iments. Indeed, Experiment 1A yielded the opposite pattern; JOLswere overestimated for items of one presentation (i.e., overconfi-dence, as shown in Table B2). Second, the interaction of measureand intrinsic cue was significant in all experiments; Experiments

1A and 1C yielded overestimated JOLs for difficult items (and foreasy items of Experiment 1C), whereas Experiments 1B, 1D, and2 yielded underestimated JOLs for related items. Neither of theresults was consistent with the hypothesis of the cue-utilizationframework. In the present research, on the whole, the conceptualdistinction of the intrinsic versus extrinsic cues failed functionally

to confirm the predictions from the cue-utilization framework thatwhereas intrinsic cues have similar effects on both recall and JOLs,extrinsic cues affect recall more strongly than JOLs.

Indeed, Koriat (1997) found inconsistent effects of intrinsic cuesin his experiments; for instance, whereas his Experiments 2 and 3yielded equivalent effects of intrinsic cues on both recall and JOLs,his Experiment 1 yielded greater effects on JOLs than on recall(see Figure 2, top panel, p. 354), and his Experiment 4 yieldedweaker effects on JOLs than on recall. Likewise, other findingsinconsistent with conclusions derived from the cue-utilizationframework have been reported; for instance, Dunlosky and Matvey(2001) concluded that both outcomes are inconsistent with pre-dictions from the cue-utilization framework [which] provides moreof an empirical generalization (i.e., a taxonomy of effects) and nota theoretical explanation for why various factors differentiallyinfluence JOLs (p. 1186). In addition, Busey et al. (2000) re-ported that exposure duration had a similar effect on JOL magni-tude as on memory performance, whereas the amount of rehearsalhad a greater effect on JOL magnitude than on memory perfor-mance (although both exposure duration and the amount of re-hearsal are extrinsic cues).

The assumption of Koriats intrinsic versus extrinsic distinctionwas based on converging/diverging interactions that were scaledependent in the sense that the conclusion of an interaction de-pends critically on the particular scaling both of JOL magnitudeand of recall. That is, although Koriats interactions were signifi-cant in the statistical sense (of rejecting the null hypothesis of

parallel curves), they were not meaningful in the measurementsense (i.e., conclusions drawn from them will carry over only tothe particular measures he reported or to some linear transforma-tion of them, but there is no evidence that the relationship betweenthose values and the underlying structure is necessarily linear). Bycontrast, a positive monotonic nonlinear transformation couldtransform Koriats interactions to parallel (cf. Krantz & Tversky,1971, and the admonishment about drawing conclusions fromscale-dependent interactions in the tutorial by Loftus, 1978). Suchconverging/diverging interactions, in contrast to crossover inter-actions (or interactions in which two curves have opposite-direction slopes), are known in the literature as being problematicas a basis for inferring underlying multidimensional structures(e.g., Dunn & Kirsner, 1988; Krantz, Luce, Suppes, & Tversky,1971; Loftus, 1978).

An advantage of the state-trace methodology used in the presentresearch over standard parametric ANOVAs is that it is not besetwith the aforementioned problem of meaningfulness of conclu-sions that occur when conclusions are based on scale-dependentinteractions. As readers can prove to themselves, any monotonic(linear or nonlinear) transformation can be applied to the values wereported for JOL magnitude and recall without eliminating theoverlap of the curves in the state-trace plots shown in Figures 4Cand 4F8C and 8F. Our conclusions about an underlying single-dimensional structure being sufficient to account for the perfor-mance in our experiments are meaningful across all monotonic



13/19


14/19


15/19

Appendix B

The complete results from 2 2 2 (i.e., measure, extrinsic cue, and intrinsic cue) analyses of variance are reported in Table B1. Theresults from follow-up simple effect tests of the interactions between measure and extrinsicintrinsic cue in the condition of immediate- judgments of learning (JOLs) are reported in Table B2, and the corresponding results in the condition of delayed JOLs are reported in Table B3.

Table B1Complete 2 2 2 (Measure, Extrinsic Cue, and Intrinsic Cue) Analyses of Variance

Experiment

Immediate JOLs Delayed JOLs

F (1, 44) MSE p ES F (1, 44) MSE p ES

Experiment 1AM 1.13 728.11 .29 1E 73.05 322.00 .001 .62 76.58 606.78 .001 .64I 142.74 285.45 .001 .76 161.81 445.93 .001 .79M E 19.58 168.59 .001 .31 9.59 48.70 .01 .18M I 27.87 169.31 .001 .39 5.66 108.50 .05 .11E I 1 1M E I 1.79 86.77 .19 4.99 68.24 .05 .10

Experiment 1B

M 3.08 655.92 .09 8.50 422.25

.01 .16E 70.98 276.92 .001 .62 53.01 365.97 .001 .55I 198.78 428.38 .001 .82 81.52 656.72 .001 .65M E 13.45 169.82 .001 .23 1M I 20.26 231.69 .001 .32 15.78 80.60 .001 .26E I 1 1M E I 1 1

Experiment 1CM 54.64 650.31 .001 .55 1E 13.80 181.63 .001 .24 11.58 578.99 .01 .21I 151.47 247.01 .001 .78 103.79 444.06 .001 .70M E 1.47 222.99 .23 2.09 60.51 .16M I 14.69 151.99 .001 .25 16.58 52.55 .001 .27E I 2.36 161.15 .13 2.82 361.99 .10M E I 1.32 145.84 .26 1

Experiment 1DM 1.07 616.28 .31 5.37 285.60 .05 .11E 24.83 145.75 .001 .36 87.34 359.75 .001 .66I 245.53 433.75 .001 .85 108.06 516.68 .001 .71M E 5.28 146.28 .05 .11 1M I 33.16 165.67 .001 .43 9.99 840.28 .01 .19E I 1 1M E I 1 1

Experiment 2M 7.75 1443.01 .05 .15 7.24 613.86 .05 .14E 75.92 209.59 .001 .63 56.50 588.60 .001 .56I 148.95 396.92 .001 .77 50.92 565.57 .001 .54M E 6.50 157.19 .05 .13 7.82 79.61 .01 .15M I 4.44 324.60 .05 .09 6.67 83.18 .05 .13E I 8.12 218.94 .01 .16 1.55 240.69 .22M E I 10.55 117.02 .01 .19 3.71 67.36 .06

Note. Effect size (ES) is reported only when the F value was significant. JOLs judgments of learning; M measure (recall vs. JOLs in all experiments);

E extrinsic cue (number of study presentations in Experiments 1A, 1B, and 2: one vs. two presentations; study duration in Experiments 1C and 1D: shortvs. long duration); I intrinsic cue (item difficulty in Experiments 1A and 1C: easy vs. difficult items; item relatedness in Experiments 1B, 1D, and 2:related vs. unrelated items).

322 JANG AND NELSON


16/19

Table B2Simple Effect Tests Following the Interactions Between Measure and ExtrinsicIntrinsic Cue: Immediate JOLs

Experiment and interaction t (44) p Over/underconfidence

Measure Extrinsic Cue

1ARecall vs. JOLs of one presentation 3.19 .01 OverconfidenceRecall vs. JOLs of two presentations .88 .38

1BRecall vs. JOLs of one presentation .10 .92Recall vs. JOLs of two presentations 3.02 .01 Underconfidence

1DRecall vs. JOLs of short presentation .08 .94Recall vs. JOLs of long presentation 1.94 .06

2Recall vs. JOLs of one presentation 1.86 .07Recall vs. JOLs of two presentations 3.41 .01 Underconfidence

Measure Intrinsic Cue

1ARecall vs. JOLs of difficult items 3.36 .01 OverconfidenceRecall vs. JOLs of easy items 1.30 .20

1BRecall vs. JOLs of unrelated items .70 .48Recall vs. JOLs of related items 4.42 .001 Underconfidence

1CRecall vs. JOLs of difficult items 9.29 .001 OverconfidenceRecall vs. JOLs of easy items 4.56 .001 Overconfidence

1DRecall vs. JOLs of unrelated items 1.94 .06Recall vs. JOLs of related items 3.26 .01 Underconfidence

2Recall vs. JOLs of unrelated items 1.56 .13Recall vs. JOLs of related items 3.55 .01 Underconfidence

Note. JOLs judgments of learning.

( Appendixes continue )



17/19

Table B3Simple Effect Tests Following the Interactions Between Measure and ExtrinsicIntrinsic Cue: Delayed JOLs

Experiment and interaction t (44) p Over/underconfidence

Measure Extrinsic Cue

1ARecall vs. JOLs of one presentation 1.40 .17Recall vs. JOLs of two presentations 1.13 .26

2Recall vs. JOLs of one presentation 1.64 .11Recall vs. JOLs of two presentations 3.41 .01 Underconfidence

Measure Intrinsic Cue

1ARecall vs. JOLs of difficult items 1.43 .16Recall vs. JOLs of easy items 1.21 .24

1BRecall vs. JOLs of unrelated items 1.02 .31Recall vs. JOLs of related items 4.54 .001 Underconfidence

1CRecall vs. JOLs of difficult items 2.57 .05 OverconfidenceRecall vs. JOLs of easy items 1.42 .16

1DRecall vs. JOLs of unrelated items .58 .56Recall vs. JOLs of related items 3.28 .01 Underconfidence

2Recall vs. JOLs of unrelated items 1.59 .12Recall vs. JOLs of related items 3.57 .01 Underconfidence


Appendix C

Mean Goodman-Kruskal gamma correlations between recall and judgments of learning arereported in Tables C1C5, and the complete results from analyses of variance of the gammas are

reported in Table C6.

Table C1 Mean Goodman-Kruskal Gamma Correlations Between Recall and JOLs of Experiment 1A as aFunction of Timing of JOLs, Item Difficulty, and Number of Study Presentations

Item difficulty

Timing of JOLs

Immediate( M .41, SEM .09)

Delayed( M .76, SEM .04)

OverallOne study

presentationTwo study

presentationsOne study


presentations

M SEM M SEM M SEM M SEM M SEM

Easy .28 .23 .64 .13 .95 .05 .66 .17 .63 .08Difficult .28 .20 .43 .17 .54 .18 .89 .06 .53 .07

Overall .28 .18 .53 .14 .74 .09 .78 .09


324 JANG AND NELSON


18/19

Table C2 Mean Goodman-Kruskal Gamma Correlations Between Recall and JOLs of Experiment 1B as aFunction of Timing of JOLs, Item Relatedness, and Number of Study Presentations

Item relatedness

Timing of JOLs

Immediate

( M .28, SEM .07)

Delayed

( M .77, SEM .05)

OverallOne study




presentations


Related .13 .17 .04 .17 .80 .07 .73 .09 .43 .06Unrelated .55 .13 .41 .15 .73 .09 .80 .08 .62 .06

Overall .34 .11 .22 .13 .76 .07 .77 .06


Table C3

Mean Goodman-Kruskal Gamma Correlations Between Recall and JOLs of Experiment 1C as aFunction of Timing of JOLs, Item Difficulty, and Study Duration

Item difficulty

Timing of JOLs



OverallStudy

duration: 5 sStudy

duration: 15 sStudy

duration: 5 sStudy

duration: 15 s


Easy .58 .16 .08 .22 .83 .09 .80 .13 .57 .09Difficult .13 .20 .38 .26 .53 .20 .90 .10 .49 .09

Overall .35 .14 .23 .18 .68 .11 .85 .08


Table C4 Mean Goodman-Kruskal Gamma Correlations Between Recall and JOLs of Experiment 1D as aFunction of Timing of JOLs, Item Relatedness, and Study Duration

Item relatedness

Timing of JOLs



OverallStudy

duration: 2 sStudy

duration: 8 sStudy

duration: 2 sStudy

duration: 8 s



Overall .15 .10 .28 .10 .74 .09 .78 .05


( Appendix continue )



19/19

Received December 10, 2003Revision received April 19, 2005

Accepted April 26, 2005

Table C5 Mean Goodman-Kruskal Gamma Correlations Between Recall and JOLs of Experiment 2 as aFunction of Timing of JOLs, Item Relatedness, and Number of Study Presentations

Item relatedness

Timing of JOLs

Immediate

( M .36, SEM .08)

Delayed

( M .83, SEM .04)

OverallOne study




presentations



Overall .39 .13 .32 .13 .76 .10 .89 .05


Table C6

Complete 2 2 2 (i.e., Timing of JOLs, Extrinsic Cue, and Intrinsic Cue) ANOVAs of Goodman-Kruskal Gamma Correlations Between Recall and JOLs

Independent variable

Experiment 1A Experiment 1B

F (1, 12) MSE p ES F (1, 19) MSE p ES

T 14.70 .22 .01 .55 44.06 .21 .001 .70E 1 1I 1 9.40 .16 .01 .33T E 1 1T I 1 4.53 .34 .05 .19E I 1.05 .28 .33 1T E I 5.30 .22 .05 .31 1

Experiment 1C Experiment 1D

F (1, 9) MSE p ES F (1, 18) MSE p ES

T 18.95 .24 .01 .68 27.73 .41 .001 .61E 1 1.01 .28 .33I 1 2.30 .37 .15T E 1.54 .27 .25 1T I 1 1E I 6.42 .25 .05 .42 1T E I 1 1

Experiment 2

F (1, 14) MSE p ES

T 28.14 .24 .001 .67E 1

I 3.38 .21 .09T E 1T I 2.85 .36 .11E I 2.52 .21 .14T E I 1

Note. Effect size (ES) is reported only when the F value was significant. JOLs judgments of learning;ANOVAs analyses of variance; T timing of JOLs (immediate vs. delayed JOLs in all experiments); Eextrinsic cue (number of study presentations in Experiments 1A, 1B, and 2: one vs. two presentations; studyduration in Experiments 1C and 1D: short vs. long duration); I intrinsic cue (item difficulty in Experiments1A and 1C: easy vs. difficult items; item relatedness in Experiments 1B, 1D, and 2: related vs. unrelated items).

326 JANG AND NELSON

judgments of learning and recall

Documents