2

Table of Contents

Introduction ................................................................................................................................................. 3 1.1. Human Learning, Memory, and Performance ........................................................................3 1.2. The Problems with Traditional Learning Methods ..................................................................3 1.3. The Cerego Method Approach ...............................................................................................4 2. Learning: The Learn Module ........................................................................................................... 6 2.1. Encoding.................................................................................................................................6 2.2. Building Memory Strength ......................................................................................................7

Active recall ...........................................................................................................................7 Optimal interval between the cue and the response .............................................................7 Attention ................................................................................................................................8

2.3. Reaching a High performance Level ......................................................................................8 Automaticity...........................................................................................................................8 Over-learning.........................................................................................................................9

3. Memory: The Review Module........................................................................................................ 10 3.1. Forgetting .............................................................................................................................10 3.2. Rebuilding Strength ..............................................................................................................11 3.3. Maintaining a High Retention Level ......................................................................................11

The spacing effect...............................................................................................................11 Expanded rehearsal series..................................................................................................12 Adapting to changing forgetting rates..................................................................................13

4. Performance: The Test Module..................................................................................................... 14 4.1. A Recall and a Recognition Test ..........................................................................................14 4.2. Signal detection theory as a model of recognition memory..................................................14 4.3. Measuring Performance .......................................................................................................15 5. Workload Management: The Schedule Module .......................................................................... 16 5.1. Balancing Memory Strength and Activation..........................................................................16 5.2. Smoothing Workload............................................................................................................17 6. Managing time ................................................................................................................................ 18 6.1. Metacognition .......................................................................................................................18

Cognitive illusions................................................................................................................18 Teaching the student about metacognitive skills.................................................................18

6.2. Cognitive Workload ..............................................................................................................19 7. Conclusion...................................................................................................................................... 20 8. Glossary .......................................................................................................................................... 22 9. References ...................................................................................................................................... 24

3

Introduction

1.1. Human Learning, Memory, and Performance

The recent explosion of research on learning and memory has elevated the understanding of human memory to an unprecedented level. However, surprisingly, this extensive research has yielded only very few practical applications to improve learning and memory (Gruneberg 1988).

Research in human learning and memory stems mainly from neuroscience and three branches of psychology: behavioral psychology, the study of our reactions to stimuli; cognitive psychology, the study of the complex mental processes responsible for behavior; and cognitive neuroscience, the study of the bases of cognition and memory at the cellular level. For historical and practical reasons, researchers have divided the study of learning and memory into three components (Anderson 1994): learning, memory, and performance. Learning is the process of encoding information. Memory refers to the storage, or retention, of that information. Performance is the measure of the success of retrieval of information from memory when needed.1

At first glance, it might be argued that human memory operates like a computer: both the brain and the computer encode, store, and retrieve information effectively and efficiently. However in contrast to the computer, the brain has several unique adaptive properties that make it exquisitely attuned to its environment. These include, among others:

Less important memories must be forgotten in order to make more important memories available when needed (Bjork 1988). Note that a corollary of this principle is that although human memory is magnificently powerful, it is intricately fragile.

It has been known since the beginning of the century (Ebbinghaus 1913) that human memories follow very specific relationships to frequency (the more times an item is presented the better it is encoded); recency (memories decay with the passage of time, and forgetting is a function of the initial amount of learning); and pattern of prior exposures (for the same amount of study time, an item is better retained if the times of study are distributed rather than close together)i.

The act of retrieving an item from memory facilitates subsequent retrieval access of that item. Rather then being left in the state it was before being recalled, the actively recalled item becomes more recallable in the future than it would have been without being accessed (Bjork 1988).

Finally, one truly unique and powerful characteristic of human memory is its knowledge of its own knowledge (Tulving and Madigan 1970). This characteristic, called metacognition, is the system that monitors and controls learning, memory, and performance (Nelson and Narens 1990).

1.2. The Problems with Traditional Learning Methods

Traditional learning or teaching methods usually fail to achieve a high effectiveness and efficiency of learning, memory, and performance because they do not take the biological characteristics of the students into account.

First students are often only trained to a level of recognition. This is sufficient to score well on recognition test (such as multiple-choice test), but insufficient for real world usage, where high-level performance is required.

Second, cramming, a sudden burst of studying immediately before an exam following a long period of neglect, is the rule rather than the exception in traditional education. Cramming is known

1 Note that even if this distinction in three components has both historical and practical justifications, it is often not possible to study only one component at a time. Any experiment involves initial acquisition followed by some minimal retention interval, followed by a test that requires the retrieval of information.

4

to produce good short-term performance but extremely poor long-term retention of knowledge (Dempster 1988).

Third, traditional learning methods rely on a one-size-fits-all model, because accommodating instruction to student differences creates serious management problems (Slavin 1986). However, it is widely recognized that everyone learns at different rates.

Fourth, because their metacognitive skills are often not adequate, students scheduling and planning is often inadequate: this poor knowledge assessment can, for instance, result in termination of study time before learning is complete, or conversely, result in unnecessary over-studying (Nelson and Leonesio 1988).

Finally, in self-paced studies, besides learning, the students must think about scheduling and planning. By diverting some of their cognitive workloadthe total of mental work that students produce during a study periodaway from learning, students cannot devote all their effort to learning.

1.3. The Cerego Method Approach

As seen above, common practice and intuition of the students are often poor guides to achieving high performance and long-term retention of knowledge. The body of research derived from behavioral psychology, cognitive psychology, and cognitive neuroscience provides a far better guide to learning. This paper describes a new method based on well-established scientific principles on which the brain operates: the Cerego Learning, Memory, and Performance Method (hereafter the Cerego Method). This method interactively and adaptively maximizes the effectiveness and efficiency of learning, memory, and performance of declarative knowledgeexplicit knowledge that we can report and of which we are consciously aware.2 The Cerego Method, shows remarkable success ensuring that the students acquire knowledge rapidly, maintain this knowledge to a high degree of retention, and retrieves knowledge quickly and effortlessly in real world situation.

The Cerego Method is embodied within three interconnected modules, the Learn Module, the Review Module, and the Test Module (see Fig. 1), whose operations are controlled by a fourth module: Schedule.

Figure 1. The Cerego learning, memory, and performance method contains three interconnected modules, the Learn Module, the Review Module, and the Test Module. The Schedule Module manages the timing of these modules.

First, the Learn Module paces the presentation of new cue and response pairs (a cue and a response together form an item), organized in lessons, for maximal encoding. As it is a primary requirement for developing memory, the Learn Module promotes active recall the students proactively construct a

2 Future versions of the method will allow learning of skills.

5

response to a presented cue. Because encoding is a function of the number of attended presentations, sensitization captures the subjects attention. Encoding is further maximized in two ways: by selecting the appropriate inter-stimulus intervals between cue and response presentation and by promoting a high degree of initial learning. Further, each item is over-learned; this leads to automatic responsesretrieval becomes extremely rapid and requires very little conscious effort.

Second, the Review Module maintains a high level of retention by scheduling rehearsal of previously learned items, according to an optimal schedule adaptive to the students and to the item difficulty. The Review Module enhances memory consolidation the initial period of time following learning when information in a relatively transient state is transformed to a more permanent, retrievable state. Further, the Review Module combines the spacing effect principle, which states that for a given amount of study time spaced presentations yield substantially better retention than massed presentations, with retrieval practice effect, which states that the act of retrieving an item from memory facilitates subsequent retrieval access of that item. These expanded rehearsal series, a pattern of increasing intervals between successive rehearsal sessions, maximizes long-term retention.

Third, the Test Module provides additional review opportunities and assesses the students current knowledge. The Test Module also measures the feeling of knowing (a metacognitive judgment made during or after knowledge acquisition as to whether a given item will be remembered on a subsequent recognition test) and confidence to assess performance the ability to use the knowledge in real world tasks.

Fourth, the Schedule Module optimally manages the students workload by controlling the scheduling of the Learn, Review, and Test modules. It does so by using workload management, which optimally prepares the students for the period when the items will be needed (such as for a test) by selecting what to study, when to study and for how long to study. Further, the Schedule Module ensures that the students motivation stays high. Workload smoothing ensures that the amount of material to be learned, reviewed or tested is about equal at each study session and graceful degradation evenly distributes the workload among learning days, even if the students change their work schedule unexpectedly.

Another advantage of the Cerego Method is that it reduces inaccurate metacognitive judgments. The Cerego Method does so by optimally scheduling the presentation of items in the Learn Module, the Review Module, and the Test Module based on continuous, objective assessment of the students performance. Thus, the students do not have to make metacognitive judgments, and hence are not prone to cognitive illusions. Because this relieves the student from the metacognitive monitoring and control tasks, studying is more effective and requires less effort. Further, the Cerego Method trains the students to become more effective learners by improving the metacognitive skills required for self-paced learning.

Finally, learning to the level of automaticity, expanded rehearsal series, and metacognitive judgment monitoring are all very demanding in terms of time management. In self-paced studies, by diverting some of the cognitive workload away from learning for scheduling and planning, students become distracted from learning. The Cerego Method helps the students to devote all their available cognitive workload to learning by optimally scheduling the presentation of items in the Learn Module, the Review Module, and the Test Module to reduce cognitive workload.

The Cerego Method is both platform independentit can be implemented on a variety of platforms with computing power, such as desktop computers, and mobile devices including handheld electronic organizers and telephonesand content independentit can accommodate a variety of text, picture or sound based material such as foreign languages, financial or medical terminology, pictures and names of most-wanted criminals, etc. The Method is embodied within three interconnected modulesLearn, Review, and Test. A fourth module, Schedule, controls the operations of these three modules (see Fig. 1). This paper will describe the Cerego Method in great detail. For each module, a chapter will describe the scientific principles and then the algorithms that implement them. Then a chapter will describe how the four modules of the Cerego Method, by managing the time of study, review, and test, reduces students cognitive illusions, train them to become better learners and helps them to devote all their time to learning instead of planning and scheduling.

6

2. Learning: The Learn Module

2.1. Encoding

Permanent memory can be conceived of as consisting of multiple memory recordsunits in which memories are encoded (Anderson 1994). Encoding is the process of creating a long-term memory record to store an experience. Strength, which is developed over time and increases with repeated practice, indicates the degree to which cues can activate the memory record. A memory record is activated when its associated cue is in the environment (or in short-term memory). Thus, activation, which is increased with recent practice, refers to the availability of the record. For example, the memory record of a childrens song rehearsed many times in kindergarten, but not since, has high strength, but low activation. On the contrary, the memory record of a person just met at a conference has high activation, but relatively low strength.

Many forms of encoding that happens in a classroom, including the learning of paired-associates such as foreign language vocabulary, can be seen as a form of learning called operant conditioning (Anderson 1994). In operant conditioning, an organism (an animal, or a student, for instance!) learns three-term associations: it learns that a response to a particular stimulus situation will be followed by a reinforcement event. In animal experiments, a cue such as a light is presented. If the animal presses a bar, it receives a reward such as a piece of food. In the classroom, suppose that a student wishes to learn that the Spanish word for dog is perro. The stimulus can be thought of as dog and the response perro, and the reinforcement may be the teachers approval.

Memory strength can be measured by any of three dependent measures of memory. First, probability of recall is a measure of the probability that students will be able to construct the correct response to a presented cue. Second, latency of recall is a measure of the time that students require to construct a response to a presented cue. Third, savings in relearning is a measure of the amount of time that students need to relearn an item to the same level attained in the initial learning session.3 When plotted as a function of practice, the memory strength, as measured by any of the above properties (probability of recall, latency of recall, savings in relearning), is a negatively accelerated curve, known as a power function (see Fig. 3). Thus, learning never stops (practice makes perfect), but, because benefits are smaller and smaller, the power function of learning is a law of diminishing returns.

Because all learning curves are power functions, the power function of learning is probably directly rooted in the neural processes that underlie learning. Long-term potentiation, one of the most studied phenomena of Neuroscience, is a form of synaptic plasticity that occurs in the hippocampus, a brain region implicated with the formation of memories.4 When pathways in the hippocampus are stimulated with high frequency electric stimuli, the synaptic strengths are lastingly increasedhence the term long-term potentiation. This synaptic potentiation in turn increases the sensitivity of neurons along that pathway to further simulations. And remarkably, the percentage of sensitivity increase, which diminishes with the number of stimulations, also follows a power function (Barnes 1979). The Cerego Method in general, and the Learn Module in particular, strive to provide the optimal stimuli for memory formation.

3 Dependent means that, to some extent, one measure can predict another.

4 Synaptic plasticity is a modification of the strength of synapses, the location at which the axon of one neuron makes a contact

with another neuron.

7

2.2. Building Memory Strength

Active recall

It has been known since Gates (1917) that generating active responses improves future performance. This form of recall, called active recall, is a primary requirement for developing memory (Skinner 1958). Active recall can be contrasted with passive recall, in which students read the question, then read the answer; this results is little memory strengthening. Thus, to increase memory strength, the students must work to retrieve the answer. Active recall not only increases memory strength, it also enhance the retrieval process. Retrieval is perhaps the most critical aspect of learning and memory in that often information can be in memory and yet not retrieved. Indeed, much of memory failure can be attributed to loss of access to appropriate retrieval cues (Shiffrin 1970). The retrieval practice effect, the act of retrieving an item from memory, has been shown to facilitate subsequent retrieval access of that item (Bjork 1975). Rather then being left in the state it was before being recalled, the actively recalled item becomes more recallable in the future than it would have been without being accessed. Further, because active recall, unlike passive recall, mirrors the condition of future use, memory is enhanced. This principle, called encoding for retrieval, states that if conditions during retrieval are the same as conditions during learning, retrieval is facilitated: the item can be more easily recalled (Morris et al. 1977).

In the Learn Module, the students learn one lesson at a time, which typically comprises a dozen items. The student must first actively recall the answer to an unknown cue. If the response matches the actively recalled answer, the item is stored as a known item. If the last item of a lesson has been learned, the process progresses to a short review, called Quick Review (see end of this chapter). Note that the Learn Module makes use of a powerful variant of the retrieval practice effect, called the expanded rehearsal series (see below in the chapter describing the Review module): the unknown items (U1) are interspersed with known items (Kn) in ever growing numbers. That is, for one unknown item U1 the Learn Module generates sequences such as U1 K1 U1 K2 K3 U1 K4 K5 U1 and so forth, until the students indicate that they know Ui.

Optimal interval between the cue and the response

Law of contiguity: The strength of association between cue and response is maximal for short intervals between cue and response

Scientific studies have shown that the timing between the cue and the response can have a dramatic effect on memory strengthening. For maximal encoding the cue must precede the response by about 300 msec to about 750 msec (Smith et al. 1969). 5 Figure 2 shows how the memory strength is maximal for cue-response intervals around 500 msec and weaker for both shorter and longer cue-response intervals. In the Learn Module of the Cerego Method, the cue precedes the response by about 500 msec. There are similar concerns regarding perception time and active recall time that are also addressed by the Cerego Method.

5 Note that this result comes from research in classical conditioning. In this form or learning, subjects display a conditioned

response (e.g. salivation) to a neutral conditioned stimulus (e.g. a bell) that is repeatedly paired with a biologically significant unconditioned stimulus (e.g. food) that evoked an unconditioned response (e.g. salivation). However, operant and classical conditioning share many similar behavioral properties, including timing properties (Anderson 1994).

8

Figure 2: Cue-response presentation intervals around 500 msec ensure maximal memory strength.

Attention

Attention the act of concentrating mentally by focusing on certain aspects of current experience to the exclusion of all others and vigilance sustained attention are crucial for encoding. Unless the subject attends to the material to be learned, virtually no information can be stored in long-term memory, even if the unattended material is repeated over and over (Moray 1959). Thus, key to learning is the number of attended presentations. This principle explains why there is little or no benefit to students writing out material to be learned, over and over. This is also why people find it difficult to draw a picture of their watch face, even though they have looked at it a thousand times.

To capture the students attention and maintain vigilance, the Cerego Method strives to enhance sensitization and reduce habituation. Sensitization is an increase in response as a result of stimulation. Habituation is a decrease in response as a result of repeated stimulation. If a car alarm suddenly blares, it initially captures the subjects attention. They are sensitized to it. However, if that car alarm just keeps blaring over and over, subjects do not notice it anymore. They become habituated to it. The Learn Module varies the timing between the cue and the response to avoid habituation or disengagement from attending to the items to be learned. This variation in timing between the cue and the response should not, however, be so obvious that it becomes a distraction in itself; it must be subtle. Thus, the timing varies at slightly less than the just noticeable difference threshold. The change is enough to maintain sensitization, but not so much that the students attention is diverted to the change itself. Furthermore, two other means are used to reduce habituation in the Learn Module. First, the sequence of items to be learned is varied. Second, declining attention is recaptured through the use of an obligatory attention cue, such as a tone or a variation in presentation.

2.3. Reaching a High performance Level

Automaticity

As measured by memory strength, we can distinguish four levels of learning: familiarity, recognition, recall, and automaticity (Fig. 3). At the level of familiarity, the students have the feeling that they knew the item at one time, but can no longer remember. At the level of recognition, the students can identify the correct answer from other distractors. That is, the students answer correctly on a multiple-choice test. At the level of recall, the students spend significant cognitive effort to attempt to remember an item

9

previously learned. Recall is measured by the number of the correct answer on a fill in the blank test. Finally, at the level of automaticity, the students retrieve the response instantly, with little or no cognitive effort. The response is known, not remembered. Automaticity is measured by a test of recall in which latency of response is the key variable.

People have a limited capacity for information processing. When information can be accessed automatically, relatively little of that capacity is used. Therefore ample capacity is left to focus on higher order cognitive skills (Carnine 1989). Because real world performance must be fast and effortless, the Cerego Method teaches to the level of automaticity.

Figure 3: The power function of learning, the power function of forgetting and the four levels of learning. The power function of learning is a law of diminishing returns. Note that the time scale for the forgetting function is typically much larger than that for the learning function.

Over-learning

In order to achieve automaticity on a particular item, it is desirable to over-learn this item. Indeed, it has been shown that learning continues to improve with practice, even after recall is perfect (Pirolli and Anderson 1985). Anderson (1981) showed that even after students reach a perfect score on the probability of recall, further practice reduces the latency of response, hence increasing the level of automaticity.6

Reaching the level of automaticity has the additional advantage of increasing long-term retention. Indeed, the effect of more practice is to shift the forgetting function (see below in Memory: the Review Module for a definition) upwards (Wixted and Ebbesen 1991). Thus, as seen in Fig, 4, materials at different degree of learning are forgotten at the same rate. Consequently, the better the students initially learn an item, the better the students remember this item after a fixed period of time (see Fig. 4). An item learned at the level of automaticity (100% correct on a test of recall, with low latency of response and low cognitive effort) will be better remembered than an item learned to a level of recall (100% correct on a test of recall).

6 Note that the three measures of memory are not dependent at the automaticity level: the latency of response decreases with over-

learning, but the probability of recall remains at 100%.

10

Figure 4: Initial degree of learning and automaticity. The higher the initial degree of learning is, the more the material is remembered at a later date. Thus for high level of retention, it is desirable to initially learn at the level of automaticity.

After the initial encoding of a lesson is completed, the Learn Module proceeds to Quick Review. By providing the students one or more opportunities to review difficult items, the Learn Module ensures that the number of presentations for each item is greater than the minimum necessary for recall. Thus, as shown in Fig. 3, the items will be learned to the level of automaticity. Another function of Quick Review is to strengthen the backward association. That is, in the initial phase of the Learn module, students learn the association A-B and they learn the association B-A in Quick Review. It has been shown that after encoding A-B, the backward association is also encoded to some extent, but not as strongly as the forward association (Wolford 1971). But because real world applications (such as learning a foreign language) require learning both A-B and B-A association, Quick Review allows strengthening of the bi-directional encoding.

Items presented during Quick Review are sorted using the drop out method. That is, when the students indicate they are able to actively recall the response, the item is dropped out of the list of remaining items. The remaining items are then re-ordered and lesser-known items are presented again. This process continues until no items remain. Thus the students spend more time on difficult items and less time on easier items.

3. Memory: The Review Module

3.1. Forgetting

Cognitive Neuroscience studies show that, in the initial period following learning, process oriented changes take place at the cellular level of the brain. If additional changes result in actual structural modifications of the brain cells then memories are consolidated (Dudai, 1996). Thus, during consolidation, information in a relatively transient state is transformed to a more permanent, retrievable state. Because consolidation often does not occur, the initial period of time following learning is when most memories are forgotten. Over-learning, however, promotes memory consolidation: items initially over-learned to a level of automaticity are more likely to survive this initial fragile period.

It is a common observation that memory seems to fade with the passage of time. Fig. 3 shows a graph of memory strength versus time that indicates how human memory decays over time (Ebbinghaus, 1913). This graph is called the forgetting curve because the distance between the curve and the maximum memory strength represents the amount of forgotten material. Conversely, the graph of Fig. 3 is also

11

referred to as the retention function because the vertical distance between the curve and horizontal line marking the minimum memory strength represents the amount of previously learned material that has been retained or remembered. Like the power law of learning, memory decay with time is also a power function: the power law of forgetting. Thus, material is forgotten quickly initially, but the rate at which material is forgotten slows over time.

Although forgetting might simply appear as if it is simply a function of time, it is not so. The amount and rate of forgetting can vary dramatically with what is learned before and after the key material (Wickelgren 1976). This phenomenon is known as interference, which refers to a negative relationship between the learning of two sets of material: the first material can affect the material learned later and vice versa. In negative transfer, the learning of earlier material impairs the learning of later material. In proactive interference, the learning of earlier material causes the forgetting of later material. In retroactive interference, the learning of later material causes the forgetting of earlier material.

3.2. Rebuilding Strength

The purpose of the Review Module is to ensure that the students re-attain and maintain a high-memory strength for previously learned items. This section discusses how memory records are re-strengthened by the Review Module and in the next section discusses how the Review Module strives to achieve long-term retention.

When an item is learned for the first time with the Learn Module, the item is scheduled for review according to a schedule curve described below. In each review session, the students attempt to actively recall the answer for each item. Then the students rate the quality of their response, after comparing their response to the correct response. Based on this rating, an item will either stay in the review group (and one or more presentations will follow) or be dropped out of the review group. A high rating will results in the item being dropped from the session. A low rating will carry the item over to the next round for further review.7

The Review Module strives to rebuild memory strength in all three dependent variables of memory strength: the probability of recall, latency of recall, and savings in relearning. The quality of response rating is a direct measure of the probability of recall. From the second round on, the latency of response influences whether or not to review the item further. If the students can quickly actively recall the answer, the rating is used to determine whether the item will go to another round of review. If it is not recalled quickly, the item will be reviewed anyway, independently of the rating. Only a short latency of response and a high rating score will result in the item being dropped out. Finally, the number of review rounds is a direct measure of the savings in relearning measure of memory strength: the more rounds, the more relearning was required. Note that to successfully terminate a round, the students must quickly recall the correct responses: thus at the end of a review session, like at the end of a learn sessions, all the reviewed items are known at the level of automaticity.

3.3. Maintaining a High Retention Level

The spacing effect

Although it is true that learning takes time and effort, some ways of expending effort are more profitable than others. In particular, it has been known since Ebbinghaus (1913) that for a given amount of study time, spaced study yield substantially better retention than massed presentation. This phenomenon is

7 Note the difference between this rating and a judgment of learning rating. The present rating is objective: the student compares

the response he actively recalled to the real response. Instead judgment of learning judgments is subjective.

12

called the spacing effect, which is ... one of the most remarkable phenomena to emerge from the laboratory research on learning. (Dempster 1988).

The key point is that the steepness of the forgetting function is much lower with distributed practice than with massed practice. Although subjects perform better in a massed practice condition when given a retention test immediately, the subjects then forget much quicker than in the distributed condition (Keppel 1964). This explains why cramming, produces good short-term performance, but extremely poor long-term retention of knowledge. Cramming induces strong memory activation, but very little memory strength. Thus, just as learning mounts to a peak just before the examination, so too does it evaporate rapidly afterward. Following the examination, forgetting is precipitous. Thus, students are rewarded for a behavior that produces short-term success but long-term failure.

Expanded rehearsal series

Landauer and Bjork (1978) combined both the principle of the retrieval practice effect (see Learn Module) and the spacing effect and proposed the expanded rehearsal strategy. This strategy, superior to simple distributed practice alone, advocates that given a fixed number of rehearsal sessions in a fixed period, a pattern of increasing intervals between successive rehearsal sessions produces optimal long-term retention.

The Review Module of the Cerego Method uses expanded rehearsal series to maintain long lasting retention.8 During the initial encoding of an item with the Learn Module and Quick Review, memory strength increases from zero to a high level that correspond to a level of automaticity (see the first climb of the trace of Fig. 5).9 Relatively quick forgetting follows. The first review session, scheduled when the Review Module predicts that the memory strength for this particular item has decreased to the minimum desired retention level, brings back the memory strength to a level of automaticity (see above). Although memory strength decays once again after this first review, this time, however, because of the spacing effect principle, the power function is less steep than previously (see Fig. 5). An item studied twice is forgotten more slowly than an item studied once. After another review, scheduled when the memory has decreased to the minimum retention level, the decay is even smaller. Eventually, review sessions are so far apart in time that the item has entered a state of permanent storage, or permastore (Bahrick 1984). The item reviewed with this expanded rehearsal series is then known for the lifetime of the learner. Note how the Cerego Method makes use if the nature of human memory described in the introduction: it uses mathematical models both of human memory decay (the power function of forgetting) and of the spacing effect (changes in the decay of the power function) to predict the timing of the next reviews.

8 As mentioned above, the Learn Module also uses expanded rehearsal series, albeit in a much shorter time frame (minutes) than

the Review Module (days). This intra-trial spacing effect is created by adding known items (Kn) in ever growing numbers (drawn randomly from the known item pool) each time an the unknown item (U1) is presented for encoding. That is, for one unknown item U1 the Learn Module generates sequences such as U1 K1 U1 K2 K3 U1 K4 K5 U1 and so forth, until the students indicate that they know Ui.

9 This increase to maximal memory strength, as well as future increase back to maximal memory strength, appear in the figure to

be a linear increase, but as described earlier, it is not so: it actually follows the power function of learning. It appears linear because the time scale of encoding (typically a few minutes) is much shorter than the time scale of forgetting (typically a few hours or days) that the increase appears to be a straight almost vertical line.

13

Figure 5: The spacing effect and expanded rehearsal series.

Because of prior history of study, subjective or real difficulty, some items are harder or easier to memorize than others. Thus, the schedule for review should not be identical for all items. As shown in Figure 6, easy items should be reviewed less often than difficult items. For this purpose, the Cerego Method uses adaptive review schedules.

Adapting to changing forgetting rates

Forgetting for specific items does not always follow a predetermined forgetting power function, however. Because of proactive and/or retroactive interferences, or failures of consolidation or retrieval, some items that may have seemed initially easy to learn could be hard to remember a few days later, or vice versa. Thus, a review curve that accurately models the forgetting rate of a particular item for a particular learner early in the Review schedule may become inaccurate at some later date. The Cerego Method accounts for this effect by allowing the items to be reviewed to move from one review schedule to another, as shown in Fig. 6.

Hopping between schedules, as shown in Figure 6, is determined by using the students response rating (see above). Depending on the number of rounds when a high rating is given, the item might move to a curve that is easier than the current curve (an easier curve is one where Review sessions occur less frequently), more difficult than the current curve, or not move at all. If an item has been presented several times and a quality rating is consistently low, the item is treated as unlearned, and the whole process of Review begins all over again.

Figure 6. Review schedule curves, with hopping between curves.

14

4. Performance: The Test Module

4.1. A Recall and a Recognition Test

The Test Module actually includes two types of test. First, as in the Learn and Review modules, there is a (cued) recall test: the students must come up with the right answer to a cue. In the recall test the student must retrieve a response to the presented cue. Then a test of recognition follows. In the recognition test, the students must choose the correct response mixed with multiple distractors (such as in a typical multiple choice examination). Unlike the common use of tests that are solely use for assessment and motivation, in the Cerego Method, tests have multiple functions:

First, tests are given as an alternative form of review to build strength. Note that in the method, assessment per se occurs during Learn, Review, and Test. As discussed above (see Learning: the Learn Module), active recall increases memory strength. Thus, as in the Learn module and the Review module, the Test Module first includes a recall test to enhance memory strength.

Second, tests give an unbiased assessment of performance (see below). Third, the tests provide a test of knowledge. The students can take blank test to see how they

would score on a real test. Fourth, it allows a different form of retrieval practice; it promotes encoding for retrieval in tests,

because it closely mirrors the conditions of an actual multiple-choice test. Fifth, tests provide metacognitive feedback to the user so that they become better learners. Finally, tests enhance motivation. Motivation refers to those factors that increase or decrease the

effort of an individual activity. Motivation has four components (Keller 1979): Relevance to goal, attention, satisfaction, and self-confidence. As seen above, attention is increased when something unexpected occurs in a persons environment. A Test schedule unexpectedly scheduled promotes attention as it breaks the cycle of Learn and Reviews. Satisfaction is ensured when the rewards are consistent with the students expectations. A Test can promote satisfaction if the items are chosen such that the students can score at a high level. Providing feedback that supports the students progress increases self-confidence.

4.2. Signal detection theory as a model of recognition memory

One usually measures performance by the percentage of correct response in recognition tests. When presented with distractors, the students may recognize all the items in the list. But, if the students also claim to recognize the distractors, they are clearly guessing and should not be given credit for high memory strength. Thus, the percentage of correct responses on a recognition test is usually a poor indicator of memory. Then, how can one measure how good a subjects memory truly is? Signal Detection Theory, a branch of psychophysics, which was originally developed in sensory psychology, provides an answer. Signal Detection Theory is based on the phenomenon that a living organism such as a human or other animal, perceives stimuli and makes decisions based upon those perceptions. This two-part process is relevant to many memory related tasks.

As shown in Figure 7, after learning, the targets, or correct responses to cues, have stronger memory strengths than the distracters, the wrong responses to the cue. Note that as seen in the Figure, one of the assumptions of signal detection theory is that the distributions of correct responses and distracters follow bell-shaped, Gaussian distributions. The distance between the means of the distributions of targets and the distracters is measured by d (d-prime), which is an indicator of pure memory strength. The more the students study, the stronger the memory strength, the farther the two distributions, that is, the larger d. In the signal detection paradigm, the subjects use a threshold to discriminate the target from the distracters. The criterion by which a subject discriminates a difference (or fails to discriminate a difference) is known as (beta). If the subject is extremely lax in their criteria for discriminating the correct answer, that is use

15

a lot of guessing, shifts to the left of the graph of Fig. 7. If the subject is extremely cautious in their criteria for discriminating the correct answer, that is if the subject prefers not answering instead of making a mistake, shifts to the right of the graph of Fig. 7.

Fig. 7. The signal detection theory model of recognition memory.

When there is overlap between the target and the distracters, the position of on the graph of Fig. 7 creates a possibility of four outcomes: correct recognitions, false alarms, false rejections and correct rejections. While memory strength depends only on d, performance depends on both d and , and only an optimal and a high d will yield good performance. Failures to output studied items can occur because the distribution of studied items and distractors are insufficiently separated (d is too small), or because although the distributions are separated, the subject chooses to place his or her decision criterion too high (high ), i.e. so as to withhold items that have not been studied.

4.3. Measuring Performance

The Test Module of the Cerego method monitors both the correctness of the students response (percentage of correct responses) and the students performance, which is the ability to evaluate accurately whether they know the correct response and the incorrect responses. For this purpose, the Test Module uses the metacognitive concepts of feeling of knowing, which is, as defined by Koriat (1994), a judgment as whether an item will remembered on a subsequent recognition test.

After their attempt at actively recalling the answer, the students must determine their beliefs, related to in the signal detection framework, that they will remember the item on a future test. The recognition test is then presented: the students must choose the correct response mixed with four of distractors (such as in a typical multiple choice examination). The students belief is then compared to the actual answer. As discussed above in the signal detection theory framework, four outcomes are possible, as shown in Figure 8, which shows a signal detection theory matrix.

A. The students believed that they have retrieved a response that is correct in the recall test, and they are indeed correct in the recognition test--- a correct recognition. B. The students believed that they do not know the answer in the recall test, but they correctly identify it in the recognition test by chance --- a false rejection. C. The students believed that they do not know the answer in the recall test, and they fail to identify it in the recognition test--- a correct rejection. D. The students believed that they have retrieved a response that is correct in the recall test, but the chosen response is incorrect --- a false alarm.

16

The Test Module can then compute measures that assess the students memory strength. The students score is the number of correct recognitions and the total of false recognitions divided by the total number of answers. But as discussed above, raw score is not a good indicator of performance as the students could have guessed the answers. The students performance is computed by recording the number of correct rejections plus the number of correct recognitions divided by the total number of answers. The students overconfidence is the number of false alarms divided by the total number of answers. And finally, the students underconfidence is the number of false rejections divided by the total number of answers.

Figure 8. To monitor the subjects performance, the Test Module monitors both the correctness of the students response and the students belief. The students are given a high performance only if they make correct recognitions and correct rejections.

5. Workload Management: The Schedule Module

5.1. Balancing Memory Strength and Activation

The Schedule Module builds both high strength and high activation by a predetermined end of study date. Suppose for instance that a student would like to learn Japanese Kanji and would like to take a Japanese proficiency test examination in three months. The student is confronted with two different, almost opposite, goals: the first is to learn and remember the Kanji and the second is to pass the exam. Thus, the student must build both lasting high strength (for good future retention) and high activation (for optimal retrieval on the test, as well as for later real world usage).

The Schedule Module strives to achieve these two goals by determining whether items are being scheduled for presentation during a Normal Zone, a Compression Zone or a Final Review Zone (see Fig. 9). The Normal zone is approximately the first 75% of time for the overall schedule. The Compression Zone is the following 20%, and the Final Review is the last few days immediately preceding the end date. In the Normal Zone, the students experience a normal schedule of Learn, Review, and Test. There is enough time remaining before the test date to achieve the desired degree of strength. During the Compression Zone, the Schedule Module must provide more opportunities to review items than in the Normal Zone. That is, the Schedule Module treats items learned in the Compression Zone as though they were more difficult, increasing the number and type of reviews, to increase the strength of that item before the Final Review. In the Final Review Zone, no items are scheduled for Learn, and all items are reviewed to develop high levels of activation immediately preceding the end date.

17

Figure 9. Workload management. For optimal performance at, and after, the end date, the Schedule Module creates three zones: a Normal zone, a Compression zone, and a Final review zone. In the Compression zone, the items are all reviewed three times. In the Final review zone, no learning occurs and all items are reviewed.

5.2. Smoothing Workload

To maintain the students motivation, and to ensure the most effective and efficient learning, review and retrieval of items, the workload must be as uniform as possible from session to session. Suppose, for instance that the students are to learn two sets of items. They must know all the capitals of the world in the next three weeks and they must learn 1000 characters in two months. However, they are going to start learning Japanese Kanji in two weeks. In such a case, if the items are evenly distributed for both sets of material, there will be a large one-week peak in two weeks. The Schedule Module of the Cerego Method will plan for this beforehand and smooth the workload. That is, it will re-compute the amount and timing of learning, reviewing and testing that must be conducted in the future such that the number of items is roughly constant during the whole study period.

Because the students might skip one or more study sessions, or get ahead of schedule, or change the date of the test, the Schedule Module also takes into account the actual use of the system. If, for instance, the students skip a day, and the items that were due to be learned on this day were just moved to the following day, the workload for the next day would double in size. This large workload might reduce the students motivation. To avoid this, the Schedule Module redistributes those items among the following days, so that they have about the same workload. This process, called graceful degradation, is illustrated in Fig. 10: the missed items are distributed on the four following days.

Note that before redistributing the items, the system needs to reorder and rank them according to their priority in Review. To illustrate this point, lets suppose that an item has just been learned today. The spacing effect tells us that this item should be reviewed once, say tomorrow, and a second time, say in five days. The review schedule is stringent: if the students miss the review tomorrow, the item must be reviewed the day after tomorrow. This item has high priority in the ranking. On the contrary, if an item has been reviewed a few times already, the interval between the reviews are quite long, maybe thirty days or so. The review schedule is more flexible. If the students miss the review tomorrow, the item might be reviewed in two days or in four days, it does not make much difference. This item has low priority in the ranking, and will be scheduled for review in five days.

18

Figure 10. Graceful degradation. To avoid too much workload on the day after a lesson has been missed, the items are redistributed over the few following days.

6. Managing time

6.1. Metacognition

Cognitive illusions

In order to take metacognitive control actions students use their subjective metacognitive monitoring judgments while encoding and retrieving knowledge. Examples of metacognitive control actions are allocation of study time and termination of study. While some metacognitive monitoring are good indicators of future performance (Dunlosky and Nelson, 1992), metacognitive judgments can be seriously misleading. For instance, the allocation of study time, that is whether the student should continue studying or should quit, is a metacognitive control process. It is affected by monitoring processes such as judgment of learning judgments judgments that occur during initial learning and that are attempts to predict future test performance on recently studied items. Because these judgments made soon after acquisition are poor indicator of future test performance, students do not spend the right amount of time studying. Thus, even when allowed unlimited study time to do so, students often prematurely terminate a study session before learning is actually complete. Conversely, over-studying can yield little or no increase in future performance; this is known as the "labor-in-vain effect" (Nelson and Leonesio 1988).

Thus, not only does monitoring and controlling study time take time and effort (see cognitive workload below), but also, because they are prone to cognitive illusions, students are not performing these tasks correctly. The best way to reduce inaccurate metacognitive judgment that lead to studying too little or too much for each item, is to determine the pattern, the sequence, and the timing of the presentation of the items to be learned. Because this relieves the learner from the monitoring and control tasks, studying is more effective and requires less effort. The Cerego Method does so by optimal scheduling the presentation of items in the Learn Module, the Review Module, and the Test Module.

Teaching the student about metacognitive skills

In self-paced studies, metacognitive judgments and actions should be well adjusted for high-level, long-term performance. One of the aims of the Cerego Method is to teach the students how to learn in self-paced studies, that is to become better learners. In other words, in terms of the signal detection model of

19

recognition memory, the Method not only increases d by increasing memory strength, but also helps the students to select the appropriate that yield maximal performance, even when they are not using the method.

As seen above, the Test Module of the Cerego Method monitors both the correctness of the students response (percentage of correct responses), the students performance, and the students under- and over-confidence. For this purpose, the Test Module measures the students feeling of knowing (see above). By telling the students how much they are under- or overconfident, the students may become better learners. Because the feeling of knowing affects the selection of search strategy and the termination of search (Nelson and Narens 1990), it plays a crucial role in setting , which itself, as we saw above, is crucial for good performance. Thus, the Cerego Method, not only enhances memory activation, it also trains the students to become more effective learners by teaching the metacognitive skills required for self-paced learning. When learning without the Method, the students will be able to take appropriate metacognitive control actions such as proper allocation of study time and termination of study at the right time.

6.2. Cognitive Workload

It is often believed that increases in allocated study time directly translate into an increase in performance. Increases in allocated study are relatively easy to implement, but three other time variables are key for high performance (Dempster 1987): time needed to learn, time on task or student engaged time, and distribution of time. First, the time to learn, which is highly variable, simply depends on the rate of learning of the student and, on prior knowledge, and on item difficulty. Second, the engaged study time is linked to attention. As seen above, key to encoding is the number of attended presentations. Third, the distribution of time is key to learning at the level of automaticity, expended rehearsal series, and metacognitive judgment monitoring, which are all very demanding in terms of time scheduling. A serious constraint to manage the distribution of time is that it requires the student to track each item separately. This clerical task of this magnitude is extremely time consuming and if the students had to do it rigorously it would distract them from learning, because it will use the majority of their cognitive workloadthe total of mental work that students produce during a study period. In self-paced studies, besides learning, the students must think about scheduling and planning. By diverting some of the cognitive workload away from learning, students cannot devote all of the available cognitive workload to learning.

The best way to utilize effectively the cognitive workload is to eliminate the burden of scheduling, while determining the pattern, sequencing, and timing of presentation of the items to be learned and reviewed. The Learn, Review, and Schedule Modules of the Cerego method all ensure that the students cognitive workload is optimally used for pure learning.

20

7. Conclusion

Although cognitive neuroscience, cognitive psychology and behavioral psychology are well-established scientific domains with a long history of discovery of major scientific principles, there are only scant applications of these principles to learning and education (e,g, Gruneberg 1988). The Cerego method applies these principles in a synergistic manner to interactively and adaptively maximizes the effectiveness and efficiency of learning, memory and retrieval of declarative knowledge. The new Cerego Method has five distinctive sets of features that make it by far the most powerful and sophisticated learning method to date.

1. High performance. Most existing methods seek to train or teach declarative knowledge only to a level of recognition. The Cerego Method teaches at a level of automaticity instead of just recognition to prepare students for real world performance. This creates a framework of useable knowledge that the learner can apply across a wide range of situations.

2. Long-term performance: In traditional learning and teaching methods, the students use cramming, which is known to produce extremely poor long-term performance. The Cerego Method determines the schedule of learn, review, and test schedules to constantly maintain high strength and high activation levels. In particular, by using adaptive expanded rehearsal patterns for each item in the Review Module, the Cerego method ensures that the learned items are permanently stored in permastore, i.e. knowledge is maintained at high levels of retention efficiently ultimately leading to permanent knowledge.

3. Individualized learning. Individuals learn at vastly different rates and some material is easier to learn than other, or more readily forgotten than others. Although some existing methods monitor the students progress, in these methods future learning, reviewing or testing are not modified based on the users actual performance. The Cerego method constantly monitors the memory strength for each student and for each item. Further, it ensures that each student knows each to a level of automaticity and is stored in permastore.

4. Cognitive skills training. The Cerego Method trains the students to become more effective learners by improving the skills required for self-paced learning.

5. Time management. When students schedule and manage learning, reviewing, and testing alone, they spend much time scheduling; this increases their cognitive workload and reduces the efficiency of learning, retaining and retrieving knowledge. The Cerego Method eliminates all scheduling responsibility of the students, who can concentrate on learning per se.

Numerous additional modifications and variations of the Cerego Method are in the process of development. As shown in Fig. 11, several new modules, including a Help module, are presently being implemented. A Progress module will monitor the usage tendencies of the students. A Create module will allow lesson creation directly by the students. A Connect module will enable easy download of lessons from the Internet directly to the device.

21

Figure 11. Future implementation of the method

In conclusion, the advantages of the Cerego Method are numerous; if well exploited, its impact could be dramatic. It cuts down enormously on the amount of time that it takes to learn any given item or lesson, and it considerably increases long-term retention of items studied. The end result of this rigorously scientifically based method will be the average student being able to learn very large amount of information to a high performance level, while demonstrating a high long-term retention rate even years later.

22

8. Glossary

Active recall: the students construct a response to a presented cue (see passive recall; recall). Attention: the act to concentrate mentally by focusing on certain aspects of current experience to the exclusion of all others (see vigilance). Automaticity: performance characterized by rapid response without conscious attention or effort. Automaticity can be measured by a test of recall where latency of response is the key variable (see recall, recognition, familiarity; latency of recall). Behavioral psychology: the study of the actions or reactions produced in response to external or internal stimuli. Cognitive neuroscience: the study of the cellular and molecular mechanisms associated with learning and memory. Cognitive psychology: the study of the internal mental processes. Cognitive workload: the amount of mental work that students produce in a specified time period. Consolidation: the initial period of time in memory formation when information in a relatively transient state is transformed to a more permanent, retrievable state. Confidence: a metacognitive judgment that corresponds to decisions to output answers during retrieval (see metacognition). Declarative knowledge: explicit knowledge that we can report and of which we are consciously aware Encoding for retrieval: the concept that memory performance is better when students process an item in the same way in which the item was processed during learning or study. Encoding: the process of creating a long-term memory record to store an experience (see learning, memory record). Expanded rehearsal series: a pattern of increasing intervals between successive rehearsal sessions, to produce optimal long-term retention Familiarity: information learned or remembered to the level of familiarity is information that the students have the feeling that they knew at one time, but can no longer remember (see automaticity, recall, recognition). Feeling of knowing: a metacognitive judgment made during or after knowledge acquisition as to whether a given item will be remembered on a subsequent recognition test (see metacognition). Graceful degradation: observing the usage of the system and modify the remainder of the schedule to compensate for overuse or under-use of the system by the user. Habituation: a decrease in response as a result of repeated exposures to a stimulus (see sensitization). Interference: a negative relationship between the learning of two sets of material. Item: a cue-response pair. Judgment of learning: judgments that occur during acquisition and are predictions about future test performance on recently studied items. Judgment of learning judgments made soon after acquisition is a very poor indicator of future test performance. Labor-in-vain effect: when, in self-paced study, students study beyond the point where any benefit is derived. Latency of recall: a measure of time required to construct a response to a presented cue (see automaticity, savings in relearning, probability of recall; active recall). Learning: the process by which relatively permanent changes in behavioral potential as a result of experience. Long-term Potentiation: when brief, high frequency electrical stimulations are administered in some areas of the brain such as the hippocampus, there is a long term increase in the magnitude of the response of the cells to further stimulations (see synaptic plasticity). Memory activation: the availability of an item in memory; sometimes thought in terms of neural activation (see memory strength, memory record). Memory record: abstract conception of the unit in which memories are encoded. Memory strength: a property of memory, which increases with repeated practice and is the degree to which a cue can activate a memory record (see memory activation, memory record). Memory: the storage, or retention, of information encoded during learning (see storage).

23

Metacognition: the process of monitoring and controlling mental processes, particularly those associated with learning. Negative transfer: the learning of earlier material impairs the learning of later material. Neurons: the cells in the brain that are most directly responsible for information processing in the brain. Operant conditioning: a process where a students learn three-term associations: they learn that a response to a particular stimulus situation will be followed by a reinforcement Paired-associate learning: a memory procedure in which a student learns to give a response when presented with a cue. Passive recall: the students simply observe a cue and response paired presented (see active recall; recall). Performance: the measure of the retrieval of information from memory when needed. Power function of learning: the observation that performance increases as a power function of the amount of practice. Power law of forgetting: the observation that performance decreases as a power function of the delay since training. Proactive interference: the observation that the learning of earlier material causes the forgetting of later material. Probability of recall: a measure of the probability that students will be able to construct the correct response to a presented cue (see savings in relearning, latency of recall). Recall: information learned or remembered to the level of recall is information that the students can retrieve when only a cue is presented. The students can get the answer correct on a fill in the blank test with information that has been learned to a level of recall (see automaticity, recognition, familiarity). Recognition: information learned or remembered to the level of recognition is information that the students can separate from other distracting choices or distractors. When presented with a cue, the students can choose the appropriate response from a number of alternatives such that the students can get the answer correct on a multiple choice test (see automaticity, recall, familiarity). Retrieval practice effect: the act of retrieving an item from memory facilitates subsequent retrieval access of that item. Retrieval: the process of getting access to memory. Retroactive interference: the observation that the learning of later material causes the forgetting of earlier material. Savings in relearning: a measure of memory strength calculated by measuring the amount of time necessary to relearn an item to the same level as that attained in the initial learning session (see latency of recall, probability of recall). Sensitization: increase in response as a result of repeated exposure to a stimulus (see habituation). Spacing effect: the observation that for a given amount of study time, spaced presentations yield substantially better learning results than massed presentations. Storage: the maintenance of memories after their initial encoding (see memory). Synaptic plasticity: modification of the strength of synapses, the location at which the axon of one neuron makes a contact with another neuron (see long-term potentiation). Vigilance: the process of paying close and continuous attention (see attention). Workload management: a process that optimally prepares for the period when the items will be needed (such as a test). Workload smoothing: a process that ensures that the amount of material to be learned, reviewed or tested is about equal each day.

24

9. References

Anderson, J R. (1994) Learning and Memory: An Integrated Approach. John Wiley& Sons, NY, NY. Anderson, J.R. and Schooler, L.J. (1991) Reflections of the environment in memory. Psychol. Sci, 2, 396-408 Anderson, R. (1981) Interference: The Relationship Between Response Latency and Response Accuracy. Journal of Experimental Psychology: Human Learning and Memory". 7, 326-343 Bahrick H.P. (1984) Semantic memory in permastore: fifty years of memory research for Spanish learned at school. J. Exp. Psychology, 113, 1-29 Barnes, C.A. (1979) Memory deficits associated with senescence: A neurophysiological and behavioral study in the rat. J. Comp. Physiol. 43, 74-104 Bjork R.A. (1975) Retrieval as a memory solidifier: an interpretation of negative recency and related phenomena. In RL Solso (Ed.) Information processing and cognition (123-124) Hillsdle, NJ: Erbaum Bjork R.A. (1988) Retrieval practice and the maintenance of knowledge. In M.M. Gruneberg, P.E. Morris, and R.N. Sykes (Eds) Practical aspects of memory: current research and issues John Wiley& Sons, NY, NY. Carnine D. (1989) Designing practice activities. J. Learning Disabilities, 22, 603-607 Demspter F.N. (1988) The spacing effect. A case study in the failure to apply the results of psychological research. Am. Psychol. 43, 627-634 Dudai, Y. (1996) Consolidation: fragility on the road to the engram. Neuron 17, 367-370. Ebbinghaus, H. (1913) Memory: A contribution to experimental psychology (H.A. Ruger & C.E. Bussenues, Trans.). New York: Teachers college, Columbia University. (Originally published 1885). Gates A.I. (1917) Recitation as a factor in memorizing. Arch. Phschol. 6, 40. Keppel G. (1964) Facilitation in short- and long-term retention of paired associates following distributed practice in learning. J. Verbal Learn. and Verbal Behav. 3, 91-111 Koriat A. (1994) The accessibility account. Subthreshold priming and memory monitoring. In Metcalfe J. and Shimamura A.P. Eds: Metacognition. Cambridge, MA. The MIT press. Landauer T.K. and Bjork R.A. (1978) Optimal rehearsal patterns and name learning. In M. Gruneberg, P.E. Morris and R.N. Sykes (Eds). Practical aspects of memory (pp 625-632). London: Academic press. Moray, N. (1959) Attention in dichotic listening: affective cues and the influence of instructions. Quart. J. Exp. Psychology, 11, 56-60 Morris C.D., Brandsford J.D. and Franks J.J. (1977) Levels of processing versus transfer-appropriate processing. J. Verbal Learning and Verbal Behav. 16, 519-533 Nelson T.O. and Narens L. (1990) Metamemory: a theoretical framework and new findings. In G. Bower (Ed.) The psychology of learning and motivation (Vol. 26) New York: Academic Press. Nelson T.O., Leonesio R.J. (1988) Allocation of self-paced study time and the "labor-in-vain effect". J Exp Psychol Learn Mem Cogn. 14, 676-86. Pirolli P.L. and Anderson, J.R. (1985) The role of practice in fact retrieval. Journal of Experimental Psychology: Learning memory and cognition, 11, 136-153 Shiffrin, R.M. (1970) Forgetting: Trace erosion or retrieval failure? Science, 168, 1601-1603. Skinner B.F. (1958) Teaching machines. Science 128, 969-977. Smith M.C., Coleman S.R. and Gormezano I (1969) Classical conditioning of the rabbits nictitating membrane response at backward, simultaneous and forward CS-US intervals. J. Comp. Physiol. Psychology. 69, 226-231. Tulving E. and Madigan S.A. (1970) Memory and verbal learning. In P.H. Mussen and & M.R. Rosenweig (Eds.), Annual review of psychology. Palo Alto, CA Wickelgren W.A. (1976) Memory storage dynamics. In W.K. Estes (Ed.), Handbook of learning and cognitive process. (Vol. 4) Hillsdale, NJ: Erlbaum Wixted J.T. and Ebbesen E.B. (1991) On the form of forgetting. Psychological Science, 2, 409-415. Wolford G. (1971) Function of distinct associations for paired-associate performance. Psychological review, 4, 303-313 Gruneberg M.M. (1988) Practical problems in the practical application of memory. In M.M. Gruneberg, P.E. Morris, and R.N. Sykes (Eds) Practical aspects of memory: current research and issues John Wiley& Sons, NY, NY. Slavin R. (1986) Educational psychology: theory into practice. Englewood Cliffs, NJ. (Keller 1979):

25