REPORTS

The Critical Importance

period, then testing them on it in a test period,
then presenting it again, testing on it again, and so
on. The dropout learning conditions of the present
of Retrieval for Learning experiment differed from the standard learning
condition in that, once an item was successfully
recalled once on a test, it was either (i) dropped
Jeffrey D. Karpicke1* and Henry L. Roediger III2 from study periods but still tested in one con-
dition, (ii) dropped from test periods but still re-
Learning is often considered complete when a student can produce the correct answer to a peatedly studied in a second condition, or (iii)
question. In our research, students in one condition learned foreign language vocabulary words in dropped altogether from both study and test pe-
the standard paradigm of repeated study-test trials. In three other conditions, once a student had riods in a third condition (Table 1).
correctly produced the vocabulary item, it was repeatedly studied but dropped from further testing, Surprisingly, standard learning conditions
repeatedly tested but dropped from further study, or dropped from both study and test. Repeated and dropout conditions have seldom been com-
studying after learning had no effect on delayed recall, but repeated testing produced a large pared in memory research, despite their critical
positive effect. In addition, students’ predictions of their performance were uncorrelated with importance to theories of learning and their prac-
actual performance. The results demonstrate the critical role of retrieval practice in consolidating tical importance to students (in using flash cards
learning and show that even university students seem unaware of this fact. and other study methods). Dropout conditions
were originally developed to remedy methodo-
ver since the pioneering work of Ebbinghaus memory, concerning the relation between the logical problems that arise from repeated practice

E (1), scientists have generally studied hu-

man learning and memory by presenting
speed with which something is learned and the
rate at which it is forgotten. Is speed of learning
in the standard learning condition (10), but they
can also be used to examine the effect of re-

Downloaded from http://science.sciencemag.org/ on November 4, 2019

people with information to be learned in a study correlated with long-term retention, and if so, is peated practice in its own right, as we did in the
period and testing them on it in a test period to the correlation positive (processes that promote present experiment. If learning happens exclu-
see what they retained. When this procedure oc- fast learning also slow forgetting and promote sively during study periods and if tests are neutral
curs over many trials, an exponential learning good retention) or negative (quick learning may assessments, then additional study trials should
curve is produced. The standard assumption in be superficial and produce rapid forgetting)? Early have a strong positive effect on learning, whereas
nearly all research is that learning occurs while research led to the conclusion that quick learn- additional test trials should produce no effect.
people study and encode material. Therefore, ad- ing reduced the rate of forgetting and improved Further, if repeated study or test practice after an
ditional study should increase learning. Retriev- long-term retention (7), but later critics argued item has been learned does indeed benefit long-
ing information on a test, however, is sometimes that, when forgetting is assessed more properly term retention, this would contradict the conven-
considered a relatively neutral event that mea- than in the early studies, no differences exist be- tional wisdom that students should drop material
sures the learning that occurred during study but tween forgetting rates for fast and slow learning that they have learned from further practice in
does not by itself produce learning. Over the conditions (8, 9). By any account, conditions that order to focus their effort on material they have
years, researchers have occasionally argued that exhibit equivalent learning curves should produce not yet learned. Dropping learned facts may create
learning can occur during testing (2–6). However, equivalent retention after a delay (9). the same long-term retention as occurs in stan-
the assumptions that repeated studying promotes Using foreign language vocabulary word pairs, dard conditions but in a shorter amount of time,
learning and that testing represents a neutral event we examined the contributions of repeated study or it may improve learning by allowing stu-
that merely measures learning still permeate con- and repeated testing to learning by comparing a dents to focus on items they have not yet recalled.
temporary memory research as well as contem- standard learning condition to three dropout condi- This strategy is implicitly endorsed by contem-
porary educational practice, where tests are also tions. The standard method of measuring learning, porary theories of study-time allocation (11, 12)
considered purely as assessments of knowledge. used since Ebbinghaus’s research (1), involves and is explicitly encouraged in many popular
Our goal in the present research was to ex- presenting subjects with information in a study study guides (13).
amine these long-standing assumptions regard-
ing the effects of repeated studying and repeated
testing on learning. Specifically, once informa- Table 1. Conditions used in the experiment, average number of trials within each study or test
tion can be recalled from memory, what are the period, and total number of trials in the learning phase in each condition. SN indicates that only
effects of repeated encoding (during study trials) vocabulary pairs not recalled in the previous test period were studied in the current study period. TN
or repeated retrieval (during test trials) on learn- indicates that only pairs not recalled in the previous test period were tested in the current test
period. Students in all conditions performed a 30-s distracter task that involved verifying multi-
ing and long-term retention, assessed after a
plication problems after each study period.
week delay? A second purpose of this research
was to examine students’ assessments of their Study (S) or test (T) period and number of trials per period Total
own learning. After learning a set of materials Condition number
under repeated study or repeated test conditions, 1 2 3 4 5 6 7 8 of trials
we asked students to predict their future recall
ST S T S T S T S T
on the week-delayed final test. Our question
40 40 40 40 40 40 40 40 320
was, would students show any insight into their
own learning?
SNT S T SN T SN T SN T
A final purpose of the experiment was to
40 40 26.8 40 8.0 40 2.0 40 236.8
address another venerable issue in learning and
1
Department of Psychological Sciences, Purdue University, STN S T S TN S TN S TN
West Lafayette, IN 47907, USA. 2Department of Psychol- 40 40 40 27.9 40 11.8 40 3.3 243.0
ogy, Washington University in St. Louis, St. Louis, MO
63130, USA.
SNTN S T SN TN SN TN SN TN
*To whom correspondence should be addressed. E-mail:
40 40 27.1 27.1 8.8 8.8 1.5 1.5 154.8
[email protected]

966 15 FEBRUARY 2008 VOL 319 SCIENCE www.sciencemag.org

 REPORTS
In the experiment, we had college students a pair. We also analyzed traditional learning study period in the STN condition, their long-
learn a list of foreign language vocabulary word curves (the proportion of the total list recalled term recall was much worse than students who
pairs and manipulated whether pairs remained in in each test period) for the two conditions that were repeatedly tested on the entire list. Com-
the list (and were repeatedly practiced) or were required recall of the entire list (ST and SNT), bining the two conditions that involved repeated
dropped after the first time they were recalled, and the results by the two measurement meth- testing (ST and SNT) and combining the two
as shown in Table 1. All students began by study- ods were identical. Thus, we restrict our dis- conditions that involved dropping items from
ing a list of 40 Swahili-English word pairs (e.g., cussion to the cumulative learning curves on testing after they were recalled once (STN and
mashua-boat) in a study period and then testing which all four conditions can be compared. SNTN), repeated retrieval increased final recall
over the entire list in a test period (e.g., mashua-?). Figure 1 shows that performance was virtually by 4 standard deviations (d = 4.03). The distri-
All conditions were treated the same in the ini- perfect by the end of learning (i.e., all 40 English butions of scores in these two groups did not
tial study and test periods. Once a word pair was target words were recalled by nearly all sub- overlap: Final recall in the drop-from-testing
recalled correctly, it was treated differently in the jects). More importantly, there were no differences conditions ranged from 10% to 60%, whereas
four conditions. In the standard condition, sub- in the learning curves of the four conditions. final recall in the repeated test conditions ranged
jects studied and were tested over the entire list in Given the similarity of acquisition perform- from 63% to 95%. Whether students repeatedly
each study and test period (denoted ST). In a ance, it is not too surprising that students in the studied the entire set or whether they restudied
second condition, once a pair was recalled, it was four conditions did not differ in their aggregate only pairs they had not yet recalled produced
dropped from further study but tested in each sub- judgments of learning (their predictions of their virtually no effect on long-term retention. The
sequent test period (denoted SNT, where SN indi- future performance). On average, the students in dramatic difference shown in Fig. 2 was caused
cates that only nonrecalled pairs were restudied). all conditions predicted they would recall about by whether or not the pairs were repeatedly tested.
In a third condition, recalled pairs were dropped 50% of the pairs in 1 week. The mean number of Even though cumulative learning perform-
from further testing but studied in each subsequent words predicted to be recalled in each condition ance was identical in the four conditions, the

Downloaded from http://science.sciencemag.org/ on November 4, 2019

study period (denoted STN, where TN indicates were as follows: ST = 20.8, SNT = 20.4, STN = total number of trials (study or test) in each con-
that only nonrecalled pairs were kept in the list 22.0, and SNTN = 20.3. An analysis of variance did dition varied greatly. Table 1 shows the mean
during test periods). In a fourth condition, recalled not reveal significant differences among the number of trials in each study and test period
pairs were dropped entirely from both study and conditions (F < 1). and the total number of trials in each condition.
test periods (SNTN). The final condition repre- Although students’ cumulative learning per- The standard condition (ST) involved the most
sents what conventional wisdom and many edu- formance was equivalent in the four conditions trials (320) because all 40 items were presented
cators instruct students to do: Study something and predicted final recall was also equivalent, in each study and test period. The SNTN condi-
until it is learned (i.e., can be recalled) and then actual recall on the final delayed test differed tion involved the fewest trials (154.8, on aver-
drop it from further practice. widely across conditions, as shown in Fig. 2. age) because the number of trials in each period
At the end of the learning phase, students in The results show that testing (and not studying) grew smaller as items were recalled and dropped
all four conditions were asked to predict how is the critical factor for promoting long-term re- from further practice. The other two conditions
many of the 40 pairs they would recall on a final call. In fact, repeated study after one successful (SNT and STN) involved about the same number
test in 1 week. They were then dismissed and recall did not produce any measurable learning of trials (236.8 and 243.0, respectively) but be-
returned for the final test a week later. Of key a week later. In the learning conditions that re- cause they differed in terms of whether items
importance were the effects of the four learning quired repeated retrieval practice (ST and SNT), were dropped from study or test periods, they
conditions on the speed with which the vocabulary students correctly recalled about 80% of the produced dramatically different effects on long-
words were learned, on students’ predictions of pairs on the final test. In the other conditions in term retention. In other words, about 80 more
their future performance, and on long-term reten- which items were dropped from repeated test- study trials occurred in the STN condition than
tion assessed after a week delay (14). ing (STN and SNTN), students recalled just in the SNTN condition, but this produced prac-
Figure 1 shows the cumulative proportion of 36% and 33% of the pairs. It is worth em- tically no gain in retention. Likewise, about 80
word pairs recalled during the learning phase, phasizing that, despite the fact that students more study trials occurred in the ST condition
which gives credit the first time a student recalled repeatedly studied all of the word pairs in every than in the SNT condition, and this produced no
gain whatsoever in retention. However, when
about 80 more test trials occurred in the learning
phase (in the ST condition versus the STN con-
dition, and in the SNT condition versus the SNTN
condition), repeated retrieval practice led to greater
than 150% improvements in long-term retention.
The present research shows the powerful ef-
fect of testing on learning: Repeated retrieval
practice enhanced long-term retention, whereas
repeated studying produced essentially no ben-
efit. Although educators and psychologists often
consider testing a neutral process that merely
assesses the contents of memory, practicing re-
trieval during tests produces more learning than
additional encoding or study once an item has
been recalled (15–17). Dropout methods such as
the ones used in the present experiment have
seldom been used to investigate effects of re-
peated practice in their own right, but compar-
Fig. 2. Proportion recalled on the final test 1 week ison of the dropout conditions to the repeated
Fig. 1. Cumulative performance during the learn- after learning. Error bars represent standard errors practice conditions revealed dramatic effects of
ing phase. of the mean. retrieval practice on learning.

www.sciencemag.org SCIENCE VOL 319 15 FEBRUARY 2008 967

REPORTS
The experiment also shows a striking ab- recalled the list of vocabulary words than if they 4. A. I. Gates, Arch. Psychol. 6, 1 (1917).
sence of any benefit of repeated studying once only recalled each word one time. Indeed, ques- 5. C. Izawa, J. Math. Psychol. 8, 200 (1971).
6. E. Tulving, J. Verb. Learn. Verb. Behav. 6, 175 (1967).
an item could be recalled from memory. A basic tionnaires asking students to report on the strat- 7. J. A. McGeoch, The Psychology of Human Learning
tenet of human learning and memory research is egies they use to study for exams in education (Longmans, Green, New York, 1942).
that repetition of material improves its retention. also indicate that practicing recall (or self-testing) 8. N. J. Slamecka, B. McElree, J. Exp. Psychol. Learn. Mem.
This is often true in standard learning situations, is a seldom-used strategy (18). If students do test Cogn. 9, 384 (1983).
9. B. J. Underwood, J. Verb. Learn. Verb. Behav. 3, 112
yet our research demonstrates a situation that themselves while studying, they likely do it to (1964).
stands in stark contrast to this principle. The assess what they have or have not learned (19), 10. W. F. Battig, Psychon. Sci. Monogr. 1 (suppl.), 1 (1965).
benefits of repetition for learning and long-term rather than to enhance their long-term retention 11. J. Metcalfe, N. Kornell, J. Exp. Psychol. Gen. 132, 530
retention clearly depend on the processes learners by practicing retrieval. In fact, the conventional (2003).
12. K. W. Thiede, J. Dunlosky, J. Exp. Psychol. Learn. Mem.
engage in during repetition. Once information wisdom shared among students and educators is Cognit. 25, 1024 (1999).
can be recalled, repeated encoding in study trials that if information can be recalled from mem- 13. S. Frank, The Everything Study Book (Adams, Avon, MA,
produced no benefit, whereas repeated retrieval ory, it has been learned and can be dropped 1996).
in test trials generated large benefits for long- from further practice, so students can focus their 14. Materials and methods are available as supporting
material on Science Online.
term retention. Further research is necessary to effort on other material. Research on students’
15. J. D. Karpicke, H. L. Roediger, J. Mem. Lang. 57, 151
generalize these findings to other materials. How- use of self-testing as a learning strategy shows (2007).
ever, the basic effects of testing on retention have that students do tend to drop facts from further 16. H. L. Roediger, J. D. Karpicke, Perspect. Psychol. Sci. 1,
been shown with many kinds of materials (16), practice once they can recall them (20). However, 181 (2006).
so we have confidence that the present results the present research shows that the conventional 17. H. L. Roediger, J. D. Karpicke, Psychol. Sci. 17, 249
(2006).
will generalize, too. wisdom existing in education and expressed in 18. N. Kornell, R. A. Bjork, Psychon. Bull. Rev. 14, 219 (2007).
Our experiment also speaks to an old debate many study guides is wrong. Even after items 19. J. Dunlosky, K. Rawson, S. McDonald, in Applied

Downloaded from http://science.sciencemag.org/ on November 4, 2019

in the science of memory, concerning the rela- can be recalled from memory, eliminating those Metacognition, T. Perfect, B. Schwartz, Eds. (Cambridge
tion between speed of learning and rate of for- items from repeated retrieval practice greatly re- Univ. Press, Cambridge, 2002), pp. 68–92.
20. J. D. Karpicke, thesis, Washington University, St. Louis,
getting (7–9). Our study shows that the forgetting duces long-term retention. Repeated retrieval in- MO (2007).
rate for information is not necessarily deter- duced through testing (and not repeated encoding 21. We thank J. S. Nairne for helpful comments on the
mined by speed of learning but, instead, is greatly during additional study) produces large positive manuscript. This research was supported by a
determined by the type of practice involved. effects on long-term retention. Collaborative Activity Grant of the James S. McDonnell
Foundation to the second author.
Even though the four conditions in the experi-
ment produced equivalent learning curves, re- References and Notes
peated recall slowed forgetting relative to 1. H. Ebbinghaus, Memory: A Contribution to Experimental Supporting Online Material
www.sciencemag.org/cgi/content/full/319/5865/966/DC1
recalling each word pair just one time. Psychology, H. A. Ruger, C. E. Bussenius, Transls. (Dover,
Materials and Methods
Importantly, students exhibited no awareness New York, 1964).
2. R. A. Bjork, in Information Processing and Cognition: Table S1
of the mnemonic effects of retrieval practice, as The Loyola Symposium, R. L. Solso, Ed. (Erlbaum,
References
evidenced by the fact that they did not predict Hillsdale, NJ, 1975), pp. 123–144. 31 October 2007; accepted 12 December 2007
they would recall more if they had repeatedly 3. M. Carrier, H. Pashler, Mem. Cognit. 20, 633 (1992). 10.1126/science.1152408

968 15 FEBRUARY 2008 VOL 319 SCIENCE www.sciencemag.org

The Critical Importance of Retrieval for Learning
Jeffrey D. Karpicke and Henry L. Roediger III

Science 319 (5865), 966-968.

DOI: 10.1126/science.1152408

Downloaded from http://science.sciencemag.org/ on November 4, 2019

ARTICLE TOOLS http://science.sciencemag.org/content/319/5865/966

SUPPLEMENTARY http://science.sciencemag.org/content/suppl/2008/02/12/319.5865.966.DC1
MATERIALS

RELATED http://science.sciencemag.org/content/sci/320/5884/1720.1.full
CONTENT

REFERENCES This article cites 13 articles, 0 of which you can access for free
http://science.sciencemag.org/content/319/5865/966#BIBL

PERMISSIONS http://www.sciencemag.org/help/reprints-and-permissions

Use of this article is subject to the Terms of Service

Science (print ISSN 0036-8075; online ISSN 1095-9203) is published by the American Association for the Advancement of
Science, 1200 New York Avenue NW, Washington, DC 20005. The title Science is a registered trademark of AAAS.
American Association for the Advancement of Science