Teach Effectively!

Many of us who advocate effective instructional practices include frequent assessment of student learning as a critical component of teaching. Witness, for example the emphasis on progress monitoring in most special education practices and its inclusion in sensible response-to-instruction or -intervention models. Indeed, consider the now-somewhat-dated-but-still-unrefuted finding by L. and D. Fuchs (1986) that teachers who use formative assessment have students who score nearly 3/4ths of a standard deviation above the students of teachers who do not use formative assessment.

Yesterday I learned that a study about to be published in Science strengthens my support for assessment. In “The Critical Importance of Retrieval for Learning,” Professors Jeffrey Karpicke and Henry Roediger and Jeffrey Karpicke reported that students’ learning of vocabulary improves when they are tested rather than simply required to study.

Learning is often considered complete when a student can produce the correct answer to a question. In our research, students in one condition learned foreign language vocabulary words in the standard paradigm of repeated study-test trials. In three other conditions, once a student had correctly produced the vocabulary item, it was repeatedly studied but dropped from further testing, repeatedly tested but dropped from further study, or dropped from both study and test. Repeated studying after learning had no effect on delayed recall, but repeated testing produced a large positive effect. In addition, students’ predictions of their performance were uncorrelated with actual performance. The results demonstrate the critical role of retrieval practice in consolidating learning and show that even university students seem unaware of this fact.

Previously, Professors Roediger and Karpicke showed that taking a test, not just studying for it, improved students’ outcomes. They allowed students to study a passage from the Test of English as a Foreign Language (ToEFL) and then assessed their performance. Some students were tested for retention of the ideas (study-test; ST), but others were given a second study session (study-study; SS). They then tested students in both groups 5 min, 2 days, or 1 week later. Initially, the study-study (SS) group performed better, but on the later tests the study-test (ST) group had higher scores. In another experiment the extended their findings, showing that students in a study-study-study-study condition initially had slightly higher scores, but that those in study-study-study-test and study-test-test-test conditions out-performed them dramatically on retention assessments. So, reading the content more frequently did not help as much as taking tests repeatedly.

The beneficial effects of brief tests such as these probably are largely irrelevant to the debate about high-stakes tests. In my view, these results show, however, that an alternative approach to assessing performance—smaller, more frequent, incrementally more difficult—assessments might have value as a means of monitoring whether students are making andmight actually help students to make that progress.

• Link to a press release about one of the studies: “Repeated test-taking better for retention than repeated studying, research shows,” by Gerry Everding.
• Link to the public materials from Science about the more recent study.

Fuchs, L. A., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional Children, 53, 199-208.

Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319, 966-968.

Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249-255.

Time graphic

To what extent does a US state’s high-stakes test correspond with a consistent standard assessment? Using data from the Education Trust and the Colorado (US) Department of Education Time, magazine’s Feilding Cage (with help from Jackson Dykman) created one of those nifty Flashy things that provides an interactive means of displaying data. The interactive map, linked from the thumbnail shown here, shows the degree to which states’ reading and math assessments differ from the results of the National Assessment of Educational Progress (NAEP), the standard US metric for achievement.

The results from fewer than 5% of the states’ tests (~2*50) tests are higher than the NAEP. Mostly, it is far easier to meet standards on the state tests. In the image that I’ve clipped here, one can see that fewer students are judged proficient on South Carolina’s 8th-grade math assessment than on the NAEP. Thus the South Carolina test is more demanding than the NAEP.

To compare across states (i.e., to see where SC’s students stand nationally), one would need to have a consistent measuring stick. The NAEP provides such. States’ high-stakes tests clearly do not.

Because of the way that NAEP scores are developed, this comparison is not based on the scores of the same student on each test, but rather, on an overall comparison. There are lots of technical ins and outs about these comparisons, but the big picture is informative.

I’m attending a meeting of the federal Reading First Advisory Committee and, as serendipity would have it, I stumbled on this article. We were just discussing the problems we’re encountering with not-comparable measures of reading progress across the states; as a result, we’re having a difficult time assessing the effects that Reading First is having on children’s outcomes.

To learn more about the Education Trust, visit the organization’s site. To view another interactive map that allows one to download a detailed comparison of each individual state’s testing, click here. To download a copy of a relevant scholaraly paper from Practical Assessment, Research & Evaluation by Bert Stoneberg, look here.

Flash of the electrons to Maya Frost for her post that alerted me to the Time article.