Teach Effectively!

A report from the US Department of Education which revealed that use of educational software in schools has yielded little improvements in students’ outcomes is receiving lots of media attention. For examples, there is a front page story in entitled “Software’s Benefits On Tests In Doubt: Study Says Tools Don’t Raise Scores” by Amit R. Paley in the Washington (DC) Post and there is coverage under various headlines (e.g., “Study Eyes Effect of Education Technology” on Fox) by Nancy Zuckerbrod, the AP Education Writer.

The study on which these and other folks report examined reading and mathematics software that had previously been identified as effective and was then used in 132 schools (439 teachers), with the products introduced according to an experimental design. Based on assessments administered a year later, students experienced no significant changes in their reading or math achievement.

1. Test Scores Were Not Significantly Higher in Classrooms Using Selected
Reading and Mathematics Software Products. Test scores in treatment classrooms that were randomly assigned to use products did not differ from test scores in control classrooms by statistically significant margins.

2. Effects Were Correlated With Some Classroom and School Characteristics. For reading products, effects on overall test scores were correlated with the student-teacher ratio in first grade classrooms and with the amount of time that products were used in fourth grade classrooms. For math pwith classroom and school characteristics.

I’m not terribly surprised by this. Most software is not predicated on appropriate instructional principles. But, there are other concerns, too. For example, as is the case in studies where multiple products are compared in aggregate, it is likely that they overall results mask some differences at a lower level. The research team promised software companies that the outcomes for individual products wouldn’t be identified, but there are likely to be different effects by product; some are likely to be more effective than others.

Even more importantly: Even if you have great curricula and instructional packages, it matters how they are used. You can put the teacher’s book that describes an excellent set of lessons for teaching reading on the teacher’s desk, but that’s not going to cause reading scores of the students in that classroom to rise. Even good technology won’t do much good unless it’s used appropriately. (To their credit, the researchers here observed in classrooms and have examined some of these issues.)

Link to stories by Ms. Paley and Ms. Zuckerbrod (Fox version). Link to a site from which you can read HTML pages about the actual report; alternatively, download the full report, which is the result of a study by a large team of researchers associated with Mathematica Policy Research and SRI International, by clicking here (PDF).