Teach Effectively!

Direct Instruction and Success for All models of school reform have better effects on elementary students’ academic outcomes than other models of reform according to a report issued by the Comprehensive School Reform Quality (CSRQ) Center. The CSRQ Center published the report in November 2005 that provides the results of a examination of 22 models for comprehensive school reform. The examination was based on ratings about the extent to which the models met criteria in each of five categories:

• Category 1: Evidence of positive effects on student achievement, including diverse populations;
• Category 2: Evidence of positive effects on additional outcomes (e.g., discipline, attendance, teacher satisfaction, etc.);
• Category 3: Evidence of positive effects on parent, family and community involvement;
• Category 4: Evidence of a link between research and the model’s design; and
• Category 5: Evidence of services and support to school to enable successful implementation.

The center developed ratings based on publically available evidence. Models for which they found multiple examples of rigorous research yeilding consistently positive and relatively strong outcomes got higher ratings than those for which the evidence was weaker, less consistent, or less positive. The ratings were (a) very strong, (b) moderately strong, (c) moderate, (d) limited, (e) zero (evidence available, but no benefits found), (f) negative, and (g) no rating; the “negative” rating was only used for quantitative data, not for qualitative assessments. There are nine pages of detail about the methods used to develop these ratings.

The 22 models evaluated using these methods include (abbreviations are mine): Accelerated Schools PLUS (AS), America’s Choice School Design (AC), ATLAS Communities (ATLAS), Breakthrough to Literacy (BL), Coalition of Essential Schools (CES), Community of Learning (CL), Comprehensive Early Literacy Learning (CELL), Co-nect (CT), Core Knowledge (CK), Different Ways of Knowing (DWK), Direct Instruction (DI), Expeditionary Learning (EL), First Steps (FS), Integrated Thematic Instruction (ITI), Literacy Collaborative (LC), Modern Red SchoolHouse (MRSH), National Writing Project (NWP), Onward to Excellence II (OE), School Development Program (SDP), School Renaissance (SR), Success for All (SFA), and Ventures Initiative Focus System (VI). For each, the report provides an easily understood chart providing a simple description of the model, the grade levels for which it is used, how many schools use it, the costs of using it, and the evidence of meeting the criteria described in the bullets here. The charts for each model are followed by a detailed recounting of the basis for the ratings.

Here’s a simple summary, predicated on the rating system. I looked through the outcomes of the review and noted which models (by abbreviation) had ratings of what level on the criteria I consider most important. If there was no rating on a criterion, I omitted the initials from that row of the table. Within cells, models are listed alphabetically; that is, the order in which I entered them into the table does not imply higher or lower ratings.

My conclusions after reading this report? First, I’m not surpirsed by the results of the evaluation of overall positive outcomes. Others who have looked at this evidence have found pretty much the same thing; take a look, for example at the American Federation of Teachers’ “What Works: Six Promising Schoolwide Reform Programs,” (download PDF) for example. Direct Instruction and Success for All have strong track records and this report recognizes them; if others are effective, they haven’t shown it.

Second, it’s clear that there is only limited evidence on the other objective criteria, effects on additional outcomes and on parents, families, and communities. I consider those outcomes less important, but they are worth considering. Others may consider them more important than students’ outcomes…personal priorities.

Third, the link-to-research criterion is understandable in that it is relatively easy for models’ advocates to document connections to research. However, the nature of the research linkage is important to me. In my opinion, the research that matters is the research reflected in the overall outcomes criterion. A secondary level of important research includes process studies, those that a model’s advocate ought to conduct to examine what parts of the model work and which need refinement, etc. I hardly care whether a model can be connected to developmental research, etc.

This report is worth reading. I’ll be passing it along to the school board members in my neighborhood. Get your own copy from this download link. Full disclosure: I conducted some of the research the report reviewed by the staff on this project; my studies were not used, however, probably because they did not have long-enough time lines.

The CSRQ Center is is operated by the American Institutes for Research (AIR; Washington, DC, US) and funded by a Comprehensive School Reform Quality Initiative Grant fromthe US Department of Education’s Office of Elementary and Secondary Education. The CSRQ Center’s mission is to assist educators and policymakers in choosing school reforms that will improve student achievement and other important outcomes by helping identify reforms that are reliably effective. Flash of the electrons to D-Ed-Reckoning for pointing me to the report.