Teach Effectively!

US federal law requires that US states report the percentages of students who attain proficiency in reading and mathematics in 3rd through 8th grades. The percentages differ from state to state. In Mississippi 88% of 4th graders test as proficient, but in South Carolina only 35% of the 4th graders test as proficient. Does that mean that some states are doing a better job than other? Should everyone move her sons and daughters to Mississippi?

Probably some states are doing better than others, but these data are misleading. The states are using different measuring sticks. The US Institute for Education Studies has released a report of an examination of the relationship between state test outcomes and the National Assessment of Educational Progress (NAEP).

It turns out that the higher the percentage of students meeting proficiency standards on state tests, the lower the average score on the NAEP. In fact, the mean NAEP score for South Carolina is ~230 whereas for Mississippi it is ˜160.

Here is the executive summary of the report.

Executive Summary

Under the No Child Left Behind Act (NCLB), states are required to report the percentages of students achieving proficiency in reading and mathematics for grades 3 through 8. For each subject and grade combination, the percentages vary widely across states. For grades 4 and 8, these percentages can be compared to the estimated percentages of students achieving proficiency with respect to the standard established by the National Assessment of Educational Progress (NAEP). Again, large discrepancies are observed. This variation could derive from differences in both content standards and student academic achievement from state to state, as well as from differences in the stringency of the standards adopted by the states. Unfortunately, there is no way to directly compare state proficiency standards because states are free to select the tests they employ and to establish their own performance standards.

This report presents the results of applying a methodology for mapping state proficiency standards in reading and mathematics onto the appropriate NAEP scale, employing data from the 2004–05 academic year. The mapping exercise was carried out for both grades 4 and 8. For each of the four subject and grade combinations, the NAEP score equivalents to the states’ proficiency standards vary widely, spanning a range of 60 to 80 NAEP score points. Although there is an essential ambiguity in any attempt to place state standards on a common scale, the ranking of the NAEP score equivalents to the states’ proficiency standards offers an indicator of the relative stringency of those standards.

There is a strong negative correlation between the proportions of students meeting the states’ proficiency standards and the NAEP score equivalents to those standards, suggesting that the observed heterogeneity in states’ reported percents proficient can be largely attributed to differences in the stringency of their standards. There is, at best, a weak relationship between the NAEP score equivalents for the state proficiency standard and the states’ average scores on NAEP. Finally, most of the NAEP score equivalents fall below the cut-point corresponding to the NAEP Proficient standard, and many fall below the cut-point corresponding to the NAEP Basic standard.

These results should be employed cautiously, as differences among states in apparent stringency can be due, in part, to reasonable differences in the assessment frameworks, the types of item formats employed, and the psychometric characteristics of the tests. Moreover, there is some variation among states in the proportion of NAEP sample schools that could be employed in the analysis.

This variation is likely to fuel a call for a national test, as Education Secretary Margaret Spellings notes in an editorial. I’m not sure I agree with her, but actually the NAEP would be fine if we could just administer it more widely and report the results very transparently.

Get your own copy of the report. Read

• Secretary Spellings’ editorial
• The New York (NY, US) Times article by Tamar Lewin about the report.
• The Associated Press article (as carried by the Baltimore (MD, US) Sun.
• Local news coverage and track the story using Google.