Having returned from Asia where I had few chances to keep up with some of the topics dear to my heart (but lots of chances to meet folks who share my interest in those topics), I wanted to see what’s happening with the What Works Clearinghouse. There’s not much new—still just the review on middle-school math.
However, I took a minute to look at the data they are using for that review. The staff uses criteria to classify reports of research according to how scientifically trustworthy each study is. There are three categories of quality for studies (plus one that holds those that are in progress at present): (a) Those that meet evidence standards, (b) those that meet standards with some limitations, and (c) those that do not meet the standards.
| Class | Meets criteria | Meets w/ res. | Does not meet |
|---|---|---|---|
| Number | 4 | 6 | 68 |
Looking at these categories, I realized that they provided a glimpse of the quality of research in education. I simply tallied the number of studies in each category for the middle-school math research. The data appear at the right. (There was one study still under review.) I’d welcome a recount but, as it appears to me, greater than 75% of studies of middle-school math curricula fail to meet the standards used by the Clearinghouse.
There are myriad possible explanations. Maybe the standards are too tough. Perhaps there are technical errors in assigning categories (unreliability in the measure). Etc. One potentially compelling (to me) explanation is that too few studies really do measure up to the standards. That explanation fits with the fact that the National Reading Panel found only a few score studies out of 1000s that met criteria for rigor. Here’s hoping the quality of educational research improves.
Links to
- What Works’ list of studies on middle-school math curricula
- What Works’ page about its evidence standards
- National Reading Panel table of contents (downloadable PDFs)
0 Responses to “Research quality”
Leave a Reply