We reported that Maryland’s scores in fourth-grade reading on the National Assessment for Educational Progress, known as the “Nation’s Report Card,” went from 231 in 2011 to 232 in 2013, representing no statistically significant increase. The Maryland State Department of Education, by press release, said scores had “improved” on fourth-grade reading since 2011, and we explained that any improvement occurred at a level of statistical significance not officially recognized by the NAEP team of statisticians.
Now we find a report in the Baltimore Sun, bearing the headlines “Md. excluded large number of special-education students in national test” online and “Md. scores on NAEP likely inflated” in today’s print edition, which sums up one of the biggest reasons people should not put a lot of confidence in state-to-state comparisons using NAEP scores.
As Liz Bowie, the Sun reporter who wrote the article, explains, Maryland usually honors individualized education programs (or plans, commonly abbreviated IEP) for special education students or those with any of a whole range of disabilities. One of the ways in which schools accommodate students during testing, especially on a statewide test, is to have a teacher or computer-simulated voice read the questions and answer choices to the student. This is common for students who have difficulty reading on a timed test.
This accommodation is not allowed on the NAEP reading test, so many schools don’t make their students with this accommodation on their IEPs take the NAEP test, since there is no way to provide them with the assistance the IEP legally requires schools to provide during tests.
Other states handle NAEP a little differently. Because there is no way to find out how schools or individual students performed on the NAEP, they say to their schools, “If Johnny gets picked randomly to take the NAEP, let him take it, even though he won’t get the accommodation.” Their interpretation of the IEP is a little more relaxed: since NAEP isn’t a test used for school (or student) accountability, there’s no benefit to the school if students with the read-aloud accommodation on their IEPs don’t take the test. The school has no dog in the fight, so to speak, so they don’t waste their time on the formal process of excluding students, which can involve letters from parents and other documentation.
This phenomenon, as Ms Bowie points out, has made Maryland’s rate for excluding students from the NAEP reading test the highest in the nation, as it has been for some time. A list of exclusion rates for the NAEP in 2011 can be found on the US Department of Education site, here.
“When exclusion rates are higher, average scores tend to be higher than if more children were tested,” said Larry Feinberg, assistant director for reporting and analysis for the National Assessment Governing Board, an independent body that sets policy for NAEP.
Mr Feinberg estimates that Maryland’s fourth-grade reading score of 221 was inflated by about 7 points, and the state’s eighth-grade reading score of 274 was inflated by about 5 points. His deductions are completely hypothetical. The state’s scores are the ones reported, and any conjecture as to how much lower they would be if a different random sample of students took the test is just a guess or a hypothesis. As a hypothesis, it falls into the category of non-testable conjecture: We can’t give the test again.
The article said 66 percent of fourth graders with disabilities were excluded from the reading test, much higher than the national exclusion rate of 16 percent. Data on NAEP’s website show that for the 2011 test, 14 percent of students selected randomly to take the test were identified as having disabilities and 8 percent of the entire sample were excluded. So, if the exclusion rate has gone up to 66 percent, from 57 percent (8/14) in 2011, that is news indeed.
But Maryland schools have every right, according to NAEP’s own rules, to exclude students whose IEPs cannot be accommodated. That’s exactly what they’re doing. Just because other states don’t exercise that right doesn’t mean Maryland cheated or gamed the results on NAEP. In fact, we don’t even care about the scores on NAEP, because they don’t affect any single thing in any single classroom in the state. The only people who care about NAEP scores are the people in Washington setting educational policy—and the editors of Education Week, who use the NAEP scores to rank Maryland #1 among all states. Posters with “#1” have been displayed for several years in a row in almost every window of the Maryland State Education Building in downtown Baltimore, thanks to this artificial ranking by editors of a magazine.
According to the NAEP website, Maryland is in second place for fourth-grade reading and in seventh place for eighth-grade reading. For the record, the state comes in 17th place in fourth-grade math and 21st place, but still above the national average, for eighth-grade math. Whatever that means. If we apply Mr Feinberg’s subtraction rule to only Maryland, the state would be in 14th place in fourth-grade reading and 18th place in eighth-grade reading.
However, there’s no point to ranking Maryland among other states, now that so many of them have adopted the Common Core State Standards. The whole point of jumping on that bandwagon, from an educational point of view—not a political one—was to encourage collegiality, collaboration, and cooperation with teachers from the rest of the country. What is Minnesota doing that’s so great in math? What is Massachusetts doing that’s so great in reading? How can we incorporate some of their ideas into Maryland’s lesson plans? And what can we share with them? The Common Core allows teachers in different states to “speak the same vocabulary,” in a sense. Before the Common Core, teachers couldn’t do that, because the standards in Wyoming were likely to be different from the standards used in Tennessee or Maryland.
Here’s our plea, once again, to stop comparing states. In fact, NAEP itself groups states into those that perform above, at, and below the national averages, from the point of view of statistical significance. The website puts them in an ordered list, but that’s not really how the NAEP results should be viewed. We should instead look at them as putting states into groups based on the statistical significance of their results. Yes, if we apply Mr Feinberg’s subtraction formula, Maryland drops into the “at national average” group for eighth-grade reading, though it’s at the top of the group, right at the border with the “above national average” group of states. That’s not really the point, though.
The way to look at these results, for teachers, is that states in the “below national average” group should look to states in the “at national average” group to make improvements. This potentially provides teachers in the lower-performing states with a small step toward better classroom instruction in the tested subjects. Baby steps are needed when implementing change with young people. And if teachers find any good ideas, they should share them with the teachers they know.
As I’ve said, the whole idea of ranking states is repugnant to the Common Core. We will never realize its full potential to bring teachers in different states together, using a common vocabulary in math and reading, unless we stop paying attention to reporters, editors, and even politicians and business leaders, who want to have convenient lists of states, schools, teachers, and so on. Competition is anathema to proponents of the Common Core, and anyone who says they support the Common Core should give this a try before they simply write another essay or enter another theoretical talking point onto the record.