As predicted, bottom drops out on N.Y.’s state tests

When we read about young kids crying in classrooms during New York state’s new standardized tests, which school officials claim are “aligned” to the Common Core state standards, we were kind of expecting the results to be bad, catastrophic even. That’s pretty much what happened, as we found out when the statewide results were officially released earlier today, the New York Times reports.

In New York City, for example, 26 percent of students in third through eighth grade passed the tests in English, and 30 percent passed in math, according to the New York State Education Department. Last year, using easier tests that didn’t claim to be aligned to the Common Core, 47 percent of the city’s students passed in English, and 60 percent in math. In nine schools in the city, not a single student passed the math exam.

Now, to say that a “test” is “aligned” to a set of learning standards—here, the Common Core State Standards—means that an understanding of those learning standards or performance expectations is required to answer the “questions” that comprise the test. The questions themselves are secret, perhaps so they can be used in future years, but some bits and pieces have been published in assorted newspapers, and at least a few of the questions do not appear to be aligned to the Common Core.

Other questions have also been disputed, namely those that incorporate trademarks and brand names like Mug Root Beer or Lego in reading passages. But trademark problems are minor compared to questions that aren’t aligned on a statewide test that officials claim, overall, is aligned. This design flaw casts doubt on the entire test.

For example, we’re going to take a look at some of the test questions released by the New York State Education Department. Our analysis will be conducted in terms of fairness and alignment to the Common Core.

Grade 3 mathematics

What is another way of expressing 8 × 12 ?
A) (8 × 10) + (8 × 2)
B) (8 × 1) + (8 × 2)
C) (8 × 10) + 2
D) 8 + (10 × 2)

Option (A) is correct. Commentary: The item measures 3.OA.5 because it asks the student to apply the distributive property.

Here’s the exact wording of the standard in the Common Core (3.OA.5), along with the examples provided: “Apply properties of operations as strategies to multiply and divide. Examples: If 6 × 4 = 24 is known, then 4 × 6 = 24 is also known. (Commutative property of multiplication.) 3 × 5 × 2 can be found by 3 × 5 = 15, then 15 × 2 = 30, or by 5 × 2 = 10, then 3 × 10 = 30. (Associative property of multiplication.) Knowing that 8 × 5 = 40 and 8 × 2 = 16, one can find 8 × 7 as 8 × (5 + 2) = (8 × 5) + (8 × 2) = 40 + 16 = 56. (Distributive property.) [Students need not use formal terms for these properties.]”

The state claims that this problem tests students’ ability to “apply” the distributive property, which would mean the question is aligned to the standard. The only problem is that “applying” the property, as (part of) a “strategy to multiply and divide” isn’t what the question does. I’m willing to discuss whether this is an “application” of the distributive property, rather than just recall of what it means, except for a couple things:

  • Standard 3.OA.3, just above this one, implies that third graders should be able just to multiply 8 × 12 and get the answer, since it’s less than 100
  • Order of operations using parentheses or grouping symbols isn’t in the Common Core until fifth grade (5.OA.1), so all the answer choices rely on knowledge third graders aren’t accountable for

As to point 1, third graders should know not to use the distributive property to find the product of 8 × 12. Then, if 96 isn’t one of the answer choices, they might recognize this as the distributive property. It’s common to teach students to break out multiplication by 10, which is easy, and just add the partial sums obtained as the products of smaller numbers. In that sense, the question is aligned to the standard, but the “application” of the distributive property is inappropriate for a statewide exam. A teacher at one school, following the standards, may have taught students how to do 8×12 using 3.OA.3, while a perfectly good teacher at another school taught kids using 3.OA.5. That means the question on the statewide exam is unfair and that it potentially cross-pollinates one standard with student response data for another. This can make analyzing the results difficult or even impossible.

As to the second point, the Common Core in third grade does provide for evaluating expressions with two operations in them (3.OA.8: Solve two-step word problems using the four operations). This question, however, has answer choices with more than two operations, plus it has parentheses, so the strict assessment limit imposed by the Common Core just fails. Real Order of Operations learning isn’t part of the Common Core until sixth grade (6.EE.2c or 6.EE.4), although a fifth-grade standard (5.OA.1) would have students understand how to work with parentheses in all of these expressions.

Back to the real problem: This is not an application of the distributive property; it’s simple recall in that it requires students only to rewrite an expression in an equivalent form. And despite the fact that some kids may do the multiplication that way, odds are they won’t, because they have supposedly mastered their multiplication facts enough to know that 8 × 12 = 96. Not seeing the answer they have learned in the choices is frustrating, especially for third graders.

Technically, then, the question isn’t aligned to the standard the state says it’s aligned to. The Common Core, beautifully, expects students to do some deeper thinking to solve problems using “strategies.” That’s not found in this question, which knocks the question right out of alignments and casts doubt on our ability to provide a standardized test in third grade that assesses the critical thinking we want to assess.

Let’s do a rewrite on this and a few other released items. Instead of using expressions, we could have put in a little effort to create test questions that are, in fact, aligned to standard 3.OA.5.

Which expression represents the total area of the yellow shaded rectangle?

Answer) 3×14 = 3×10 + 3×4

To increase the item’s difficulty a little, we could provide only 3×14 = 30 + 12. This would require students to break out the 10 on their own and apply the distributive property. We could also have tested this learning standard using a short answer question, something like “Write an expression that uses multiplication, addition, and the number 10 to represent the total area of the two shaded rectangles.”

In an extended response, we could also test this objective, along with the critical thinking the Common Core is supposed to promote, by presenting students with an incorrect answer and asking them to both provide a correct expression (or equation, if we want them to actually do the arithmetic in applying the distributive property, rather than just setting it up) and explain how our wrong answer was incorrect. This problem does rely on knowledge of area (3.MD.7b), but at least it’s a third-grade concept.

In fairness, application of the distributive property isn’t easy to test in third grade. New York has apparently resorted to testing simple recall, but if that’s all we’re doing, we might as well stick with the old tests. Also, if we’re going to test just the set-up, we’re not making it relevant to the real world. Nobody is ever going to pay you money for just setting up the expression. Still, it would be better to choose a problem where the application of the mathematics behind the distributive property at least represented a grade-appropriate application.

Say you want to paint part of a floor yellow and part green, as in the diagram above. You have to buy so much paint per square foot of floor, and since you haven’t memorized what 14×13 is by third grade, you’re going to have to use the distributive property to compute the area. Show your work, and don’t let your final answer include any multiplication steps with numbers greater than 10.

That would test a student’s ability and understanding in applying the distributive property.

Grade 4 math

Rosa wrote a pattern using the rule “subtract 7.” The first two numbers in her pattern were 83 and 76. Which number below is part of Rosa’s pattern?
A) 41
B) 49
C) 57
D) 61

Option (A) is correct: 83, 76, 69, 62, 55, 48, 41. The item measures 4.OA.5 because it asks the student to generate a number pattern that follows a given rule and rewards recognition of key features of the pattern not explicitly stated in the given rule.

This question also has issues with alignment, strictly speaking, but they are minor compared to the dreadful word choice in the sanctioned test item. People do not “write” a pattern, for Pete’s sake, at least not according to the Common Core. The standard itself is worded as follows: “… given the rule “Add 3″ and the starting number 1, generate terms in the resulting sequence …” Fourth graders might not be familiar with the terminology (terms, etc.), but we should at least call a sequence a sequence, not a pattern.

We could say something like this: Rosa wrote a sequence of numbers using a pattern that followed the rule: “Subtract 7 from each number in the sequence to get the next number in the sequence.” Which number below is part of Rosa’s pattern?

That would be so much more “mathematical” in terms of the wording. But now for the real problem with this question. The learning standard in the Common Core says that students should “identify apparent features of the pattern that were not explicit in the rule itself. For example, given the rule “Add 3″ and the starting number 1, generate terms in the resulting sequence and observe that the terms appear to alternate between odd and even numbers. Explain informally why the numbers will continue to alternate in this way.”

All the answer options are odd, so the student is not “rewarded for recognition of” that key feature of the pattern not explicitly stated in the given rule. That is, the state of New York officially annotates this question and in so doing, says it “rewards recognition of key features” in the pattern. That simply is not what actually happens in the question or in students’ response to the question. All the answer choices are odd numbers.

You say that was just an example, perhaps. And you would be right. But other features of the pattern aren’t checked either. For decreasing patterns (not this one, but another decreasing pattern), a feature might be that the number subtracted from each term increases as the pattern goes on, such as 51, [–10 =] 41, [–9 =] 32, [–8 =] 24, [–7 =] 17, … If students were to put these values into a table, they would be able to quickly spot errors and make corrections. In other words, recognizing this feature within the pattern can be rewarded, but it only works if there actually is a feature for students to identify. And with multiple choice questions, it only actually aligns with the standard if at least two of the four answer options don’t have the feature the student has identified.

To help a little with the alignment here, we could just make two of the answer options even, or we could turn the question into a short answer question and just ask the student what, if any, features he noticed about the resulting sequence generated by following the given rule.

Grade 5 math

Which term can be put in the blank to make the statement below true?

3,000,000 = 30      

A) thousands
B) ten-thousands
C) hundred-thousands
D) millions

The correct answer is, according to NYSED, (C). Three million is equivalent to 30 hundred-thousands. The item measures 5.NBT.1 because it asks the students to recognize that in a multi-digit number, a digit in one place represents 10 times as much as it represents in the place to its right and 1/10 of what it represents in the place to its left. In this case, the 3 in 3,000,000 represents 10 times what a 3 in the hundred thousands place would represent; equivalently, the 3 in 3,000,000 represents 30 hundred thousands.

I almost feel like not saying anything about this question. Although there may be such a thing as “30 hundred-thousands,” if anybody ever said that number to me in a “true statement,” I would be very confused. There is no example provided in the Common Core document, and the wording from NYSED quoted above is exactly what the Common Core says.

But, the department’s explanation defies logic. The question didn’t ask what the “3” in 3,000,000 represented: if it did, millions would be the correct answer. It asked what “3,0” represents, which is beyond the assessment limit of the Common Core State Standards. Students are required to say what a “digit” represents. The “3” in the millions place certainly represents 10 times as much as it represents if it had been in the hundred-thousands place, but the question doesn’t put the “3” in the hundred-thousands place, even hypothetically. Rather, it asks students about “30” being in the hundred-thousands place.

There’s also a problem with parallelism here. If what’s on the right side of the equal sign in the “true statement” is supposed to be filled in with words, then what’s on the left side of the equal sign should also be written out in words. Two things that are supposed to be equal should be written in the same format. See Chicago Manual of Style, 14e, at 8.8, for example. I would recommend the following: 3 million = 30      —if the question doesn’t change as a result of non-alignment.

Grade 6 math

Let’s give the state of New York a break from our horrible discovery of questions that aren’t aligned to the Common Core and instead focus on the subject of mathematics. On the sixth-grade math test sample released, there’s an item that shows a table of how many quarts of tea can be made from so many tea bags.

The table below shows the number of tea bags needed to make different amounts of iced tea.

Tea Bags Quarts of Tea
8 2
16 4
24 ?
36 9

What is the total number of quarts of iced tea that can be made with 24 tea bags?
A) 5
B) 6
C) 7
D) 8

The correct answer is (B). This item measures 6.RP.3a because it asks students to find the missing value in a table of equivalent ratios relating quantities with whole number measurements.

This is a common instructional scenario for most middle school students: a function table. What you’re supposed to do is get the number in the right-hand column by performing some mathematical operation on the number in the left-hand column. The trick is, the same operation has to work for every row in the table.

A student might approach this problem correctly by noticing that the numbers on the right are lower than the numbers on the left. You’re not going to add or multiply, so that means in order to get the numbers on the right, which the question focuses on, you’re going to subtract or divide.

Take the first row. To get 2 from 8, you could either subtract 6 or divide by 4. Let’s move on. The second row has 16 on the left and 4 on the right. You can’t get to 4 by subtracting 6, so clearly, the rule for this function table is divide by 4, a k a, it takes four bags to make every quart.

To solve the problem, we simply apply the rule to 24: 24 ÷ 4 = 6. “Students may also recognize that as the number of tea bags increases by eight (from 8 to 16) the total quarts of iced tea rise by two (from 2 to 4) and determine that another increase by eight (from 16 to 24) will lead to an equivalent increase of two in the total quarts of iced tea,” New York tells us. That’s an equally valid way to solve the problem, but it doesn’t really test students’ understanding of function tables.

They can get the right answer by just looking at the right-hand column. It starts out with a pattern: 2, 4, … 6 is probably next. I think this problem shows how students can arrive at the right answer without understanding the standard. It’s technically aligned to the standard, but the item itself is poorly written. It should never be possible in a math item to do anything but the right operation, or some mathematically appropriate derivative of it, to get the right answer. Students shouldn’t be able to perform some random operation with the numbers on the page and come to the right answer.

This problem is easily fixed, though it should have been corrected before the state submitted these items to the public for review or to teachers for instructional purposes. To fix the problem, we simply change the first row of the table to use 12 tea bags to make 3 quarts of tea. Problem solved. So easy. And the item would then have been perfectly aligned to the standard.

Grade 7 math

Which steps can be used to solve for the value of y?

⅔ (y + 57) = 178

A) divide both sides by ⅔, then subtract 57 from both sides
B) subtract 57 from both sides, then divide both sides by ⅔
C) multiply both sides by ⅔, then subtract 57 from both sides
D) subtract ⅔ from both sides, then subtract 57 from both sides

The correct answer is (A). The item measures 7.EE.4a because it involves solving equations of the form p(x + q) = r, where p, q, and r are specific rational numbers. This item specifically assesses if students can identify the sequence of operations required to solve equations in the form p(x + q) = r. While there are many different ways to solve an equation of this type, A is the only choice that leads to a solution in two steps.

The problem here is that the student doesn’t know the learning standard the item writer is trying to test. There’s more than one way to solve this problem, and the question is repugnant to the overall reason for the existence of the Common Core. I might, for example, want to × both sides by 3 first in order to get rid of the pesky fraction. Unfortunately, my perfectly good strategy isn’t one of the choices. That’s frustrating.

The item writer incorrectly assumes that the Common Core specifies the steps students should take to solve the problem. This item, while probably aligned to the standards cited by NYSED, is inherently unfair: it does not respect the designers of the Common Core, who held that there are many approaches to solve math problems and no one set of “steps” is universal.

The item should have been converted to short or extended answer, where students could show their work and that work could be evaluated for its appropriateness to the solution strategy. As a multiple choice question, this one needs to go on the trash heap. The wrong answers are in no way correct, which means they aren’t even plausible, and the question just screams to be asked as a short answer question.

Grade 8 math

On one eighth grade math question, students are presented with four graphs, three of which represent functions and one which does not.

The correct answer is (C). Inputs between 1 and 2 inclusive have more than one output on the graph. The item measures 8.F.1 because it involves understanding that a function is a rule that assigns to each input (x) exactly one output (y) (though two different inputs may have the same output, as in graphs B and D). This item specifically requires that the students determine which graph does not represent a function of x.

Although functions per se aren’t really treated properly in the Common Core until high school, students in eighth grade should understand the vertical line test. That means if you drop a vertical line anywhere on the graph, the graph is not a function if that line hits more than one point on the graph. For example, in option (C), if you drop a vertical line at x = 1.5, it will hit two points on the graph, one at y = 1 and one at y = 3. Therefore, (C) is definitely not a function.

The problem with this item is that it sort of relies on a double-negative, in mathematical terms. We know (C) isn’t a function because we can see how it fails the “vertical line test,” but we don’t know that (B), for example, is a function. Who knows what happens to the graph? Maybe it just swings around and makes an ellipse or an oval that’s taller than it is wide.

Of course, high school students will probably recognize this as a parabola, and they would know it doesn’t curl back around to make an ellipse, but eighth graders have not been introduced in the Common Core to the graphs of quadratic functions. How are they supposed to know this is not the bottom half of an ellipse?

This could be fixed by making (B) more obviously incorrect, such as with the graph of a third-order polynomial instead of a second-order one. This would have one side, maybe the left side, pointing downward with an arrow, and the other side pointing up with an arrow. No way they’ll ever reverse, and more importantly, no way an eighth grader would ever think it could possibly fail the vertical line test, which is what the item purports to assess.

So, in conclusion, we have seen how many of the items on the release form do not align to the Common Core standard the New York State Education Department says they align to. These items may not have been on any given student’s test, and I don’t make any claims to know the content of actual test items. But it wasn’t difficult to find alignment and fairness issues with the items the NYSED has made public, and I am concerned about the items that weren’t released and were actually put onto the test.

About the Author

Paul Katula
Paul Katula is the executive editor of the Voxitatis Research Foundation, which publishes this blog. For more biographical information, see the About page.