Does the PARCC test need to be online?

We have completed an analysis of the practice tests posted by the Partnership for Assessment of Readiness for College and Careers, which you can also use for practice by clicking on the “Sample Items” tab here. Our analysis is relevant because Maryland schools are requesting $100 million of additional taxpayer money in order to upgrade computer and network hardware for the purpose of taking the tests from PARCC.

The items on the practice tests aren’t intended to be the actual items students will encounter on the tests, but manipulating the online environment is expected to be similar to what they’ll see. According to the website, the practice tests “give students the chance to engage with simple interactions and technology-enhanced items, including drag-and-drop, multiple select, and text highlighting.”

More complex interactions may very well be developed in the future, but we find, except for the inability to deliver a video in a paper-and-pencil test, that every single question type presented online has a directly comparable format on paper.

To reach our conclusion, we took each test and compared the types of test questions the state of Maryland has given on publicly-released paper-and-pencil tests in its 20-year testing history with the types of test questions on the PARCC practice tests. We also considered, for each question type, whether such a question type could possibly be asked on a paper-and-pencil test, regardless of whether Maryland has ever put such a question type on a paper-and-pencil test.

Some abbreviations used include “4-MC” for multiple-choice questions with four choices and “2/6 MC” for multiple-choice questions that present students with six options and require them to select two of those six options to respond fully.

Elementary school practice tests

Grade/Subject Question Format Number of Questions Done in Maryland? Possible on Paper?
3rd-5th/ELA A (4-MC)→B (4-MC) 6 Yes Yes
3rd-5th/ELA A (4-MC)→B (2/6 MC) 1 No1 Yes
3rd-5th/ELA Constructed Response 1 Yes Yes
3rd-5th/ELA Drag&Drop → 1 bay 1 No Yes
3rd-5th/ELA Drag&Drop → 2+ bays 3 No Yes
3rd-5th/Math Fill in number 5 Yes2 Yes
3rd-5th/Math Droplists/equation 1 No Yes
3rd-5th/Math Constructed Response 2 Yes Yes
3rd-5th/Math Plot on Grid 1 Yes Yes

(1) In at least one case, I would consider the 2/6 Multiple Choice dysfunctional in that the question prompts students to select one answer from A, B, and C, and then one answer from D, E, and F to make up the two selections. The system, however, allows A and B to be selected. This is not a strong objection, since the alternative would give the student a hint and turn the question into two three-option multiple choice questions. This also means that, as the question is presented, we’re testing students’ ability to follow directions, not just understand the content of the question. To the extent that one interferes with the other, these questions are invalid. To the extent that partial credit is not given for selecting, say, the A-B-C choice correctly, these questions are unreliable.

(2) Although Maryland allows students to fill in the answer on several questions, the TestNav system used for the PARCC field tests restricts their input to numbers. If you try to type a letter, you are prompted with a little tool-tip that says “Invalid input.” In one sense, then, the PARCC test is better than a paper-and-pencil test at preventing students from typing an answer that makes no sense in the context of the problem. But in another sense, maybe what we need to know about the student’s progress is that his answer makes no sense, which the PARCC test will not be able to detect.

Middle school practice tests

Grade/Subject Question Format Number of Questions Done in Maryland? Possible on Paper?
6th-8th/ELA A (4-MC)→B (4-MC) 10 Yes Yes
6th-8th/ELA 2/6 MC in part3 4 No Yes
6th-8th/ELA Video Stimulus 3+14 No No5
6th-8th/ELA Highlight Text 1 No Yes6
6th-8th/ELA Constructed Response 2 Yes7 Yes
6th-8th/ELA Drag&Drop → 1 bay 1 No Yes
6th-8th/ELA Drag&Drop → 2+ bays 2 No Yes
6th-8th/Math Fill in number 4 Yes Yes
6th-8th/Math Droplists/equation 1 No Yes
6th-8th/Math Constructed Response 2 Yes Yes
6th-8th/Math Drag to Order 1 No Yes
6th-8th/Math Drag&Drop → 2+ bays 1 No Yes
6th-8th/Math Plot on Grid 1 Yes Yes

(3) In one case, the 2/6 Multiple Choice occurs in both Part A and Part B of the question, but the most common occurrence of the 2/6 Multiple Choice is Part B after a Part A question that calls for additional support. This type of setup cues the student to the correct answer and may bias the test in favor of those students who can master this question format rather than those who have a deeper understanding of the content. For example, the answer options are at least four lines long in one of the 2/6 Multiple Choice questions, requiring students to read the passage and then read and understand all answer options, including the incorrect ones. Having six long options to read is confusing to students taking a high-pressure test. This may bias the results against kids who lack superior patience and render the test unfair.

(4) A video stimulus is, of course, not possible on a paper-and-pencil test. The display is high-tech and very nice in appearance, conforming to all the standard user interface guidelines for running a video (play/pause button, etc.). We say “3+1” here because in one of the questions, the question calls for a constructed response that compares three different presentations: a web page, an article, and the video.

(5) Our response of “No” in the “Possible on Paper?” column does not reflect an inability to test the higher skill of incorporating multiple sources into a constructed response or any other type of response; we simply point out it’s not possible to use a video stimulus on paper.

(6) We answer “Yes” here because, while students cannot select and highlight text on a paper-and-pencil version of a test where that highlighting contributes to the score, it’s possible for an image of the test booklet page to be sent to scorers for consideration. Maryland just doesn’t have any questions like this, but their development on paper-and-pencil forms would not be difficult.

(7) Maryland has used constructed responses for almost two decades on statewide standardized tests. However, the state has never released questions that have more than one passage as a stimulus material. In this sense, the PARCC test asks students to explore the content more deeply, since three different presentations about similar subject material must be analyzed in a cohesive essay. This is not simply a harder question than Maryland has ever asked students before. Rather, it’s testing a higher skill: incorporating multiple sources of information presented in different formats.

High school practice tests

Grade/Subject Question Format Number of Questions Done in Maryland? Possible on Paper?
High school/ELA A (4-MC)→B (4-MC) 4 Yes Yes
High school/ELA 2/6 MC in part8 1 No Yes
High school/ELA Drag&Drop → 1 bay 2 No Yes
High school/ELA Constructed Response 2 Yes9 Yes
High school/Math Fill in number 1 Yes Yes
High school/Math Droplists/sentences 2 No Yes
High school/Math Short Constructed Response 310 Yes Yes
High school/Math Long Constructed Response 311,12 Yes13 Yes
High school/Math Drag&Drop → 2+ bays14 1 No Yes
High school/Math Video Stimulus 1 No15 Yes16
High school/Math Plot on Grid 1 Yes Yes

(8) See Notes (1) and (3) above regarding the use of 2/6 Multiple Choice on the PARCC English language arts test.

(9) See Note (7) above regarding the testing of students’ ability to incorporate multiple sources in developing a cohesive essay about the subject material. Although Maryland uses constructed response questions to assess students’ understanding of a passage, one question on the practice test combined multiple sources, which Maryland has not done. The question type itself, though, is done on several Maryland tests on paper.

(10) Short constructed response refers to text areas in which students are simply required to enter a formula, mathematical expression, and so on. Very few, if any, words are required in these responses.

(11) Long constructed responses require students to justify their answer, explain their reasoning, show their work, etc. This often involves the use of words, sentences, and paragraphs.

(12) In one case, students are told they may justify their result “graphically,” but there are no drawing tools present in the answer space on any of the math CRs. One of the CRs, on a live test, would purport to test students’ estimation skills; it doesn’t, making the question invalid (it doesn’t measure what it purports to measure). This will be my only comment about the practice item content, since we’re concerned here mainly with evaluating the need for the tests’ online delivery.

(13) Consider the function y = x2, for example. Students would have no trouble drawing it, but describing it in words wastes valuable time. In Maryland, we allow students to draw, which tests only their knowledge of functions, not their ability to verbalize the qualities of a function that is much easier—and clearer—to represent with a drawing. This is a serious deficiency on the PARCC tests, compared to Maryland’s current paper-and-pencil tests. It impairs our ability to test what students know about mathematical functions and therefore casts doubt on the validity of these questions.

(14) In the drag & drop question, the behavior differs slightly from the drag & drop questions in English language arts. Here, the expressions the student drags into the receiving bays are replenished, making it possible to drag the same source expression into all the receiving bays. That’s not advisable, of course, but the possibility exists, which makes it more difficult for students to “guess” the right answer by a process of elimination—there is no elimination.

(15) Our “No” response here reflects the fact that Maryland has never been able to present students with a video during a standardized test.

(16) Our response of “Yes” in the “Possible on Paper?” column does not reflect an inability to show students a video during a standardized test. That, of course, is not possible. However, it is completely possible to test the learning standard measured by the actual question on the PARCC test using a paper-and-pencil test. Maryland tests construction all the time on several tests in math; the state just does it in a way that doesn’t involve a video.

About the Author

Paul Katula
Paul Katula is the executive editor of the Voxitatis Research Foundation, which publishes this blog. For more biographical information, see the About page.