Wednesday, September 8, 2010

I read the assessment grant applications, so you don’t have to.

I spent part of the Labor Day weekend reading the grant applications for the assessment consortia that won the Race to the Top grantsSMARTER and PARCC. I know, I know—but don’t worry, I still found time to cook.

The applications left most of the details to the future, so it was not sparklingly clear what will make the new assessments considerably more comprehensive and rigorous than the hodgepodge they replace. There were assurances; for example, the PARCC application promised “items that assess higher-order knowledge skills better than most traditional selected-response items.” I would love to see examples of what they mean.

Among the observations and questions that arose, many of which lend themselves to further exploration by curious journalists:

—Who will be developing the tests (and getting the money)?

—Will weaknesses in current scoring processes be addressed? Having interviewed many scorers when I wrote Tested, who all went into great detail about how haphazard the system is, this is a bugaboo of mine. The eleven or so pages the applications devoted to scoring, out of about 500, don’t give much to work with. PARCC emphasizes vendor or teacher scoring; SMARTER favors teacher or computer scoring; neither really address the degree to which human scoring can be a sloppy process now, even with relatively simple tests. And SMARTER basically says, “Hey, automated scoring should be up to speed by the time we need it.” I think it would be great to read about the state of play of computer scoring.

—SMARTER mentions computer adaptive testing; unless I am missing something (which is possible), PARCC does not. I have watched many students take tests so above or below their level that the assessments were a poor measurement tool; potentially, adaptive tests can address this, though I am not sure how they fit into accountability systems.

—Both consortia will incorporate some sort of research task, at least for older students. SMARTER assessments include “performance events,” up to six for high schoolers, and I wasn’t sure whether they “count” or how they would be scored. The PARCC speaking and listening assessment, part of the language arts battery, won’t count. Why not?

—It was not clear to me what information, exactly, will be available to whom, and when. I got the feeling teachers will be getting item-by-item results on student responses on the benchmark assessments, which is great, but I wasn’t sure about the end-of-year ones.

—YAY: PARCC says that higher education officials will play a role in developing the high school assessments. It’s crazy, now, the disconnect between what students are expected to know to pass high school exams and what they have to know to pass placement exams once in college (to avoid remediation). The feeling I got was that PARCC envisions a system where those tests are, essentially, one and the same.

—PARCC emphasizes “transparent prompts” (for constructed responses, I believe). Does this mean teachers are going to know exactly what questions to anticipate? Understanding expectations is good; preparing narrowly for a specific test question can be awful. I watched third graders, for a whole school year, copy from the overhead dozens of times a paragraph that started, “I know this is a poem because it has rhyme, stanzas and rhythm”—because that was the prompt teachers expected on the test. Not once did the kids actually write a poem. The teachers knew that was not the way to build an enduring understanding of poetry; they knew it wasn’t good teaching. But the kids could write the paragraph when the test came.

This is all just me thinking out loud. Nothing too radical here, nothing that stinks, nothing that shines ... but given that the world revolves around whatever these tests are going to be, I feel like these projects should get at least as much attention as all the other grants that have come down the pike.

1 comment:

  1. I'm afraid my first, longer comment was bloggered. The short answer is, for ELA, read the standards closely and literally, they are practically templates for assessment questions, and they're tailored to easy scoring of constructed responses by either AI or temp workers. That's why they focus on textual analysis.

    ReplyDelete