Monday, August 16, 2010

Is John Smith the worst teacher in Los Angeles?

I feel really bad for John Smith. Smith was the subject of an extraordinary Los Angeles Times article Saturday, the first piece of a big database project that uses student test score data to rate teacher effectiveness. Smith, according to the Times, was one of the least effective elementary school teachers in LAUSD.

“Who cares about John Smith?” I can hear a lot of you saying. “I feel bad for his students.” I get that. I do too. And I am thrilled that the Times is devoting so many resources to the issue of teacher effectiveness. Value-added measures are coming to a school near you, if they have not already, and deep journalistic study of the issues involved at the classroom level is rare if not nonexistent. On a policy level, it has always seemed the height of crazy, as the piece puts it, that districts “act as though one teacher is as good as another.” Challenging the primacy of collegiality over quality—yes, this is a dichotomy in school culture, though it shouldn’t be—is overdue. The reporters on this piece are talented, and I am looking forward to their future stories on the database, as I am confident they will be telling, provocative and important.

So why do I feel bad for John Smith, and relieved that his anonymous name* probably spares him from a fair bit of e-hate to his inbox? Most of all, because this may well be the first time he has gotten information saying his skills are lacking. Your students perform poorly under you for years, and the first indication you get is accompanied by your photo on the front page of the Los Angeles Times? Ouch.

My other concern is more practical. The article sums up criticisms of value-added measures of teacher effectiveness as follows: the tests are flawed, they cannot measure intangibles of teaching, what do statisticians know about teaching? This felt, to me, too dismissive of the nuances of value-added and standardized testing, the logistical complications, the legitimate shortcomings. The strengths, of course, were made plain.

You can read a summary of the study done for the Times, by Richard Buddin of Rand Corp., here. I left my undergraduate math major once my classes stopped containing actual numbers, so I do not understand all the formulas. I do know that the report does not seem to address student mobility, student absences, co-teaching, pullout interventions and other workaday factors that potentially complicate value-added. It suggests that the effectiveness of a given teacher is best studied over several years, but the Times article seems to be based on one-year measures. These considerations don’t negate the value of the measure, but they make for important context.

I assume—I hope—these nuances will be addressed in later stories. Teachers are given the chance to comment on their value-added scores in the Times database, but that is not the same as reporters using their skills and authority to explain what these numbers (the students’ scores, the teachers’ scores) do and do not show us. We need transparency all along the pipeline, not just at the end.

*A few people have asked me why I thought John Smith was a pseudonym. I didn’t mean that. I meant that such a name is so common people might have a hard time tracking him down online.

8 comments:

  1. Hi Linda,

    I am an advocate of using Value Added analysis methods, but it is critical that they be used appropriately.

    While VAA is arguably the most accurate and sophisticated way we have of looking at assessment data, its not without its flaws - as the article points out.

    For accuracy, VAA should be used in combination with several other sources of evaluative data on a teacher and used over-time so you start to mitigate the possibilities of falsely identifying a teacher as ineffective (or effective).

    Some VAA critiques put the number of errors in these methods at 1 in 6 each year. Meaning, 1 in 6 times you'll make a false inference about teacher quality from them. Clearly, this is not an acceptable rate. However, when you look at VAA data and you have "below expected" for 3 years or more, the probability that you're making a false inference drops to a much more acceptable rate of about 99.5% accurate.

    Think of it like this. What's the probability you would roll a 1 on a six-sided die? Well one in 6 of course. But think about how unlikely it would be you would roll a one three times in a row. If I've got a teacher who gets a "below" rating three years in a row, I'm pretty sure we've got an ineffective teacher.

    So, if I've got 3 years worth of VAA data telling me I've got a poor teacher, plus 3 years of sub-par evaluations, plus a principal and master teacher's qualitative assessments of teaching performance all lining up ... I can be pretty darned sure we've got someone who doesn't need to be in front of kids.

    The problem with the LA Times article is that it notes all the technical issues with value-added and then just goes ahead and uses one year of data to make an inference.

    As for your sympathy of Mr. Smith, there is nothing wrong with your compassion. Even if we assume Mr. Smith has 3 years of poor value-added results (indicating a very high probability he is a poor teacher) and we have other data showing he's just not good at teaching, we don't have to be hateful or personal about it towards him.

    We should be moving this guy out of the classroom because its better for kids and because its better for him. How terrible it must be for someone to be trapped in a profession where they are a failure. This guy might actually be tremendous at doing something else with his life besides teaching.

    Teaching is not easy and much of it is an art. Despite the protestations from the LA union, not everyone can do this.

    Jason Glass

    ReplyDelete
  2. I assumed the John Smith was a psuedonym, because I couldn't believe that we have sunk so low. I couldn't believe Hechinger would have anything to do with such a violation of common decency. The people who should be complaining the most should be the advocates of VAMs for evaluation and decision-making. Just as bad money drives out good, bad reforms and experiements drive out good.

    Did this go through any ethics panels? They essentially draft Smith et al into being lab rats for their experiment and then revealed his name. At first I assumed there must have been a formal process to decide on the ethics of naming names. But maybe the bitterness is even deeper than I realized. Maybe we're going from and educational civil war to a mob mentality.

    ReplyDelete
  3. I just finished the second reading of Buddin's article and I could not find evidence that absenteeism or mobility was mentioned, and he assumed hetrogenity to be 0. But think of the three students who Smith couldn't handle, who were moved to other classes. Of course, the three could all be the result of Smith's incompetence or they may have made their new teacher seem incompetent, or there might be a combination of the two.

    You are right that the subsequent articles may show more nuance, or they may not. Community leaders and parents reading the articles may show more nuance or not. Other newspapers and districts may show more nuance or not.

    Biddle says that change between grades only has a small effect, and for those grades, that may be true or not. But the change between 5th and 6th will have a huge effect, as will the change from middle to high school. Buddin may know enough to listen on that issue or not.

    ReplyDelete
  4. Jason Glass, it's inherently hateful and personal to publish the teacher's name in the newspaper. It would only be fair to publish the names of bad reporters, if that's the way the new game is played.

    ReplyDelete
  5. Bad reporters' names are publicly attached to their work, and the public is already perfectly able to evaluate the quality of that work if they so choose. So that's a pretty bad example.

    ReplyDelete
  6. That's why I said IF they choose. Indeed, off the top of my head, I can name lots of bad reporters -- Stephen Glass, Jayson Blair, Ruth Shalit, etc., all of whom plagiarized or invented stories. When reporters do a bad job like that, their identities are certainly not kept secret.

    ReplyDelete
  7. Those are reporters who aggressively committed wrongdoing, not just less-effective or less-competent reporters. Your comparison absolutely doesn't hold up.

    ReplyDelete
  8. Are test scores important... Sure they are. but is this where's we really want education to go? Teaching to the test? What about all the other important stuff that we teachers teach in school. Sure there are ineffective and down rift bad teachers but there are also bad waitresses, bad salesman, bad lawyers, doctors etc. Back to test scores, why aren't we recognizing that it's not fair to judge teachers who teach in a low income poverty school to a teacher who teaches in an area that has more money. There are so many different factors that go with testing, bad test takers, social factors, peers, home life. If a kids parents are fighting before a test do you really think that they are going to do well? No. Many of my students are the parents in their household and school is not their first priority no matter that I am a good teacher.
    I am so sick of all these people not in education making judgement about what should be. Look at the whole picture. Parents, students, teachers, admin, city, area. You want us to teach to the test? That is what you will start to get. Then you will really start to see kids
    not even close to being prepared for real life, highschool, and college because their k-8 education was taught to the test. I hope people will band together, and truley think it through before they decide that test scores are the most important thing.

    ReplyDelete