Monday, December 27, 2010

Maybe we should call it an “unconfidence interval”

Sharon Otterman wrote over the weekend in the New York Times about the complications of value-added scores for teachers, including teacher-of-record issues and a confidence interval that may give observers more pause than confidence. A commenter to my previous post on teacher-of-record noted the confidence interval issue, which is clearly demarcated in the Times story: A city teacher ranked in the 63rd percentile could actually stand anywhere from 46th to 80th. The metric may do okay identifying the worst and best teachers, but in the middle it is somewhat of a muddle, and it is important if you are writing about such systems to be clear about what they do and do not show.

“As a general rule, you should be worried when the people who are producing something are the ones who are most worried about using it.” This was from Douglas Harris, whose University of Wisconsin colleagues produced the New York measure.

6 comments:

  1. Hi Linda,

    Sharon Otterman is quite right about the relative inability of VAM to make distinctions of teachers in the middle of the distribution. Bill Sanders has written on this very point (see his chapter with June Rivers in Ted Hershberg and Claire Robertson-Kraft's "The Grand Bargain"). However, as Sanders and Rivers also point out, the measure is extremely accurate at identifying teachers in the very highest and lowest parts of the distributions.

    The debate over VAM continues to be clouded on both sides by zealots who would want to use it as a single measure to evaluate teacher effectiveness, and those trying so hard to discredit the method so as to avoid having ANY kind of objective measure of educator quality.

    It is unfortunate we have come to this place. VAM is a powerful measure, but it has limitations. We should use it with prudence and good judgment, but we should use it.

    Jason Glass
    Somewhere in Indiana

    ReplyDelete
  2. Hi Caroline,

    I fear in your rush to nail me as "false and invalid," you missed the point of my post, which is that there is danger in the extremes on both sides of this debate. I would think truth of this point would be fairly difficult to argue.

    I am glad we have found common ground in saying that value added warrants consideration and I appreciate your response to my post.

    Jason Glass

    ReplyDelete
  3. Jason Glass, I didn't blast YOU as false and invalid but your statement that anyone who opposed the L.A. Times' wrongheaded and harmful use of value-added methodology is "trying ... hard to discredit the method so as to avoid having ANY kind of objective measure of educator quality." I don't know of anyone trying hard to discredit the method so as to avoid having any kind of objective measure of educator quality. That's a straw-man argument, and it IS dishonest and invalid.

    If anyone were trying hard to discredit value-added methodology so as to avoid having any kind of objective measure of educator quality, I would oppose them too.

    I will take this opportunity to once again emphasis that the Los Angeles Times committed an unforgivable sin against journalistic standards and ethics and needs to renounce and apologize.

    ReplyDelete
  4. I mean emphasize -- sorry. And happy New Year.

    ReplyDelete
  5. CarolineSF:

    "The Los Angeles Times committed an unforgivable sin against journalistic standards and ethics and needs to remounce and apologize."

    Very well said. I agree completely. For those who do not agree, think of it this way: How many dedicated and excellent teachers were falsely and publicly labeled as "least effective" by L.A. Times journalists? This sort of thing usually is not tolerated in our country and for good reason. It's frightening to think what can happen to a nation's principles once economic desperation has manifested itself. Let's hope for a complete renunciation of this type of scapegoating in 2011.

    ReplyDelete
  6. Previously, quite a number of newspaper editorial boards have de facto anointed themselves the arbiters of what charter schools should be approved and how charter schools should be held accountable (or not, NOT being the universal decree). Those include the editorial boards of my hometown paper and my husband's longtime employer, the San Francisco Chronicle, and my former employer, the San Jose Mercury News.

    Appropriate or out of bounds for journalists? Discuss among yourselves.

    So perhaps it was the next logical step for newsroom staff to anoint themselves the arbiters of how teacher quality should be gauged, and to proceed to do the gauging.

    ReplyDelete