Wednesday, February 10, 2010

“Hope or Hype in Harlem?”

Knowing nothing but the zillion things I have read, and setting aside that the charter schools themselves are somewhat of a mystery to me, I think the Harlem Children’s Zone is all sorts of awesome. But I have always thought it intriguing how Geoffrey Canada managed to receive so much support (money, praise, etc.) from the type of people who usually insist on seeing results before they proclaim something a success. This has been probably the most written-about single endeavor for children in decades, yet we had not seen a truly thorough look into the two key questions that should be asked about HCZ: Does it work, and can it be copied?

Too early to draw firm conclusions, but now we at least have a start. Helen Zelon at CityLimits.org has reported and written a thorough and very interesting piece about Canada and the Zone, focusing on those two questions. To the first, there is no conclusive answer—Canada himself says they have such a long time horizon for judging success, decades, because the arc of the effort runs from infancy to college. Then why, as Zelon points out, does the HCZ not track students who left the program’s charter schools for public high schools outside the Zone, because the charter part of the pipeline was ending? “We don’t evaluate them in the sense we evaluate our own kids,” an official said. What a mistake. Isn’t the point of the pipeline that they are your “own kids” till they go to college, and wouldn’t you track them even beyond, if you are as results-oriented as you say?

The article leaves me with other questions. Zelon writes, “According to Canada’s tipping point theory, once Harlem reaches a 65 percent level of success—academic, economic, social and health—future success and academic achievement will be the natural outcome.” But 65 percent of what? Academic, okay: 65 percent of students scoring proficient on tests. Economic: 65 percent of people with jobs? Living above the poverty level? Health and social: 65 percent of people having healthy teeth? Managing their asthma? Living with a mother and father? It is bizarre for Canada to talk frequently about a metric that is completely undefined.

I remember Zelon asking me, a year ago, who was criticizing Canada and the HCZ. Really, nobody. I know the organization can make it difficult, if not impossible, for reporters to visit their schools if it is not obvious they are coming to write a puff piece. Like I said, I love lots of what they are doing. But a project that the president wants to replicate around the country should be open to analysis—and not just their own.

I am glad Zelon took the first big step. However: PAYWALL! Bits of the project are posted for free; far more, including interesting graphs on the Zone schools’ population and achievement, you gotta pay for. The piece is superlative and important and the kind of thing that makes me question my reflexive stance on piracy. I mean, I wouldn’t cry if someone put this online where everyone could see it. In lieu of that, just pay the $4.95.

14 comments:

  1. I agree that HCZ is "all sorts of awesome" and that results can be a difficult thing to measure when the goals are so broad and ambitious and the time horizon is so far away.

    It might be interesting to look into a program that resembles HCZ in many respects: Say Yes to Education. They have been around for quite a while and have data on high-school graduation. college persistence and college graduation among their cohort of largely low-income children. Here's how Say Yes president Mary Ann Schmitt-Carey characterized their results with a third grade cohort:

    "We have results from our cohort that started in 3rd grade and have gone all the way up to and through college. We had just shy of 90% of our students graduating high school, and we had over 60% that achieved a postsecondary degree. Fully half of that 60% received a 4-year bachelor’s degree."

    "When you look at those results, you are very close to achieving the results of suburban communities. Our hypothesis and strongly held belief is when you start in kindergarten you can absolutely level the playing field."

    That's quite a claim, and I think it deserves attention.

    ReplyDelete
  2. I've been dying to see real analysis of these schools. If this really, truly works, then we should all be onboard. But nothing I read answered the type of questions you raised in this post.

    I would say that I can understand the HCZ's reluctance to intensive scrutiny, though. I'm taking at face value their sincerity in the belief that HCZ works; but because it indeed takes such a long time to deliver conclusive evidence that it works, it would be a shame to derail the project or slow its momentum based on insufficient data.

    On the other hand, it can be kind of irritating to see an organization accept so much lavish praise and be reticent in allowing meaningful access.

    The Quizzical Teacher

    ReplyDelete
  3. good post, linda -- i hadn't seen the city limits article and your questions are key. however they aren't really all that new -- educators and bloggers like me have been raising them for years now, even as mainstream reporters have been hyping the hcz story. still, welcome!

    ReplyDelete
  4. To me the best way to help at-risk children is to give them public school vouchers so they can apply to any public school. However, I realize that many parents prefer that their children stay in the neighborhood and so, for them, I would support community centers similar to those started by Mr. Canada.

    I am very curious about one thing: Is it true that HCZ students did poorly on the Iowa Test of Basic Skills? If this is true, it represents a discrepancy that must be explored. Obviously, it's extremely important to have valid and reliable data so we can be certain that these programs are truly successful.

    Sadly, when one looks below the surface in many of these "miracle" schools, one frequently finds "gaming" or drilling on test items. I hope the Obama administration will insist on strict transparency before spending millions of dollars on programs that might not be replicable in other cities or states.

    ReplyDelete
  5. This was another bootcamp project!

    ReplyDelete
  6. Perhaps I should read the article first, and I've challenged aspects of the HCZ's coverage, but we also have to be fair. How could the HCZ accurately track kids after leaving for other schools. Public schools have just started trying that, and they are governmental entities that can require data be turned over. It would be hard for the HCZ to even inventory all of the legal issues involved, but less address them and come up with a valid methodology. Tracking them afterwards sounds like a good idea, but I bet they'd just find themselves seeking a representative sample. That would also be valuable but it would also be challenging. Perhaps they should have the Columbia Teachers College take a shot at that research. Perhaps Ed Schools still have a role to play ...

    If traditional reformers don't want data-driven reformers to ignore the opportunity costs inherent in the mandates they impose on over-burdened schools, then we should be fair also. I also appreciate Linda's point asking for evidence whether "drilling" is occuring and whether its harmful. I hope the HCZ keeps an open mind on that issue, and adjusts when/if evidence says they should.

    ReplyDelete
  7. The KIPP operation absolutely claims to accurately track kids who finish 8th grade at their schools. And when they give out info based on that tracking, I don't see any indication that the press shows any skepticism at all -- you've surely heard those claims, so have you shown the same skepticism, Mr. Thompson? So if KIPP can do it, why wouldn't HCZ be able to? It would cost money, but both of them seem to have plenty.

    ReplyDelete
  8. I'd like to make one additional point. When teachers talk about "drilling on the test" or "teaching to the test" they are not referring to teaching to the curriculum, which a teacher should do. When I talk about "drilling on the test" I mean drilling on specific test items. Naturally if multiplication is going to be on a test, the teacher should drill the students on the multiplication tables. However, principals and teachers sometimes look at the specific examples (say 8x6 and 9x4) and drill on those items while ignoring the rest. I was a witness to this sort of "test prep" so I imagine it's quite prevalent. Of course, this kind of "preparation" invalidates the test. Before declaring any school a "miracle" we need to know how much of this is going on.

    ReplyDelete
  9. I hope I haven't outraced my headlights, but I see two basic ways for tracking afterwards. Ask the students' parents for accessing info (like an identifier) so that you don't get stymied by confidentiality laws, or contract with districts that do operate under the authority of law. But wouldn't those districts and states also have to jump through legal hoops? So Caroline, after reading your work on TFA attrition I'd find it awfully hard to believe that KIPP can accurately and completely trace outcomes. Someday, I'd assume, these logistics will be worked out. But remember what happened to first generation data systems at hopsitals. As I understand it, they typically were ripped out after outcomes worsened because the systems were just too complex.

    I hate to say it, but I expect we'll soon be reading exposes about the failure of Obama-funded data systems in many areas, just like we've read about thsoe failures in previous administrations. We need to invest in data, but as Linda notes, we have some huge problems right in front of our noses - namely excessive and destructive test prep.

    And I better shut up and find time to read more before writing further. Now the sun is out though.

    ReplyDelete
  10. At least the press says KIPP instead of TFA when it knows to write KIPP. I read the HCZ articles and I still know very little, but Caroline you sound like you've made a great case. If KIPP can actually trace kids as well as they claim, that strikes me as an admission that their graduates are not as challenging as they claim.

    Here's why I say this. We have a nonstop shuttle between kids in OKC and their families in California and back, as well as between the suburbs and small towns. We have really good people at the State setting up a highly respected longitudinal system of tracking. But its years behind schedule, and things still seem to be getting worst. We used to get quick and efficient notice on IEPs, however, but even with that essential need, we're overwhelmed. So based on the size of the challenge I see, and the information you provide, when I hear people talking about their data results I'm even more suspicious.

    ReplyDelete
  11. Thank you for acknowledging that. It does seem likely to be difficult to track -- I know it's difficult for school districts even to track the whereabouts of kids who suddenly stop coming to school -- so it just stands out to me that KIPP never gets questioned or challenged when it firmly claims to know exactly where every single one of its alumni are.

    ReplyDelete
  12. Tracking students is such a difficult issue. I work with College For Every Student and we have found it very difficult to track students after high school graduation. Now Florida and about 15 other states have implemented or are implementing programs that can track local students who go to public in-state colleges/universities.

    It's hard to track students year-to-year, but last year 97% of our 12th-grade "CFES Scholars" went on to postsecondary education.

    ReplyDelete
  13. Yours are graduating from 12th grade, though. KIPP's are graduating from 8th grade, and it still claims to track each and every one all the way through high school and beyond.

    ReplyDelete
  14. @Caroline: That's a great point. We work with a lot of middle schools as well, and we just don't have the access or resources to track those moving on through high school to college. I'd be shocked if KIPP can actually pull that off, especially with those students who move, end up incarcerated, or whatever else.

    ReplyDelete