Merit pay and the candle problem

Let us pretend for one moment that many of the corporate education reforms being proposed offered more than just a metaphorical big stick with which to fire teachers more easily, but also a few carrots too in the form of extra money toward paying high performers as determined by their students test scores. Yes, yes, we know.

Let's go even further, and pretend that student test scores were the perfect means with which to judge the effectiveness of any teacher. What do we know of financial incentives? From Nature magazine.

here's a simple fact we've known since 1962: using money as a motivator makes us less capable at problem-solving. It actually makes us dumber.

In the 1940, an experiment was carried out, now referred to as the "Candle Problem". The experiment has the participant try to solve the problem of how to fix a lit candle on a wall in a way so the candle wax won't drip to the floor. The participant can only use (along with the candle) a book of matches and a box of thumbtacks.

Let's go back to that Nature article to explain the rest of the experiment, and it' counterintuitive results

The only answer that really works is this: 1.Dump the tacks out of the box, 2.Tack the box to the wall, 3.Light the candle and affix it atop the box as if it were a candle-holder. Incidentally, the problem was much easier to solve if the tacks weren't in the box at the beginning. When the tacks were in the box the participant saw it only as a tack-box, not something they could use to solve the problem. This phenomenon is called "Functional fixedness."

Sam Glucksberg added a fascinating twist to this finding in his 1962 paper, "Influence of strength of drive on functional fixedness and perceptual recognition." (Journal of Experimental Psychology 1962. Vol. 63, No. 1, 36-41). He studied the effect of financial incentives on solving the candle problem. To one group he offered no money. To the other group he offered an amount of money for solving the problem fast.

Remember, there are two candle problems. Let the "Simple Candle Problem" be the one where the tacks are outside the box -- no functional fixedness. The solution is straightforward. Here are the results for those who solved it:

Simple Candle Problem Mean Times :
WITHOUT a financial incentive : 4.99 min
WITH a financial incentive : 3.67 min
Nothing unexpected here. This is a classical incentivization effect anybody would intuitively expect.

Now, let "In-Box Candle Problem" refer to the original description where the tacks start off in the box.

In-Box Candle Problem Mean Times :
WITHOUT a financial incentive : 7:41 min
WITH a financial incentive : 11:08 min
How could this be? The financial incentive made people slower? It gets worse -- the slowness increases with the incentive. The higher the monetary reward, the worse the performance! This result has been repeated many times since the original experiment.

We've published a video on this phenomenon before, titled "As Teacher Merit Pay Spreads, One Noted Voice Cries, ‘It Doesn’t Work’", and an article from the Harvard Business Review, titled "Stop Tying Pay To Performance".

Here's another video - The surprising truth about what motivates us

Knowing all this begs the question, why are we going down the path of some of these corporate education reforms, when we have known for over half a century many of them are flawed concepts that have been demonstrated to fail time and time again?

Simple thinking, bad reporting

Providing a broad based education for K-12 is a very complex endeavor. It's that complexity which makes it difficult to distill what policies work and what don't, when so many variables affect student outcomes.

However, a cottage industry is being developed to reduce the entire topic of public education to a letter grade or a single number. It's as though these architects of simplicity have read Hitchhikers Guide To The Galaxy and determined that maybe because the answer to "the earth, universe and everything" is 42, it's should be even easier to grade a school as a simple C, or a teacher to a 1.07.

The problem of course is that much like Douglas Adams' novel, this thinking is also science fiction. To distill complexities down to such simple terms, means making highly subjective decisions and ignoring, or worse, being oblivious to, multiple variables.

A few cases in point. Ohio is about to deploy a new school rating system, based upon subjective measures, and ignoring a host of other factors.

As an educator and parent I could rail for days about the lack of actual meaning behind any letter grade, whether an A or an F, and this is a decision that even Rick Santorum would call anachronistic. If your child brings home a ‘C’ on his report card, what does that mean? Does than mean he’s working his ass off and completing all of his homework but struggling with expressing his knowledge on written tests? Or does it mean that he’s uninterested in completing homework that isn’t challenging him while attaining perfect scores on assessments? Or does it just mean that he is earning consistent C’s on every single assignment whether in-class, homework, quizzes, or tests? Perhaps it’s some combination of the above, so what does that tell you or your son about what he needs to do to improve?

See how clear those letter grades are?

Separating the effects of economic factors from school performance doesn't appear to have been one of the major efforts undertaken, even though we have known for a long time that a students socioeconomic status, and that of the school district is the leading predictor of performance, by far.

Another recent example has been the use of teacher level value add scores by New York's print media

The article is a pretty remarkable display of both poor journalism and poor research. The reporters not only attempted to do something they couldn’t do, but they did it badly to boot. It’s unfortunate to have to waste one’s time addressing this kind of thing, but, no matter your opinion on charter schools, it’s a good example of how not to use the data that the Daily News and other newspapers released to the public.
However, we can’t take the performance categories – or the Daily News’ “analysis” of them – at face value. Their approach has one virtue – it’s easy to understand, and easy to do. But it has countless downsides, one of them being that it absolutely cannot prove – or even really suggest – what they’re saying it proves. I don’t know if the city’s charter teachers have higher value-added scores. It’s an interesting question (by my standards, at least), but the Daily News doesn’t address it meaningfully.

Though far from the only one, the reporters’ biggest problem was right in front of them. The article itself notes that only about half (32) of the city’s charter schools chose to participate in the rating program (it was voluntary for charters). This is actually the total number of participating schools in 2008, 2009 and 2010, most of which rotated in and out of the program each year. It’s apparently lost on these reporters that only a minority of charters participating means that the charter teachers in the TDR data do not necessarily reflect the population overall. This issue by itself renders their assertions invalid and irresponsible.

Simple thinking, and bad education reporting is a major impediment to real education reform that will improve the quality of our schools.

Why is it that a politician, such as Mayor Frank Jackson, can put forward plans to eliminate teacher seniority, and it not be pointed out that teaching experience matters, and that if his goal is to improve the quality of Cleveland's public schools, his chosen policy preference is antithetical to that?

Research suggests that students learn more from experienced teachers (those with at least five years of experience) than they do from less experienced teachers (NCES 2000d; Rivkin, Hanushek, and Kain 1998; Murnane and Phillips 1981.) These studies point primarily to the difference between teachers with fewer than five years of experience (new teachers) and teachers with five or more years of experience.

If that wasn't simple enough, of course there are even more complex answers to this question.

But these overall findings ignore the fact that the experience/achievement relationship differs a great deal by context. For instance, the returns to experience appear to vary by where teachers work. The relationship is more consistent among elementary school teachers (especially compared with those in high schools). The effect of experience on teacher productivity may also be mediated by the quality of their peers in the same school – i.e., that novice teachers with more effective peers in the same school do better.

Similarly, there is evidence that experience matters less – or less consistently – in poorer schools (also see here). There are several plausible explanations for this discrepancy, such as the possibility that teachers in poorer schools burn out more rapidly, or that there are difficulties in teaching lower-income children that are harder to adjust to.

The experience factor not only varies by where you teach, but also by what you teach. Math teachers seem to improve more quickly (and consistently) than reading teachers, while newer evidence suggests that the same is true for teachers who remain in the same grade for multiple years.

Finally, it bears mentioning the obvious point that the effect of teacher experience might be totally different if we were able to look at outcomes other than test scores. The idea that experience doesn’t matter after five or so years incorrectly implies that test scores are the only relevant outcome. Nobody believes that is the case. (And, for what it’s worth, teachers with whom I’ve spoken find the idea that they stop improving after four or five years laughable.)

Instead the debate over the Mayor's plan has not revolved around whether it has any basis in supportable fact, but instead around simplistic stories of the politics involved.

There are enough bad actors in the corporate school reform movement willing to put aside hard truths and solid facts in favor of their desire to profit from public education, but that should be no excuse for others to not challenge simplistic thinking and unsupported asertions which can be equally as damaging to the goal of delivering a quality education to all students.

A bridge too far

If you're a school administrator, wondering what your next budget is going to look like, waiting for the release of a new school funding formula, our advice is "don't hold your breath".

Ohio had a school funding formula. Strangley, it still has a website dedicated to it

After 20 years of controversy over its school funding system, Ohio now has a new method for providing funding to its public schools. Enacted as part of the 2010-2011 state budget, the Ohio Evidence-Based Model is designed to fund strategies that have the best chance to help students learn.

While economic realities require that the new approach be phased in over 10 years, the principles underlying the evidence-based model are now in place.

What's more, the new funding model is tied to education reforms designed to build a 21st-century system of education for Ohio.

Unlike the current Attorney General, Mike DeWine, who is winning plaudits for continuing and building upon much of the work of the previous administration, the Governor decided that everything the previous administration had done must go. Whether it worked or not. The Evidenced based model, which brought together hundreds of stakeholders and took years to develop was immediately scrapped. Replaced with a make-it-up-as-we-go-along "bridging formula". It is increasingly likely that a continuing "bridging formula" is on the horizon

But nearly a year later, Kasich, like governors before him, has found that overhauling the way Ohio funds education is not simple math.

The Republican administration concluded a series of public meetings on the issue in September but has yet to release a draft proposal promised for October. And now the governor’s office appears certain to miss a self-imposed deadline of January for unveiling its method of paying for Ohio schools.

Kasich spokesman Rob Nichols said the administration is working on its plan, and he doesn’t know when it will be ready.

We asked the Governor's education Czar, Bob Sommers, if he could provide some timetable guidance.

@RDSommers can you give us some guide as to when we might see a funding formula? Is it close, not close? Thanks!
@jointhefutureOH wish I could, but the issues are complex. We continue to study the possibilities. Ideas welcome

We suggested they look at successful models elsewhere in the country, but apparently they don't think there are any. We'd also suggest that they were a little trigger happy in shooting down the Evidence Based Model, and perhaps they could perform some CPR and bring it back with their own modifications.

Either way, the administration has clearly learned that this is no simple task with obvious answers.

Their difficulties will certainly have been further complicated by severe funding cuts as a result of HB153 raiding school budgets, and alienating most school districts and communities with bills like SB5 and HB136. It's hard to collaborate with hundreds of stakeholders when the previous 12 months have been spent attacking them and their mission.

If the administration have learned this lesson we should expect to see more outreach and consultation, and eventually arrive at a funding formula that works for most. Otherwise the administration is going to find itself having traveled a bridge too far.

Final note. We'd like to thank Bob Sommers for engaging in our questions with honest and forthright answers. While we sometimes disagree on fundamental policies, being able to have open and honest policy dialogue is our number one goal, his efforts in this repect advance that.

Simple facts about Issue 2 (SB 5)

From our mailbag, some simple facts about Issue 2 (Senate Bill 5):

  • SB 5 weakens collective bargaining rights for police officers, teachers, nurses and prison guards. It takes away their ability to negotiate for health care, retirement, and sick time.
  • It makes our communities less safe because police officers and firefighters could lose their ability to negotiate for safer staffing ratios and better equipment.
  • It threatens the quality of our schools. SB 5 prohibits school districts from negotiating with teachers for smaller class sizes.

It's time to repeal Senate Bill 5. Vote no on Issue 2!

Teaching isn't as simple as it appears

Ever since Gov. John Kasich barely beat the red light in last November's election, his school-reform bus has been careening at breakneck speed, rolling over all in its path. But no one is in sight to flag him for reckless driving.

He and other public-education "reformers" are transferring millions of tax dollars from public schools to less scrutinized charters and private schools. They are dissing classroom teachers by taking away both their dignity and their voices at the bargaining table, while watering down teacher-license requirements and dancing to the tune of the highly paid elites from tax-exempt foundations.

If the governor and our lawmakers can escape their fact-free zones and policy gurus, they might visit public schools - to listen, watch and learn what it really takes to teach children.

[readon2 url=""]Continue reading...[/readon2]