Teacher performance incentives have negative impact

A recently published study from the Journal of Labor Economics looked at performance incentives for teachers in the NYC school system. The results should cause corporate reformers to pause
Providing financial incentives for teachers to increase student performance is an increasingly popular education policy around the world. This paper describes a school-based randomized trial in over two-hundred New York City public schools designed to better understand the impact of teacher incentives on student achievement. I find no evidence that teacher incentives increase student performance, attendance, or graduation, nor do I find any evidence that the incentives change student or teacher behavior. If anything, teacher incentives may decrease student achievement, especially in larger schools. The paper concludes with a speculative discussion of theories that may explain these stark results.
As Margarita Pivovarova, Assistant Professor of Economics at Arizona State University notes
The estimates from the experiment imply that if a student attended a middle school with an incentive in place for three years, his/her math test scores would decline by 0.138 of a standard deviation and his/her reading score would drop by 0.09 of a standard deviation.

Not only that, but the incentive program had no effect on teachers’ absenteeism, retention in school or district, nor did it affect the teachers’ perception of the learning environment in a school. Literally, the estimated 75 million dollars invested and spent brought zero return!

Share your comment:

How much variance in test scores is due to variance in teachers?

Via Larry Ferlazzo

It’s not uncommon to hear someone inaccurately state that the teacher has the biggest influence on student achievement — period. Of course, the true statement is that — of the in-school factors — teachers have the biggest influence. On top of that, research has shown that over two-thirds of the factors that influence student achievement occur out of school.

To illustrate this, here's a pie chart

Lots of good links ot dem,onstrate the evidence behind this pie chart at the link.

If we're primarily focusing on teacher quality to increase student achievment, we're focusing on the wrong place. It's why corporate education reforms are doomed to failure.

Share your comment:

Can Value-Added Measures Be Used for Teacher Improvement?

Susanna Loeb, Professor of Education Stanford University and Faculty Director for the Center for Education Policy Analysis, has a brief published by the Carnegie Foundation for the Advancement of Teaching
The question for this brief is whether education leaders can use value-added measures as tools for improving schooling and, if so, how to do this. Districts, states, and schools can, at least in theory, generate gains in educational outcomes for students using value-added measures in three ways: creating information on effective programs, making better decisions about human resources, and establishing incentives for higher performance from teachers. This brief reviews the evidence on each of these mechanisms and describes the drawbacks and benefits of using value-added measures in these and other contexts.

The brief concludes
    Value-added measures are not a good source of information for helping teachers improve because they provide little information on effective and actionable practices.
  • School, district, and state leaders may be able to improve teaching by using value-added to shape decisions about programs, human resources, and incentives.
  • Value-added measures of improvement are more precise measures for groups of teachers than they are for individual teachers, thus they may provide useful information on improvement associated with practices, programs or schools.
  • Many incentive programs for staff performance that are based on student performance have not shown benefits. Research points to the difficulty of designing these programs well and maintaining them politically.
  • Value-added measures for selecting and improving programs, for informing human resource decisions, and for incentives are likely to be more useful when they are combined with other measures.
  • We still have only a limited understanding of how best to use value-added measures in combination with other measures as tools for improvement.
The use of value-added measures to evaluate individual teachers, especially when connected to high stakes personnel decisions is impossible to defend if one is guided by the research evidence, and the disastrous practical applications.

The full brief can be read below.


Share your comment:

Bogus Evaluations Used to Fire Teachers

Washington DC has long been the poster child for high stakes tests used to label teachers as successes or failures. Now news comes that errors in the Value-add formulas used to measures theses apparent successes or failures resulted in 44 teacher being incorrectly labelled, and as a consequence, 1 teacher was fired.
More than 40 teachers in D.C. public schools received incorrect evaluations for 2012-2013 because of errors in the way the scores were calculated and one was fired as a result.

The president of the Washington Teachers’ Union, Elizabeth A. Davis, has asked for details from D.C. Schools Chancellor Kaya Henderson in a letter (text below) that says that the problems were found by Mathematica Policy Research, a partner of the school system’s. The mistakes were found in the individual “value added” scores for teachers, which are calculated through a complicated formula that includes student standardized test scores.

This “VAM” formula is part of the evaluation system called IMPACT, begun under former chancellor Michelle Rhee in 2009. Henderson, Rhee’s successor, continued with IMPACT, though this year she reduced the amount of weight given to test scores from a mandatory 50 percent to at least 35 percent. (See below for IMPACT chart).

Testing experts have long warned that using test scores to evaluate teachers is a bad idea, and that these formulas are subject to error, but such evaluation has become a central part of modern school reform.
44 teachers may not sound like a lot, but it turns out it was a significant percentage of teachers
Those affected are about 1 percent of about 4,000 teachers in the school system. But they comprise nearly 10 percent of the teachers whose work is judged in part on annual city test results for their classrooms.
When an evaluation system is so poor that 1 in 10 results are error riddled and result in a teacher wrongfully being terminated, the system needs to be put on hold. Here in Ohio, the results could be even worse.

Ohio uses a secret proprietary for-profit formula

Some of the confusion may be due to a lack of transparency around the value-added model.

The details of how the scores are calculated aren't public. The Ohio Department of Education will pay a North Carolina-based company, SAS Institute Inc., $2.3 million this year to do value-added calculations for teachers and schools. The company has released some information on its value-added model but declined to release key details about how Ohio teachers' value-added scores are calculated.

The Education Department doesn't have a copy of the full model and data rules either.

The department's top research official, Matt Cohen, acknowledged that he can't explain the details of exactly how Ohio's value-added model works. He said that's not a problem.

"It's not important for me to be able to be the expert," he said. "I rely on the expertise of people who have been involved in the field."
If something similar were to happen in Ohio, which is highly probable, no one would be any the wiser, because no one can double check the work of SAS Institute Inc., not even ODE - which remarkably doesn't even seem to care.

The formula for calculating the value-add score for Ohio's teachers must be open to inspection so that our teachers are not falsely named and shamed and fired as they are being in Washington DC.

Share your comment:

For Women, Unionism is Huge Boost

A recent study by the Center for Economic and Policy Research looked into the effects of unionism on women in the workplace. The results of women being in a union compared to non unionized women, even with a college degree, are dramatic.

Being in or represented by a union compared well to completing college in terms of wages, especially once tuition costs are factored in. All else equal, being in a union raises a woman's pay as much as a full year of college does. For the average female worker, a four-year college degree boosts wages by over half (51.9 percent) relative to a similar woman who has only a high school degree. In comparison, unionization raises a woman’s pay by 14.7 percent – over one-quarter of the effect of a college degree.

The union impact on the probability that a female worker has health insurance or a retirement plan through her employer was even larger than the impact on wages. At every education level, unionized women are more likely to have employee benefits than their non-union counterparts with similar characteristics. In fact, for a women worker with a high school degree, being in or represented by a union raises her likelihood of having health insurance or a retirement plan by more than earning a four-year college degree would.

The union advantage is largest when looking at employer-provided retirement plans. Women in or represented by unions are 53.4 percent more likely to have pension coverage than those not in unions, which is also larger than the corresponding effect of a four-year college degree (43.6 percent).

You can read the full report here. One thing is clear, so-called "right-to-work" is very bad for everyone, but especially women.

Share your comment:

What to Expect in Ed in 2014

With 2014 just beginning we thought it would be useful to lay down a marker on what to expect in education this year. In no particular order, then.

The 3rd Grade Reading Guarantee

This list may be in no particular order, but one of the first big issues we're likely to see is large numbers of 3rd graders failing their high stakes reading test and having to repeat 3rd grade. According to ODE, more than 1/3 of 3rd graders recently failed the state reading test this fall. We expect the number of failures in the spring to be somewhat lower than 1/3, but it is still going to be a substantial number, unevenly spread-out across the state. Urban and rural districts will be especially at risk.

The impacts of this policy will be severely felt this summer as districts scramble to find the resources to coach up students in order to promote them, and then in the new school year having to deal with a much larger 3rd grade cohort, and a smaller 4th grade class. Many elementary teachers (and teachers with reading endorsements) are going to find themselves being shuffled around.

Parents of students who are retained are not going to be happy either, and lawmakers are undoubtedly going to hear from the education community and parents seeking more flexibility and local control - something the legislature is going to find hard to resist.

Teacher Evaluations
Districts will continue to struggle implementing OTES, with SLO creation and resources for the non-test related components stressing past breaking point. A large number of districts (especially none RttT) have delayed implementation work on OTES, that tactic isn't going to be sustainable. The legislature has an opportunity to relieve some of the unnecessary burden by passing SB229 (which passed the Ohio senate unanimously late in 2013).
*Academic growth factor: Lowers the academic growth factor percentage required on teacher evaluations to 35% from the current 50%. A school district may attribute an additional percentage to the academic growth factor not to exceed fifteen percent of an evaluation. The academic growth factor under the OTES is based on value-added and/or other student growth measures, depending on the subjects and grades in a teacher’s course load.

*Frequency of evaluations: Authorizes local school boards to reduce the frequency of evaluations required for teachers who receive an evaluation rating of “Skilled” or “Accomplished” (the top two ratings).

Word is that the House is having a hard time reconciling the fact this is something they must do. If they fail to pass SB229, or water the relief down, then OTES is almost certain to collapse under its own weight.

Common Core and the online PARCC assessments that come with it require schools to have substantial and robust technology in place to deliver these online tests and handle the massive amounts of data that is going to be flowing as a consequence. The legislature, once again, has failed to recognize the scale of this endeavor and provide adequate funding. Just $10 million has been set aside - a drop in the ocean for over 600 districts to purchase technology and upgrade infrastructure to handle the bandwidth requirements.

ODE recently reported on their technology survey of districts, and the news wasn't good. 1/3 of respondents said they weren't going to be ready, and a staggering 50% more didn't even respond.

What this means is that chaos is going to ensue. Schools, with limited technology resources, are going to be on weeks long rotation of testing, likely with outages and serious downtime problems. A large number will simply be performing paper and pencil testing - causing Ohio to have a 2 track testing system. An online testing regime for rich districts and pencil and paper for the poorer.

We envisage the legislature delaying the requirements by at least a year, and having to commit serious resources to technology purchases in the next budget


Charter Schools
The charter school boondoggle is now a $1 billion a year business. Big enough that their catastrophic failure is becoming a mainstream issue. Tax payers and parents are noticing that the charter school promise is an empty one, and instead of providing more quality choices and competition, is siphoning resources away from higher performing public schools who are having to curb curriculum, institute pay-to-play, and delay building upgrades.

2014 will see calls to reform Ohio's charter schools laws gain in volume and diversity. The legislature, paralyzed by the millions of dollars in campaign contributions from charter operators, will try to resist calls for reforms. It will be the seminal fight over public education in Ohio over the next few years.

Common Core
Perhaps the hardest policy area to predict is Common Core. It's a complex issue tied up with standards and testing and something for everyone to hate, and perhaps like. The development of Common Core and its implementation has been a huge disaster. Far too little input from educators and parents, not enough early explanation of what it is and why it is needed. This has led to all manner of crazy conspiracy theories and very real distrust.

We suspect as different states react differently to Common Core implementation, we're likely to see less standardization adopted, and a slow down in implementation.

Corporate reformers are going to be on the other end of the accountability stick moving forward. Their ideas and policies are going to be scrutinized against their original claims. Given that most corporate education reform isn't supported by sound research and on the ground findings, they are going to be found wanting. One thing's for sure, the "no excuses" gang are going to be offering a lot of their own excuses.

Join the Future will be here for 2014 documenting, analyzing and reporting on all these issues, and many, many more.

Share your comment:

Get Involved

 First Name
 Last Name
 School District