A great top 10 research based list of the problems of using Value added modelling to sort and rank teachers, via Vamboozled .
Via Nicholas Meier
- VAM estimates should not be used to assess teacher effectiveness. The standardized achievement tests on which VAM estimates are based, have always been, and continue to be, developed to assess levels of student achievement and not levels growth in student achievement nor growth in achievement that can be attributed to teacher effectiveness. The tests on which VAM estimates are based (among other issues) were never designed to estimate teachers’ causal effects.
- VAM estimates are often unreliable. Teachers who should be (more or less) consistently effective are being classified in sometimes highly inconsistent ways over time. A teacher classified as “adding value” has a 25 to 50% chance of being classified as “subtracting value” the following year(s), and vice versa. This sometimes makes the probability of a teacher being identified as effective no different than the flip of a coin.
- VAM estimates are often invalid. Without adequate reliability, as reliability is a qualifying condition for validity, valid VAM-based interpretations are even more difficult to defend. Likewise, very limited evidence exists to support that teachers who post high- or low-value added scores are effective using at least one other correlated criterion (e.g., teacher observational scores, teacher satisfaction surveys). The correlations being demonstrated across studies are not nearly high enough to support valid interpretation or use.
- VAM estimates can be biased. Teachers of certain students who are almost never randomly assigned to classrooms have more difficulties demonstrating value-added than their comparably effective peers. Estimates for teachers who teach inordinate proportions of English Language Learners (ELLs), special education students, students who receive free or reduced lunches, and students retained in grade, are more adversely impacted by bias. While bias can present itself in terms of reliability (e.g., when teachers post consistently high or low levels of value-added over time), the illusion of consistency can sometimes be due, rather, to teachers being consistently assigned more homogenous sets of students.
- Related, VAM estimates are fraught with measurement errors that negate their levels of reliability and validity, and contribute to issues of bias. These errors are caused by inordinate amounts of inaccurate or missing data that cannot be easily replaced or disregarded; variables that cannot be statistically “controlled for;” differential summer learning gains and losses and prior teachers’ residual effects that also cannot be “controlled for;” the effects of teaching in non-traditional, non-isolated, and non-insular classrooms; and the like.
- VAM estimates are unfair. Issues of fairness arise when test-based indicators and their inference-based uses impact some more than others in consequential ways. With VAMs, only teachers of mathematics and reading/language arts with pre and post-test data in certain grade levels (e.g., grades 3-8) are typically being held accountable. Across the nation, this is leaving approximately 60-70% of teachers, including entire campuses of teachers (e.g., early elementary and high school teachers), as VAM-ineligible.
- VAM estimates are non-transparent. Estimates must be made transparent in order to be understood, so that they can ultimately be used to “inform” change and progress in “[in]formative” ways. However, the teachers and administrators who are to use VAM estimates accordingly do not typically understand the VAMs or VAM estimates being used to evaluate them, particularly enough so to promote such change.
- Related, VAM estimates are typically of no informative, formative, or instructional value. No research to date suggests that VAM-use has improved teachers’ instruction or student learning and achievement.
- VAM estimates are being used inappropriately to make consequential decisions. VAM estimates do not have enough consistency, accuracy, or depth to satisfy that which VAMs are increasingly being tasked, for example, to help make high-stakes decisions about whether teachers receive merit pay, are rewarded/denied tenure, or are retained or inversely terminated. While proponents argue that because of VAMs’ imperfections, VAM estimates should not be used in isolation of other indicators, the fact of the matter is that VAMs are so imperfect they should not be used for much of anything unless largely imperfect decisions are desired.
- The unintended consequences of VAM use are continuously going unrecognized, although research suggests they continue to exits. For example, teachers are choosing not to teach certain students, including those who teachers deem as the most likely to hinder their potentials to demonstrate value-added. Principals are stacking classes to make sure certain teachers are more likely to demonstrate “value-added,” or vice versa, to protect or penalize certain teachers, respectively. Teachers are leaving/refusing assignments to grades in which VAM-based estimates matter most, and some teachers are leaving teaching altogether out of discontent or in protest. About the seriousness of these and other unintended consequences, weighed against VAMs’ intended consequences or the lack thereof, proponents and others simply do not seem to give a VAM.
The rhetoric behind vouchers is that if everyone had vouchers parents could select the best school for their child instead of being forced to go to “government” schools*.
Where does such logic fall apart? There are two main logistical reasons it is really a false promise. One is economic and the other is question of who gets to choose.
The private schools that the elite send their children to cost tens of thousands of dollars a year to attend. I looked up a few progressive private schools and tuition ranged from $20,000 to well over $30,000, more than many private colleges. And the actual amount they spend per pupil is well over the tuition since they raise lots of extra money from alumni. (They also tend to pay their non-unionized teachers significantly less than public schools.)
Since at best the voucher proposals I have seen only pay a small fraction of that, these vouchers will leave the recipients with few real choices without putting out a lot more money. I do not think the public is going to go for vouchers of $20,000+ and have never even heard such figures discussed. If they did, the public education bugets would soar. (And those already in private schools would and should claim they should get the subsidies too). What it would do in effect, at the rates being proposed, is subsidize the middle class and rich to abandon public schools and send their children to private school, and while leaving such choices out of reach financially for the poor.
The other issue is who chooses. Most private schools have selective admission, and limited space. Since unlike public schools they get to choose their students, even if the voucher fully paid for them (which of course it will not), they would still most likely cream the easiest students to teach, leaving the more difficult to teach children in the public schools.
These two factors in combination would end up subsidizing private schools and middle and upper class families at the expense of public schools and the poor that are left in them. This would further segregate our schooling system into the haves and the have-nots.
Since I have never heard voucher proponents either suggest that vouchers should be at the levels necessary to have them cover the full cost of most private schools, nor to force private schools to take those children, I find their arguments disingenuous.
Charter schools, in theory at least, get around both of the above limitations. There is no tuition; schools receive the same funding as the other public schools, and (at least in California) schools cannot select the students. (In reality, though, they often find ways of using other means to “encourage” and “discourage” certain types of students.) So, is this not a solution?
Why I still do not favor even this is that it fundamentally changes the purpose of public schools. Traditionally we have considered the education of the next generation to be a concern of society as a whole. In fact, virtually every society has considered this to be true throughout history. For this reason, locally elected school boards have governed our public schools.
Charter schools and voucher systems make schooling a private consumer choice. In the charter and voucher systems consumers choose among the choices offered them, but have no guaranteed right to have a say about the schooling other than making that choice. Those who do not have children in the schools have no say at all. Private schools are run privately, and do not have to answer to the public. Charter schools usually have to answer for test scores and financial responsibility, but even there it is to the state and not in any direct way to the local public. While charter schools have governing boards, they select their own members of those boards. This gives control of the content of schooling to those who run the schools, often for-profit concerns, but even if not, private concerns of some sort. While our government is not perfect, should I really trust those who have private agendas and do not have to answer to the public to decide the how and what of our next generation’s schooling? Public school boards are elected, and have open meetings; private schools do not have to. Even if the charters do have open meetings, they are often run by national organizations and so are inaccessible and would probably just say, “Don’t send you child here if you don’t like it our agenda.”
Vouchers and charters are about redefining the public as consumers rather than citizens, which is part of a larger corporate agenda to destroy public institutions and the limit the power of the public.
In Ohio, at least 15 charter schools have abruptly closed this year – most don’t even bother to list a reason.
In Detroit, a city wracked by debt and bankruptcy, officials scrambled to close a failed charter school by Oct. 31 this year, due to the school’s debts, which exceeded $400,000.
According to The Washington Post, Washington, D.C., spent over $1 million on closing failed charter schools from 2008-2012.
More cities are following the lead of districts like Chicago, where the largest shutdown of public schools in the nation’s history occurred at the very same time that new private charter schools were being expanded by the district.
Abruptly opening and closing schools – leaving school children, parents and communities in the lurch and taxpayers holding the bag – is not a matter of happenstance. It’s by design.
The design in mind, of course, is being called a “market.” Parents and taxpayers who used to rely on having public schools as anchor institutions in their communities – much like they rely on fire and police stations, parks and rec centers, and the town hall – are being told that the education of children is now subject to the whims of “the market.”
The supposed benefit to all this is that parents get a “choice” about where they send their children to school. But while parents are pushed to pick their schools on the increasingly turbulent bazaar of “choice,” the game resembles much less a level playing field and much more a game of chance in which the house rules determine the odds. And too many of the nation’s families – and their communities – are getting caught up in a crapshoot with our children’s education at stake.
Whether from charters or voucher-funded private schools, the explosive growth of crapshoot schools is fast becoming the norm. And too few are asking, “At what risks?”
The US Department of Education released the latest NAEP (National Assessment of Educational Progress) results a short while ago. Despite all the corporate education reforms that have been pushed in Ohio, Ohio’s students continued to score marginally highher than the national average, but haven’t shown any significant improvement over previous results. In other words, for all the additional workload, stress and controversy, corporate reforms are not producing increased performance results.
What we are seeing though is the utter perfoermance disaster of Charter schools in the state. The Fordham Foundation crunched the numbers to compare traditonal pblic schools with tier low performing charter school counterparts
The figures below display the charter versus non-charter comparison of students who are eligible for the Free and Reduced Price Lunch (FRPL) program, the most-utilized poverty metric available. This provides a fair comparison of similar students, since Ohio’s charters enroll a relatively high number of impoverished students.
Here's Fordhams findings in graphiocal format
The results from this snapshot in time are not favorable to charter schools. In all four grade-subject combinations, charter school NAEP scores fall short of the non-charter school scores. And in all cases, I would consider the margin fairly wide—more so in 4th than 8th grade. In 4th grade reading, for example, non-charter students’ average score was 211, while charter students’ average score was 191, a 20 point difference.
The only question that remains is, what is to be done about this performance disaster?
We were reading through the latest issue of Ohio Schools Magainze, and came upon this terrific article by a retired teacher, on her experiences working in a right to work state. Here’s that article.
OEA-Retired Member Dawn Wojcik Offers Insight About Why These Laws Are Harmful And What To Expect If They Are Enacted In Ohio.
In 1981, when Dawn wojcik began her teaching career, Ohio’s economy was faltering. As a recent graduate from the University of Dayton with a Masters degree and teaching credentials, she applied for every English teacher opening in the state. She was the Avis candidate—trying harder, yet coming in second place in the 10 positions for which she interviewed.
As the school year started, she found herself living at home with her parents and looking forward to subbing until something better turned up.
A week into the school year, Wojcik received a phone call from the personnel director for the Franklin Parish Schools in Louisiana. The parish (county) needed a ninth-grade English teacher at Morgan City High School. Wojcik was hired over the phone based solely on her resume and transcripts.
She was excited at the prospect of her first teaching position, but unprepared for the realities of working as an educator in a so-called Right to Work state.
Wojcik soon learned that despite the name, Right to Work laws limit rights, meaning that educators have less power to advocate for student learning conditions and educator working conditions.
Public education takes a hit in SCRTW states as teachers move to other states with lower class sizes, better salaries and benefits, and more job security. School funding suffers, too. States with Right to Work laws spend $3,392 less per pupil on elementary and secondary education. And without the ability to bargain for better teaching and learning conditions, academic achievement declines. The majority of states with SCRTW laws are among the lowest performing in the nation.
“There was a shortage of teachers in Louisiana’s “oil parishes.” I was thrilled to have a job, especially one with a beginning salary of $15,100 a year $400 more a year than the best paying position I had applied for in Ohio. These higher salaries were possible because of taxes on the oil drilled
in the parish. Teachers had not bargained these salaries. It was the luck of being in an “oil parish” that provided the resources for these salaries. Salaries throughout the rest of the state were much lower.
My first day on the job, I received grammar and literature textbooks. I asked my department chair where I should start. She thumped her arthritic finger into her desk and, in a southern drawl, said, “I always start with the verb” as she glared at me from her desk. I never received any additional direction from her.
There was no effort to mentor new teachers. We were our own to do the best we could without guidance or support. After a few months in the classroom, I received some instruction on how to write lesson plans. I learned that I was to identify Pupil Performance Objectives in my lesson plans. All teachers were to be teaching to these same standardized objectives. These objectives were to be on my desk and available for administrators to see at all times.
I didn’t realize how important being a part of the education association was until I worked as a professional at Northmont.
In Louisiana, the limited role I had in the education profession was directly related to legal limitations on bargaining and educators’ ability to organize. My first three years as a teacher, I was left to sink or swim as an employee. because of collective bargaining in ohio, I was actively involved as an education professional for 27 years.
Here was no support for teachers, no mentors, no professional development. And there was no mechanism for teachers to have input for our profession.
As money tightened up, there was talk that franklin Parish would increase class sizes and close some of its schools. My colleagues were upset. Talk in the workrooms encouraged us all to go to the school board meeting. enough teachers showed up to pack the boardroom. But no one spoke up with our concerns.
After listening at the school board meeting, the self-appointed leaders determined that there was nothing we could do short of a statewide strike. They concluded that even that action would be fruitless. As we faced this financial crisis, we had no avenue to communicate our students’ needs to those who made the decisions about the learning conditions for our students. It was discouraging to not have a voice. Working in a No Rights at Work state made us powerlessness in this situation.
My $15,100 Louisiana starting salary had impressed me until I had some time to study the salary schedule.
It was then that I saw a different story. While my starting salary was higher in Louisiana than in Ohio, even with a Ph.D. and 30 plus years of experience, the top of the salary schedule would be no more than $22,000.
During my second year in Louisiana, I began applying for jobs in Ohio. I was fortunate that by the end of my third year, Northmont City Schools hired me, and I returned home to Ohio in 1984 to teach at Northmont high School.
At the time I was hired, the Northmont District education Association (NDeA) was preparing to take a strike vote. The local leader- ship took time to meet new teachers, patiently answering our questions about the bargaining crisis. I joined in order to have a vote. We voted to give a 10-day strike notice, an action that brought the parties back to the table and resulted in significant improvements in the contract.
With the new Northmont contract, I would be making nearly $4,000 more a year than I had been after three years in Louisiana. I had better medical insurance and vision and dental coverage. Not only were the financial rewards of returning to my home state significantly better but also, because of the leadership of my local, I was welcomed as a new teacher. Other teachers shared their lesson plans and materials. The school district offered professional development throughout the year. Our collective bargaining agreement called for a labor management team that met regularly to share concerns through- out the district. Our NDeA held a district-wide Monte Carlo event to raise money for student scholarships to summer enrichment programs and college.
I worked for 27 years at Northmont high School and was always treated like a professional. This was a huge contrast to the experience I had at Morgan City high School in Louisiana, a so-called Right to Work (or as I say, a No Rights at Work) state.
Recently, I checked to see what I would have earned at the end of my career in Louisiana and what that would have meant for my retirement income. had I stayed in Louisiana, my retirement would be about one third less than what it is now.
It seems the Rube Goldberg system of teacher evaluations in Ohio might be getting some improvements. We have written extensively about the unfairness of basing so much of a teachers evaluation on Value-add, when it is subject to so much statistical variation, inaccuracies, and student demographics. In the recent budget, educators pressed the legislature to reduce the Value-add component form its current 50% to 35%. The Senate included language to do just that, but the House, under direction from the Governor stripped it out.
Now Senator Gardner, along wth 3 cosponsors, Manning, Lehner, and Hite has introduced SB 229 which once again seeks to make OTES fairer. Along with making OTES a little fairer, it also seeks to reduce the administratvie burden on distrcits by reducing the amount of observation required for "skilled" and "accomplished" teachers. We wrote about this massive burden over 2 years ago.
Dawn Wojcik, Retired Teacher
HEre's what SB229 contains
Educators should contact their representatives and urge them to support SB229, otherwsie it is possible that the Governor will once again kill these needed changes. You can read the bill for yourself, here.
- Lowers the academic growth factor percentage required on teacher evaluations to 35% from the current 50%. A school district may attribute an additional percentage to the academic growth factor not to exceed fifteen percent of an evaluation. The academic growth factor under the OTES is based on value-added and/or other student growth measures, depending on the subjects and grades in a teacher’s course load.
- Authorizes local school boards to reduce the frequency of evaluations required for teachers who receive an evaluation rating of “Skilled” to once every 2 years and “Accomplished” once every 3 years.