VAM-Based Decisions Are Less Reliable Than Flipping a Coin

A new study of Value Added Measures (VAM) once again finds the use of this statistical tool to be inappropriate for measuring the effectiveness of teachers.

The question of stability [reliability/consistency] is not a question about whether average teacher performance rises, declines, or remains flat over time. The issue that concerns critics of VAM is whether individual teacher performance fluctuates over time in a way that invalidates inferences that an individual teacher is “low-” or “high-” performing. 

This distinction is crucial because VAM is increasingly being applied such that individual teachers who are identified as low-performing are to be terminated. From the perspective of individual teachers, it is inappropriate and invalid to fire a teacher whose performance is low this year but high the next year, and it is inappropriate to retain a teacher whose performance is high this year but low next year. 

Even if average teacher performance remains stable over time, individual teacher performance may fluctuate wildly from year to year.

After looking at numerous studies of VAM, the author concludes...

What this means is that value-added teacher rankings are insufficiently reliable for the purpose of high-stakes decisions regarding hiring and firing. High-Stakes decisions are clearly unwarranted if this volatility in the rankings is due to unmeasured variables or random measurement error. 

However, even in the unlikely event that there are no unmeasured variables and measurement error is zero, implying that all volatility is due to true variation in teacher performance, it would not be appropriate to hire or fire based on the ranking in a given year (designated “year t+1”) by such an extent as to invalidate the year t ranking. 

If VAM is used to identify and fire the bottom quartile (or quintile) of teachers, the results in Tables 1 and 2 indicate that this decision is incorrect, according to the year t+1 teacher rankings, between 59 and 70% of the time. If VAM-based culling is less reliable than flipping a coin, as these results suggest, then productive teachers would be culled more frequently than unproductive bottom quartile (or bottom quintile) teachers.

The whole notion that you could measure a teacher the same way a farmer measures a pig was always insulting and ridiculous. The ongoing science continues to prove that.

Here's the full paper


What Justice Scalia's Passing Means for Teachers

Setting aside the politics of replacing Justice Antonin Scalia after his unexpected death, what of the issues already before the court that may affect public education and beyond?

The biggest case, widely reported is Friedrichs v California Teachers. SCOTUS Blog describes the issue thusly

“Issue: (1) Whether Abood v. Detroit Board of Education should be overruled and public-sector “agency shop” arrangements invalidated under the First Amendment; and (2) whether it violates the First Amendment to require that public employees affirmatively object to subsidizing nonchargeable speech by public-sector unions, rather than requiring that employees affirmatively consent to subsidizing such speech.”
— http://www.scotusblog.com/case-files/cases/friedrichs-v-california-teachers-association/

Simply put, right wing billionaires bankrolled an attack on the rights of unions to charge fair share fees for representing all members in a public sector bargaining unit - such as teachers. Without the ability to charge fair share fees, non members could simply not pay anything but still benefit from the services of the union - everything from increased pay via bargaining to better benefits, safety and training.

It was widely expected that the Supreme Court with it's 5 conservative justices would rule 5-4 against decades of precedent and deal a blow to unions. With the death of one of those 5 conservative justices it is now believed that the court would be tied 4-4.

What are the implications of this?

  1. Votes that the Justice cast in cases that have not been publicly decided are void.  
  2. In a 4-4 case where there is no majority for a decision the lower court’s ruling stands, as if the Supreme Court had never heard the case
  3. The lower court ruled in favor of the unions (Ironically, something Friedrichs wanted so that the case could be expedited to the Supreme Court)

So as it stands right now, fair share, underpinned by the longstanding Abood decision remains the law of the land.

For this to continue to stand in the years to come, it will be important for supporters of strong unions and working people to ensure the replacement for Scalia supports the pillars of fairness that labor is built upon.

Senator Recommends Districts use "Self-Reflection" to Reduce Pay-to-Play Fees

It has been repeatedly demonstrated that the biggest impediment to a student's school achievement is the poverty level to which that student is exposed. It's why public education advocates have called for an equitable funding system so that students in districts with high levels of poverty can have access to the same opportunities as their more prosperous brethren.

One of the ways this inequity shows up is in pay-to-play fees. Only the students with parents who can afford to pay get to play. Pay-to-play fees have become an increasing mainstay as districts have had to find ways of dealing with budget cuts, especially state cuts, without affecting "core curriculum".

Senator Cliff Hite (R) has released findings stemming from a statewide tour that garnered community insights on participation fees for extra- and co-curricular activities in Ohio schools. The full white paper can be read here.

Here's the 1 page sumary

Sen. Hite had this to say about his report

“We can all agree on one thing: extra- and co-curricular activities play an important role in the overall development of students, both inside and outside the classroom. It was important that we heard from all impacted parties in order to best address this issue as the goal has always been to ensure Ohio’s students are given the greatest opportunities to succeed now and throughout their lives.”
— http://ohiosenate.gov/hite/press/hite-shares-findings-from-statewide-tour-on-student-participation-fees

While we can applaud Sen. Hite for promoting this issue, it's frustrating that the single best solution to this problem, and one which the General Assembly of which he is a member has the greatest control is not mentioned. The General Assembly should increase funding to public schools, and do so in an equitable way.

Instead we have the recommendation that schools should "self-reflect", which frankly is a pathetic recommendation that will lead to no changes. Self reflection doesn't pay for education - money does.

US Department of Education's Belated Fake Epiphany

The US Department of Education has had a belated epiphany. After years of using waivers to beat down educators, they've realized it was counter productive and has only served to destroy morale within the profession. USA today reports:

In his first major speech, the acting U.S. Secretary of Education John King apologized to the nation’s teachers.

Speaking to a small group of teachers, students and local politicians here last month, just three weeks after taking over the post, King admitted the USA’s education debate over the past few years has been “characterized by more heat than light,” and that despite reformers’ best intentions, “teachers and principals, at times, have felt attacked and unfairly blamed for the challenges our nation faces.”

King acknowledged the attacks had come from as high up as his own federal agency in Washington, D.C., where he’d served as a top advisor to former Education Secretary Arne Duncan for more than a year.

"All of us — at the local, state, and federal level, the Education Department included — have to take responsibility for the climate that exists," he said. ”There is no question that the contentious tone has made it harder to have productive conversations."

This new outlook is likely less a real epiphany and more a reaction to the removal of dictatorial powers the US Department of Education was wielding when the Every Student Succeeds Act was signed into law.

ESSA radically reduces the U.S. Department of Education's authority over state curriculum, standards and testing and curtails the secretary of education's authority over state and local policy making. The law states:

Nothing in this title shall be construed to authorize the Secretary or any other officer or employee of the Federal Government to mandate, direct, or control a State, local educational agency, or school's instructional content or materials, curriculum, program of instruction, academic standards, or academic assessments; teacher, principal, or other school leader evaluation system; specific definition of teacher, principal, or other school leader effectiveness; or teacher, principal, or other school leader professional standards, certification, or licensing.

King has so little influence now that all he is left with is to show respect and do some listening. Something the Department of Education should have been doing all along - instead of listening to the whims of billionaire corporate reformers.

ODE Understated Number of Failing Charter Schools by 950%

When ODE Applied for a Federal grant to expand even further the number of charter schools it Ohio, it lied on its application form. they were quickly caught, because no one was going to believe only 6 charter schools in the entire state were failing. Embarrassed and forced to correct their lies, the new wishful thinking number they have come up with is 57 - a 950% increase in failing charter schools in the state.

They have also cut the number of schools they claimed were successful by 50% - down to just a paltry 59.

According to ODE's own performance data, the latest of which that is available is for 2013-14, even these numbers seem suspect.

Performance Rating of Ohio's Charter schools 2013-14.

Performance Rating of Ohio's Charter schools 2013-14.

More than half of all Charters in Ohio are rated D or below, and those rated A? Barely perceptible on the chart.

What Are Standardized Tests Measuring?

If the same test produced different results depending upon whether it was taken online or on paper, what exactly is being measured? Here's ODE recently

And here's an article from Real Clear Education titled "PARCC Scores Lower for Students Who Took Exams on Computers"

Students who took the 2014-15 PARCC exams via computer tended to score lower than those who took the exams with paper and pencil—a revelation that prompts questions about the validity of the test results and poses potentially big problems for state and district leaders.

Officials from the multistate Partnership for Assessment of Readiness for College and Careers acknowledged the discrepancies in scores across different formats of its exams in response to questions from Education Week.

“It is true that this [pattern exists] on average, but that doesn’t mean it occurred in every state, school, and district on every one of the tests,” Jeffrey Nellhaus, PARCC’s chief of assessment, said in an interview.

“There is some evidence that, in part, the [score] differences we’re seeing may be explained by students’ familiarity with the computer-delivery system,” Nellhaus said.

In general, the pattern of lower scores for students who took PARCC exams by computer is the most pronounced in English/language arts and middle- and upper-grades math.

ODE would be better taking the time to understand exactly what all of its tests are actually measuring before getting too excited about how many students took paper vs online tests.