Reliability and Validity of Inferences About Teachers Based On Student Test Scores

A report titled "Reliability and Validity of Inferences About Teachers Based On Student Test Scores" by Edward H. Haertel of Stanford University, published by ETS Research & Development Center for Research on Human Capital and Education Princeton, becomes yet another paper that casts grave doubt on the use of Value-add for the purposes of evaluating teachers
Policymakers and school administrators have embraced value-added models of teacher effectiveness as tools for educational improvement. Teacher value-added estimates may be viewed as complicated scores of a certain kind. This suggests using a test validation model to examine their reliability and validity. Validation begins with an interpretive argument for inferences or actions based on value-added scores. That argument addresses (a) the meaning of the scores themselves — whether they measure the intended construct; (b) their generalizability — whether the results are stable from year to year or using different student tests, for example; and (c) the relation of value-added scores to broader notions of teacher effectiveness — whether teachers’ effectiveness in raising test scores can serve as a proxy for other aspects of teaching quality. Next, the interpretive argument directs attention to rationales for the expected benefits of particular value-added score uses or interpretations, as well as plausible unintended consequences. This kind of systematic analysis raises serious questions about some popular policy prescriptions based on teacher value-added scores
The whole report, included below is worth a read, or at least a skip to the conclusion
My first conclusion should come as no surprise: Teacher VAM scores should emphatically not be included as a substantial factor with a fixed weight in consequential teacher personnel decisions. The information they provide is simply not good enough to use in that way. It is not just that the information is noisy. Much more serious is the fact that the scores may be systematically biased for some teachers and against others, and major potential sources of bias stem from the way our school system is organized. No statistical manipulation can assure fair comparisons of teachers working in very different schools, with very different students, under very different conditions. One cannot do a good enough job of isolating the signal of teacher effects from the massive influences of students’ individual aptitudes, prior educational histories, out-of-school experiences, peer influences, and differential summer learning loss, nor can one adequately adjust away the varying academic climates of different schools. Even if acceptably small bias from all these factors could be assured, the resulting scores would still be highly unreliable and overly sensitive to the particular achievement test em- ployed. Some of these concerns can be addressed, by us- ing teacher scores averaged across several years of data, for example. But the interpretive argument is a chain of reasoning, and every proposition in the chain must be supported. Fixing one problem or another is not enough to make the case.

Reliability and Validity of Inferences About Teachers Based On Student Test Scores

Share your comment:

Kasich Has Pushed Ed Beyond Breaking Point

At the onset of the great recession, Ohio employed 107,047 classroom teachers according to the ODE Teacher Information Report for 2008-09. The figure for 2012-13 stands at 100,156. That's a decline of 6,891 teachers during Governor Kasich's tenure. Barely less than the number of teachers in Columbus, Cleveland, and Cincinnati - combined.

Beyond this massive attrition due to austere budget cuts made by the Governor and his legislative allies has also come onerous new requirements, most of which have been unfunded.

The implementation and execution of an evaluation system so crippling that the Ohio Senate recently passed a bill to fix it - unanimously, before OTES has even fully come online. A 3rd grade reading guarantee, common core, PARCC assessments - all with little or no money to fund implementation.

Not satisfied with draconian cuts to districts forcing a reduction of almost 7,000 teachers, the Governor and his allies have also expanded the choice failure. Combined, failing charter schools and unaccountable for-profit private schools are now draining over $1 billion dollars a year from traditional public schools.

If the Governor's goal was to push Ohio's public education system to the breaking point, he has succeeded.

Share your comment:

Is Support for Value-Add Collapsing?

We noticed a story in Politico a short while ago
NEW TACTIC ON TEACHER EVALUATIONS: Randi Weingarten, president of the American Federation of Teachers, is launching a campaign against using value-added metrics to evaluate teacher effectiveness. Her mantra: "VAM is a sham." That’s a notable shift for the AFT and its affiliates, which have previously ratified contracts and endorsed evaluation systems that rely on VAM. Weingarten tells Morning Education that she has always been leery of value-added "but we rolled up our sleeves, acted in good faith and tried to make it work." Now, she says, she’s disillusioned.
Having the President of the second largest teachers union come out against Value-add might not seem all that startling to most observers. Teachers after all have been leery of the whole idea from its inception.

But as study after study reveals who unreliable and inaccurate using student test scores to evaluate teachers is, the skepticism is growing wider and deeper.

In Tennessee, the birth place of Value-add, policy makers are beginning to wise up to the problems too
Tennessee’s education leaders have been collecting national accolades since August, after the state board of education adopted a rare policy that ties teacher licensing to learning gains.
At its meeting in Nashville on Friday, the board stepped away from the new policy, promising an April rewrite eliminating learning gains as the overriding factor in whether teachers can work in Tennessee.
the vote coincides with a bipartisan bill gaining ground in the legislature this session. The Educator Respect and Accountability Act, sponsored by Rep. Matthew Hill, R-Jonesborough, and Sen. Mike Bell, R-Riceville, would prevent the state from yanking teachers’ licenses based on “any statistical estimate utilizing standardized test scores.”
But not just in Tennessee, in Connecticut, the Governor is also asking for a pause in implementing teacher evaluations based on value add
Connecticut Gov. Dannel Malloy has just asked for a “pause” in implementation of a controversial new teacher evaluation system that uses student standardized test scores to assess teachers
In his letter to legislators, the Governor was quite clear, the reforms are not working
However, I would like to make the case that these reforms will not result in improved conditions since they are not grounded in research, the evidence that supports professional decision-making, like a doctor or engineer. It is simply a matter of substance. The evidence is clear in schools across the state. It is not working.

We have spent the better part of the last 12 years with a test-based accountability movement that has not led to better results or better conditions for children. What it has led to is a general malaise among our profession, one that has accepted a narrowing of the curriculum, a teaching to the test mentality, and a poorly constructed redefinition of what a good education is. Today, a good education is narrowly defined as good test scores. What it has led to is a culture of compliance in our schools.

We have doubled-down on the failed practices of No Child Left Behind. Not only do we subscribe to a test and punish mentality for school districts, we have now drilled that mentality down to the individual teacher level.
Meanwhile, here in Ohio, legislators are also realizing that the teacher evaluation system is flawed. SB229 would modify OTES to rely somewhat less on student test scores, however that bill which passed unanimously in the Ohio Senate is now stalled on the House.

Share your comment:

Agenda Driven Charter Boosters

The National Alliance for Public Charters, like so many charter school boosters appears to care more about the quantity of charter schools than they do the quality.

This is clearly evident when looking at their state rankings for charter laws

Nicole Blalock, PhD a postdoctoral scholar at Arizona State University writes
In four of the states with a statistical difference between charter school students’ NAEP scores and public school students’ NAEP scores, statistical differences were observed for all grade/subject pairs tested. This occurred in the states of Alaska, Maryland, Ohio, and Pennsylvania.

On average, in Alaska, students attending charter schools outperformed students in public schools by approximately 10 points in most grade/subject area tests and by more than 20 points in reading in grade 4. However, the National Alliance for Public Charters that ranked the 42 states with charter schools and the District of Colombia as per their charter school laws, ranked Alaska nearly the lowest (i.e., 41st of 43) for the “best” charter laws (“Measuring up to the model”, 2013). Put differently, the state whose charter school students performed among the best as compared to their public school peers just happened to be one of the worst charter states as externally ranked.

Otherwise, public school students outperformed charter school students in the other three states (i.e., Maryland, Ohio, and Pennsylvania) with consistent and significant score differences across the board. Maryland was one of two states to be ranked lower than Alaska for the “best” charter laws overall (i.e., 42nd of 43), and Ohio and Pennsylvania ranked in the middle of the pack (27th and 19th of 43 respectively). Each of these states demonstrated charter school student performance that lagged behind public school students by an average of 23 points.
Until charter school boosters begin to care more about quality than they do quantity, we're going to continue to have horribly performing charter schools in Ohio that are not serving our students. We cannot continue to focus on quantity over quality. The National Alliance for Public Charters state charter law ranking are absurd. Ohio should be ranked dead last, based on quality.

Share your comment:

If a teacher was simply a babysitter

According to the Ohio Department of Education, the average teacher salary in 2013 was $56,715. For employees with advanced degrees, this doesn't crack the top ten industries when just looking at starting pay

1. Computer science: Average starting pay: $73,700
2. Business administration/management: Average starting pay: $69,200
3. Mechanical engineering: Average starting pay: $66,800
4. Electrical/electronics and communications engineering: Average starting pay: $66,100
5. Finance: Average starting pay: $64,300
6. Nursing: Average starting pay: $63,800
7. Economics (business/managerial): Average starting pay: $63,400
8. Health and related sciences: Average starting pay: $62,900
9. Accounting: Average starting pay: $62,300

Teenage babysitters could earn more. Literally.

That's right. Let's give them $3.00 an hour and only the hours they worked; not any of that silly planning time, or any time they spend before or after school. That would be $19.50 a day (7:45 to 3:00 PM with 45 min. off for lunch and planning -- that equals 6-1/2 hours).
So each parent should pay $19.50 a day for these teachers to baby-sit their children. Now how many students do they teach in a day...maybe 30? So that's $19.50 x 30 = $585 a day.
However, remember they only work 180 days a year!!! I am not going to pay them for any vacations.
That's $585 X 180= $105,300 per year. (Hold on! My calculator needs new batteries).
What about those special education teachers and the ones with Master's degrees? Well, we could pay them minimum wage ($7.75), and just to be fair, round it off to $8.00 an hour. That would be $8 X 6-1/2 hours X 30 children X 180 days = $280,800 per year.

Wait a minute -- there's something wrong here!

$56,715 turns out to be quite the deal.

Share your comment:

Teacher performance incentives have negative impact

A recently published study from the Journal of Labor Economics looked at performance incentives for teachers in the NYC school system. The results should cause corporate reformers to pause
Providing financial incentives for teachers to increase student performance is an increasingly popular education policy around the world. This paper describes a school-based randomized trial in over two-hundred New York City public schools designed to better understand the impact of teacher incentives on student achievement. I find no evidence that teacher incentives increase student performance, attendance, or graduation, nor do I find any evidence that the incentives change student or teacher behavior. If anything, teacher incentives may decrease student achievement, especially in larger schools. The paper concludes with a speculative discussion of theories that may explain these stark results.
As Margarita Pivovarova, Assistant Professor of Economics at Arizona State University notes
The estimates from the experiment imply that if a student attended a middle school with an incentive in place for three years, his/her math test scores would decline by 0.138 of a standard deviation and his/her reading score would drop by 0.09 of a standard deviation.

Not only that, but the incentive program had no effect on teachers’ absenteeism, retention in school or district, nor did it affect the teachers’ perception of the learning environment in a school. Literally, the estimated 75 million dollars invested and spent brought zero return!

Share your comment:

Get Involved

 First Name
 Last Name
 School District