OEA Response to PD and NPR Teacher shaming

Here's the statement from the Ohio Education Association, which represents over 121,000 educators

Responding to a series of newspaper, web and radio stories on value-added scroes of individual Ohio teachers, Patricia Frost-Brooks, President of the Ohio Education Association criticized the fairness of the stories and the wisdom of using value-added scores as such a prominent index of teacher success:

"The Ohio Education Association was not contacted for comment on the Plain Dealer/StateImpact Ohio stories, despite our expertise, which would have provided desperately needed context and perspective. Reporters and editors admitted this value-added data was 'flawed,' but they chose surprise and impact over fairness, balance and accuracy," Frost-Brooks said.

"We are all accountable for student success – teachers, support professionals, parents, students and elected officials. And the Ohio Education Association is committed to fair teacher evaluation systems that include student performance, among other multiple measures. But listing teachers as effective or ineffective based on narrow tests not designed to be used for this purpose is a disservice to everyone.

"Value-added ratings can never paint a complete or objective picture of an individual teacher’s work or performance. Trained educators can use a student’s value-added data, along with other student data, to improve student instruction. But the stories promote a simplistic and inaccurate view of value-added as a valid basis for high-stakes decisions on schools, teachers and students."

Very questionable that reporters would not contact the largest teachers assoication in crafting their story.

Battelle Blasts Papers decision

From our mailbag, Battelle for Kids condemns the Plain Dealer and NPR's decision to publish teacher's value-added scores, calling it "the poster child for name, blame, and shame and the antithesis of our approach to using value-added data"

To: All SOAR districts
From: Jim Mahoney and Bobby Moore
Date: June 17, 2013

Yesterday, a three-part series on value-added was launched by The Cleveland Plain Dealer and State Impact Ohio. It includes both articles and radio segments specific to value-added analysis as a measure of teacher effectiveness. Highlighted in the articles is a link to a database of teacher ratings, hosted by The Plain Dealer and the State Impact Ohio partnership.

Currently, Ohio laws governing the release of teacher records would apply to teacher value-added results. Thus, teacher level value-added information is subject to public records requests through ODE. Through The Plain Dealer and State Impact Ohio database, the general public can now access a teacher's overall composite rating derived from two years of his/her results in grades 4-8 math and reading. These data reflect information for less than 1/3 of the math and reading, grades 4-8 teachers in Ohio.

Battelle for Kids was not aware these ratings would be published in this way, at this time.

While Battelle for Kids does support the use of value-added information for school improvement and as one of several components of a multi-measures evaluation system, value-added should NOT be used in isolation to draw conclusions about a teacher's effectiveness.

Multiple data points over time from multiple perspectives are crucial because teaching and learning and the evaluation of teaching and learning are complex.

Therefore, we are NOT supportive of these ratings being publically available and discourage promoting the use of this public database.

Talking points and articles, to support your local conversations, are available on the Ohio Student Progress Portal.

http://cts.vresp.com/c/?BattelleForKids/f43a0e1b46/fb8aa9ca4e/313346eb88/sflang=en

Obviously, this is the poster child for name, blame, and shame and the antithesis of our approach to using value-added data.

Please call if you have any questions.

Thank you for all you do for Ohio's students!

-Jim and Bobby

Value-added: How Ohio is destroying a profession

We ended the week last week with a post titled "The 'fun' begins soon", which took a look at the imminent changes to education policy in Ohio. We planned on detailing each of these issues over the next few weeks.

Little did we know that the 'fun' would begin that weekend. It came in the manner of the Cleveland Plain Dealer and NPR publishing a story on the changing landscape of teacher evaluations titled "Grading the Teachers: How Ohio is Measuring Teacher Quality by the Numbers".

It's a solid, long piece, worth the time taken to read it. It covers some, though not all, of the problems of using value-added measurements to evaluate teachers

Those ratings are still something of an experiment. Only reading and math teachers in grades four to eight get value-added ratings now. But the state is exploring how to expand value-added to other grades and subjects.

Among some teachers, there’s confusion about how these measures are calculated and what they mean.

“We just know they have to do better than they did last year,” Beachwood fourth-grade teacher Alesha Trudell said.

Some of the confusion may be due to a lack of transparency around the value-added model.

The details of how the scores are calculated aren’t public. The Ohio Education Department will pay a North Carolina-based company, SAS Institute Inc., $2.3 million this year to do value-added calculations for teachers and schools. The company has released some information on its value-added model but declined to release key details about how Ohio teachers’ value-added scores are calculated.

The Education Department doesn’t have a copy of the full model and data rules either.

The department’s top research official, Matt Cohen, acknowledged that he can’t explain the details of exactly how Ohio’s value-added model works. He said that’s not a problem.

Evaluating a teacher on a secret formula isn't a practice that can be sustained, supported or defended. The article further details a common theme we hear over and over again

But many teachers believe Ohio’s value-added model is essentially unfair. They say it doesn’t account for forces that are out of their control. They also echo a common complaint about standardized tests: that too much is riding on these exams.

“It’s hard for me to think that my evaluation and possibly some day my pay could be in a 13-year-old’s hands who might be falling asleep during the test or might have other things on their mind,” said Zielke, the Columbus middle school teacher.

The article also performs analysis on several thousands value add scores, and that analysis demonstrates what we have long reported, that value-add is a poor indicator of teacher quality, with too many external factors affecting the score

A StateImpact/Plain Dealer analysis of initial state data suggests that teachers with high value-added ratings are more likely to work in schools with fewer poor students: A top-rated teacher is almost twice as likely to work at a school where most students are not from low-income families as in a school where most students are from low-income families.
[…]
Teachers say they’ve seen their value-added scores drop when they’ve had larger classes. Or classes with more students who have special needs. Or more students who are struggling to read.

Teachers who switch from one grade to another are more likely to see their value-added ratings change than teachers who teach the same grade year after year, the StateImpact/Plain Dealer analysis shows. But their ratings went down at about the same rate as teachers who taught the same grade level from one year to the next and saw their ratings change.

What are we measuring here? Surely not teacher quality, but rather socioeconomic factors and budget conditions of the schools and their students.

Teachers are intelligent people, and they are going to adapt to this knowledge in lots of unfortunate ways. It will become progressively harder to districts with poor students to recruit and retain the best teachers. But perhaps the most pernicious effect is captured at the end of the article

Stephon says the idea of Plecnik being an ineffective teacher is “outrageous.”

But Plecnik is through. She’s quitting her job at the end of this school year to go back to school and train to be a counselor — in the community, not in schools.

Plecnik was already frustrated by the focus on testing, mandatory meetings and piles of paperwork. She developed medical problems from the stress of her job, she said. But receiving the news that despite her hard work and the praise of her students and peers the state thought she was Least Effective pushed her out the door.

“That’s when I said I can’t do it anymore,” she said. “For my own sanity, I had to leave.”

The Cleveland Plain Dealer and NPR then decided to add to this stress by publishing individual teachers value-added scores - a matter we will address in our next post.

Survey shows disturbing patterns

The latest Metlife survey, conducted annually since 1984, shows educators under incredible stress as they cope with large budget cuts coupled with increased demands.

The whole survey is worth a read, but we've pulled out some of the most important findings.

Principals and Teachers Give Positive Ratings to the Job Teachers Are Doing

Nearly all principals (98%) give positive ratings to the classroom teachers in their school. This level is similar to the ratings provided by principals in 1986 (95%). The majority of principals (63%) say that their teachers are doing an excellent job and an additional 35% describe the job teachers are doing as pretty good.

In contrast to teachers’ ratings of their principals, the most experienced principals are most likely to rate their teachers highly. Principals with more than 10 years’ experience as a principal are more likely than those with six to 10 years’ experience or those with five years’ or less experience to rate the classroom teachers in their school as excellent (72% vs. 56% vs. 59%).

Of course corporate education reformers will continue to claim too many teachers are not performing in the classroom, despite all the available evidence.

As a consequence of the relentless teacher bashing, and budget cuts, politicians are causing serious moral problems with the workforce, as is evidenced in the next two findings

Teacher Job Satisfaction Continues to Decline

Teacher satisfaction has declined to its lowest point in 25 years and has dropped five percentage points in the past year alone, from 44% to 39% very satisfied. This marks a continuation of a substantial decline noted in the 2011 MetLife Survey of the American Teacher; teacher satisfaction has now dropped 23 percentage points since 2008.

Stress among teachers has increased since 1985

In 1985—the last time this question was asked and when job satisfaction was also low—more than one-third (36%) of teachers said they felt under great stress at least several days a week. Today, that number has increased; half (51%) of teachers feel under great stress at least several days a week. Elementary school teachers experience stress more frequently.

They are more likely than middle school or high school teachers to say they feel under great stress at least several days a week (59% vs. 44% vs. 42%). The increase since 1985 in the number of elementary school teachers who experience great stress at least several days a week is also noteworthy—59% today compared to 35% in 1985

Corporate reformers are also having negative impacts on Principals too

Most principals say that their responsibilities today have changed compared to five years ago and that the job has increased in complexity

Moreover, three-quarters (75%) of principals agree that the job of the principal has become too complex, a view shared by principals regardless of demographic characteristics such as school level, school location, the proportion of low-income or minority students, or the proportion of students performing at or above grade level in English language arts and math.

Half (48%) of principals feel under great stress several days a week or more. This finding is perhaps not surprising given the previously cited results that most principals feel their jobs are too complex, their responsibilities have changed during the past five years, and that they have a high degree of accountability with varying levels of control over decisions

The ever increasing Rube Goldberg machines being constructed by corporate education reformers is making the job of principal all but impossible. The survey notes

It is important to note that as educators begin to implement new, higher standards, many face other competing mandates related to teacher and student assessment as well as decreasing teacher morale,and reductions in budgets and other resources such as staff, professional learning opportunities, and time for collaboration.

When asked about limited resources and what would help them most in addressing the needs of diverse learners, majorities of teachers consistently say other teachers. In 2009, nine in 10 teachers agreed that other teachers contribute to their success in the classroom, including 51% who strongly agreed. Most teachers and principals also said that greater collaboration among teachers and school leaders would have a major impact on improving student achievement.

Given limited resources, teachers believed opportunities for collaborative teaching would have a major impact on their ability to address different learning needs of individual students.

Yet most teachers continued to report that their time to work with other teachers remained the same or had been reduced.

On top of these strains being faced by teachers, the strains being felt by principals is leading to teachers taking on additional leadership roles

As the job of the principal has become more complex with the need to balance instructional leadership, high-stakes accountability, and non-academic management, the survey has documented the emergence of teachers more prominently as leaders in their schools, districts and beyond. The voice of the teacher as an educator has also become a voice of leadership in education.
[…]
Teachers Are School Leaders; Many Have a Formal Leadership Role in Their School Half (51%) of teachers currently have a formal leadership role in their school, such as department chair, instructional resource, teacher mentor, or leadership team member. Teachers who have a formal leadership role are more experienced; they are more likely than other teachers to have at least six years of teaching experience (86% vs. 73%). These teacher leaders are also more likely to report that their school’s budget has decreased during the past 12 months (60% vs. 51%), perhaps reflecting a greater need among these schools to have teachers take on more responsibilities.
[…]
In the context of additional challenges for leading schools toward greater improvement, the continuing decline in teacher morale identifies itself as an urgent priority. During a time when expectations and standards are increasing for effective teaching and learning, teacher morale is yet another declining resource, one that is associated with schools with diminished budgets and other resources, fewer students meeting standards and fewer colleagues highly rated for how well they are doing their job. Teacher leadership emerges as a potential resource for translating big challenges into opportunities, served by hybrid roles for teachers as leaders and as a method for addressing professional growth and satisfaction.

It's time that politicians began to properly value and respect their most valuable asset the education system has - the educators who work in it.

Poor schools can’t win

Without question, designing school and district rating systems is a difficult task, and Ohio was somewhat ahead of the curve in attempting to do so (and they’re also great about releasing a ton of data every year). As part of its application for ESEA waivers, the state recently announced a newly-designed version of its long-standing system, with the changes slated to go into effect in 2014-15. State officials told reporters that the new scheme is a “more accurate reflection of … true [school and district] quality.”

In reality, however, despite its best intentions, what Ohio has done is perpetuate a troubled system by making less-than-substantive changes that seem to serve the primary purpose of giving lower grades to more schools in order for the results to square with preconceptions about the distribution of “true quality.” It’s not a better system in terms of measurement – both the new and old schemes consist of mostly the same inappropriate components, and the ratings differentiate schools based largely on student characteristics rather than school performance.

So, whether or not the aggregate results seem more plausible is not particularly important, since the manner in which they’re calculated is still deeply flawed. And demonstrating this is very easy.

Rather than get bogged down in details about the schemes, the short and dirty version of the story is that the old system assigned six possible ratings based mostly on four measures: AYP; the state’s performance index; the percent of state standards met; and a value-added growth model (see our post for more details on the old system). The new system essentially retains most of the components of the old, but the formula is a bit different and it incorporates a new “achievement and graduation gap” measure that is supposed to gauge whether student subgroups are making acceptable progress. The “gap” measure is really the only major substantive change to the system’s components, but it basically just replaces one primitive measure (AYP) with another.*

Although the two systems yield different results overall, the major components of both – all but the value-added scores – are, directly or indirectly, “absolute performance” measures. They reflect how highly students score, not how quickly they improve. As a result, the measures are telling you more about the students that schools serve than the quality of instruction that they provide. Making high-stakes decisions based on this information is bad policy. For example, closing a school in a low-income neighborhood based on biased ratings not only means that one might very well be shutting down an effective school, but also that it’s unlikely it will be replaced by a more effective alternative.

Put differently, the most important step in measuring schools’ effectiveness is controlling for confounding observable factors, most notably student characteristics. Ohio’s ratings are driven by them. And they’re not the only state.

(Important side note: With the exception of the state’s value-added model, which, despite the usual issues, such as instability, is pretty good, virtually every indicator used by the state is a cutpoint-based measure. These are severely limited and potentially very misleading in ways that are unrelated to the bias. I will not be discussing these issues in this post, but see the second footnote below this post, and here and here for some related work.)**

The components of the new system

The severe bias in the new system’s constituent measures is unmistakable and easy to spot. To illustrate it in an accessible manner, I’ve identified the schools with free/reduced lunch rates that are among the highest 20 percent (highest quintile) of all non-charter schools in the state. This is an imperfect proxy for student background, but it’s sufficient for our purposes. (Note: charter schools are excluded from all these figures.)

The graph below breaks down schools in terms of how they scored (A-F) on each of the four components in the new system; these four grades are averaged to create the final grade. The bars represent the percent of schools (over 3,000 in total) receiving each grade that are in the highest poverty quintile. For example, looking at the last set of bars on the right (value-added), 17 percent of the schools that received the equivalent of an F (red bar) on the value-added component were high-poverty schools.

[readon2 url="http://shankerblog.org/?p=5511"]Continue reading[/readon2]

Superintendents say proposed school grading system gets F

The proposed new Ohio school district grading scale isn’t getting enthusiastic support by some local superintendents.

“I don’t understand what our state is trying to do to our public schools,” Elyria City Schools Superintendent Paul Rigda said. “What do they want from us?”

It's a good question.Let's take a look at what the state intends to do. The table below analyzes how schools using the current rating system will be allocated new grades. For example, only 94 of 1452 schools currently rated as "excellent" will receive an "A" grade, 4 of them currently rated "Excellent", will receive a "D", which is sure to come as a shock, though perhaps not as shocking as the 2 schools currently rated "Effective" which will receive an "F".

	Overall Grade
2011 Performance Rating	A	B	C	D	F	Grand Total
Academic Emergency	-	-	-	40	119	159
Academic Watch	-	-	-	145	58	203
Continuous Improvement	-	-	72	322	44	438
Effective	-	182	464	193	2	841
Excellent	94	1227	127	4	-	1452
Excellent with Distinction	130	186	-	-	-	316
Grand Total	224	1595	663	704	223	3409

You can view the entire list of simulated school ratings, here (xls).

How does this happen? Well, the state has decided to change not only the grading system, because apparently parents aren't smart enough to differentiate between "excellent" and "Academic Emergency", a move described as "Insulting" by one suprintendent, but to also change how grades are calculated. The new formulation will rely heavily upon test scores.

• Percentage of State Indicators Met
• Performance Index (based on test scores)
• Achievement and Graduation Gap (based on test scores)
• Value-Added (based on test scores)

As we have talked about at length before, there's a significant element of socioeconomic factors that affect these results.

According to an Excel regression analysis of ODE data on the new system at the district level, demographics (poverty, income, property valuation, teacher salaries, educational attainment levels, etc.) produce an R-squared value of .48 for the new system vs. an R-squared of .45 for the previous system. The closer to 1 (or -1), the stronger the correlation.

Under this new system is it likely that at least 70% of schools will see their ratings drop.

“What system is this,” he said. “What’s the point?”

With the increased focus on school district ratings, he stated it has been some what of an uphill battle to keep teachers and staff focused.

By having to keep the teachers motivated and pumped up, in lieu of increased pressure by the state to perform on standardized test, the district is finding it hard.

He simply tells his staff to look to their students as their motivation.

“They are doing all of this (winning awards and honing their skills and talents) because of their education,” he said. “It’s taking a lot of work to keep them (teachers and staff) pumped up.

Motivating students and staff isn't the only problem. It may also have a deleterious effect on house prices, at a time when prices are already depressed

Obviously no one is arguing that the ONLY reason that people pay 214% more to live in New Albany is because of its schools, but let’s face it, it’s a huge selling point. You’ve seen the billboards advertising otherwise uninteresting, treeless tract houses in recently-converted farmland trumpeting access to “Dublin schools!” That will no longer mean much.

Under the new plan, parents seeking “the best schools” for their kids in Central Ohio would no longer look at Dublin, Upper Arlington or New Albany. In fact, the only top-rated district remaining in the entire region is Granville, 40 miles away in Licking County. Better get ready for a long commute, moms and dads! The same challenges face Cuyahoga county parents. Where home buyers previously had seven “top-rated” districts to choose from, now they only have two. Sorry North Olmsted, sorry Mayfield.

The obvious concern with this plan is the impact it will have on levies. Despite recent success in passing crucial levies, voters might be less inclined to continue to support schools they see as having been downgraded, or not rated "Excellent". When the state is rolling back its financial support for public schools, forcing them to rely more and more upon local sources of funding, this seems counter to that, in effect helping to cut off all sources of money leaving only drastic cuts as a means to balance already broken budgets.