The study found four key issues that consistently came up with regard to the use of value-added for teacher evaluations:Abstract: State and local education agencies across the United States are increasingly adopting rigorous teacher evaluation systems. Most systems formally incorporate teacher performance as measured by student test-score growth, sometimes by state mandate. An important consideration that will influence the long-term persistence and efficacy of these systems is stakeholder buy-in, including buy-in from teachers. In this study we document common questions from teachers about value-added measures and provide research-based responses to these questions.
These questions still persist today, and are larely unanswered.1. Differentiated Students. How can the model deal with a teacher who has students who are different for some reason (e.g., poverty, special education, etc.)? Will that teacher be treated unfairly by the model?
2. Student Attendance. Will teachers be held accountable for students who do not regularly attend class?
3. Outside Events and Policies. How can the model account for major events (e.g., school closings for snow) or initiatives (e.g., Common Core implementation) that impact achievement?
4. Ex Ante Expectations. Why can’t teachers have their predicted scores – the target average performance levels for their students – in advance?
The report find the following: Total Testing Time for the Average Student in a School Year, in Hours
That's a lot of testing, and is not fully comprehensive as the report notes.
ODE goes on to provide 8 action steps being taken, and ends with a number of recommendations, including
A clear admission that the testing regime in Ohio has gotten out of hand by at least 20% To get there ODE lays out the following recommendationsThis report includes a comprehensive package of legislative recommendations to shorten the amount of time students spend taking tests. These recommendations place limits on the overall time students spend taking tests each year, eliminate unnecessary tests and modify the Ohio Teacher Evaluation System. The following recommendations are contingent on each other and would require implementation as a comprehensive set of reforms. If this package of recommendations is adopted, the state can reduce the amount of time students are taking tests by nearly 20 percent.
That last recommendation is a huge admission. Teachers and administrators have been hugely burdened developing and deploying SLO's and students have received little to no benefit from them. Here's the full reportRecommendation 1: Limit the amount of time a student takes tests at the state and district levels to 2 percent of the school year, and limit the amount of time spent practicing for tests to 1 percent of the school year. These limits will encourage the state and districts to prioritize testing and guarantee to students and parents that the vast majority of time in the classroom will focus on instruction, not testing.
[...]Recommendation 2: Eliminate the fall third-grade reading test and administer the test in the spring. Students who do not reach the required promotion score on the spring test will have a second opportunity to take the test in the summer.
[...]Recommendation 3: Eliminate the state’s requirement that districts give mathematics and writing diagnostic tests to students in first grade through third grades.
[...]Recommendation 4: Eliminate the use of student learning objective tests as part of the teacher evaluation system for grades pre-K to 3 and for teachers teaching in non-core subject areas in grades 4-12. The core areas are English language arts, mathematics, science and social studies. Teachers teaching in grades and subject areas in which student learning objectives are no longer permitted will demonstrate student growth through the expanded use of shared attribution, although at a reduced level overall. In cases where shared attribution isn’t possible, the department will provide guidance on alternative ways of measuring growth.
That's good to hear. However, we are still waiting for his report on Horizon schools. He seemed much swifter, and public, with his investigation of data tampering at Columbus City Schools than he has been with this important investigation. A quick look at recent campaign finance reports show that Yost has been a significant beneficiary of contribtiuons from the charter sector, taking in almost $50,000 from David Brennan (White Hat) and William Lager (ECOT) alone. Only time will tell whether that was a sound investment by the for profit charter folks.Charter schools will remain a major focus in Auditor Dave Yost's second term, the Republican announced Monday. Mr. Yost, who was sworn in during a morning Statehouse ceremony, told reporters that while he plans to also prioritize public records, board accountability and data integrity, charter schools will be "front and center" during the first year of his new term. "We audit every charter school now, if the legislature chooses to give us additional tools or greater responsibility we'll do that - we'll be part of that discussion too," he said. "I think there's some things that need to be addressed. There's multiple ways of doing it and that debate will unfold and I'll be part of it over the next few months." The auditor launched an investigation into 19 charters managed by Concept Schools last year. (See Gongwer Ohio Report, October 14, 2014) In addition to charter schools, Mr. Yost said he intends to focus on board accountability, saying he's "very concerned about the uneven quality of the unpaid boards that we charge with supervising various public functions." "I'm not sure the lines of responsibility are sufficiently clear or that boards are being given the tools they need to succeed," he said. "And that goes from a small charter school board acting in a public behalf all the way up to large institutions spending millions of dollars."
As you can see, the findings here are directly relevant to educators in Ohio. The report looked at a number of factors, including reliability, which once again proves to be anything butToday, SAS EVAAS® is the most widely used VAM in the country, and North Carolina, Ohio, Pennsylvania and Tennessee use the model state-wide (Collins & Amrein-Beardsley, 2014). Despite widespread popularity of the SAS EVAAS®, however, no research has been done from the perspective of teachers to examine how their practices are impacted by this methodology that professedly identifies effective and ineffective teachers. Even more disconcerting is that districts and states are tying consequences to the data generated from the SAS EVAAS®, entrusting the sophisticated methodologies to produce accurate, consistent, and reliable data, when it remains unknown how the model actually works in practice.
A random number generator isn't an appropriate tool for measuring anything, let alone educator effectiveness that might lead to high-stakes career decisions.Reliability
As discussed in related literature (Baker et al., 2010; Corcoran, 2010; EPI, 2010; Otterman, 2010; Schochet & Chiang, 2010) and preliminary studies in SSD (Amrein-Beardsley & Collins, 2012), it was evident that inconsistent SAS EVAAS® scores year-to-year were an issue of concern. According to teachers who participated in this study, reliability as measured by consistent SAS EVAAS® scores year-to-year was ironically, an inconsistent reality. About half of the responding teachers reported consistent data whereas the other half did not, just like one would expect with the flip of a coin (see also Amrein-Beardsley & Collins, 2012). Reliability Implications
Unless school districts could prevent teacher mobility and ensure equal, random student assignment, it appears that EVAAS is unable to produce reliable results, at least greater than 50% of the time.
Furthermore, the study found that teachers are discovering that despite claims to the contrary, the SAS formula for calculating Value-add is highly dependent upon the student population
In conclusion, the study findsteachers repeatedly identified specific groups of students (e.g., gifted, ELL, transition, special education) that typically demonstrated little to no SAS EVAAS® growth. Other teachers described various teaching scenarios such as teaching back-to-back grade levels or switching grade levels which negatively impacted their SAS EVAAS® scores. Such reports contradict Dr. Sanders’ claim that a teacher in one environment is equally as effective in another (LeClaire, 2011).
the full study can be read below.The results from this study provide very important information of which not only SSD administrators should be aware, but also any other administrators from districts or states currently using or planning to use a VAM for teacher accountability. Although high-stakes use certainly exacerbates such findings, it is important to consider and understand that unintended consequences will accompany the intended consequences of implementing SAS EVAAS®, or likely any other VAM. Reminiscent of Campbell’s law, the overreliance on value-added assessment data (assumed to have great significance) to make high-stakes decisions risks contamination of the entire educational process, for students, teachers and administrators (Nichols & Berliner, 2007). Accordingly, these findings also strongly validate researchers’ recommendations to not use value-added data for high-stakes consequences (Eckert & Dabrowski, 2010; EPI, 2010; Harris, 2011). While the SAS EVAAS® model’s vulnerability as expressed by the SSD EVAAS®-eligible teachers is certainly compounded by the district’s high-stakes use, the model’s reliability and validity issues combined with teachers’ feedback that the SAS EVAAS® reports do not provide sufficient information to allow for instructional modification or reflection, would make it seem inappropriate at this point to use value-added data for anything.