Same Teachers, Similar Students, Similar Tests, Different Answers

Via Vamboozled

One of my favorite studies to date about VAMs was conducted by John Papay, an economist once at Harvard and now at Brown University. In the study titled “Different Tests, Different Answers: The Stability of Teacher Value-Added Estimates Across Outcome Measures” published in 2009 by the 3rd best and most reputable peer-reviewed journal, American Educational Research Journal, Papay presents evidence that different yet similar tests (i.e., similar on content, and similar on when the tests were administered to similar sets of students) do not provide similar answers about teachers’ value-added performance. This is an issue with validity, in that, if a test is measuring the same things for the same folks at the same times, similar-to-the-same results should be realized. But they are not. Papay, rather, found moderate-sized rank correlations, ranging from r=0.15 to r=0.58, among the value-added estimates derived from the different tests.

Recently released, yet another study (albeit not yet peer-reviewed) has found similar results…potentially solidifying this finding further into our understandings about VAMs and their issues, particularly in terms of validity (or truth in VAM-based results). This study on “Comparing Estimates of Teacher Value-Added Based on Criterion- and Norm-Referenced Tests” released by the U.S. Department of Education and conducted by four researchers representing Notre Dame University, Basis Policy Research, and American Institutes of Research, provides evidence, again, that estimates of teacher value-added as based on different yet similar tests (i.e., in this case a criterion-referenced state assessment and a widely used norm-referenced test given in the same subject around the same time) yielded moderately correlated estimates of teacher-level value added, yet again.

If we had confidence in the validity of the inferences based on value-added measures, these correlations (or more simply put “relationships”) should be much higher than what they found, similar to what Papay found, in the range of 0.44 to 0.65. While the ideal correlation coefficient is a, in this case, r=+1.0, that is very rarely achieved. But for the purposes for which teacher-level value-added is currently being used, correlations above r=+.70/r=+.80 would (and should) be most desired, and possibly required before high-stakes decisions about teachers are to be made as based on these data.

In addition, researchers in this study found that on average, only 33.3% of teachers’ estimates from both sets of value-added estimates positioned them in the same range of scores (using quintiles or ranges including 20% bands of width) on both tests in the same school year. This too has implications for validity in that, again, teachers or teachers’ value-added estimates should fall in the same ranges, if and when using similar tests, if any valid inferences are to be made using value-added estimates.

Share your comment:

Class Size Matters

A report recently published by the National Education Policy Center, titled "Class Size Matters" found that:
  • Class size is an important determinant of student outcomes, and one that can be directly determined by policy. All else being equal, increasing class sizes will harm student outcomes.
  • The evidence suggests that increasing class size will harm not only children’s test scores in the short run, but also their long-run human capital formation. Money saved today by increasing class sizes will result in more substantial social and educational costs in the future.
  • The payoff from class-size reduction is greater for low-income and minority children, while any increases in class size will likely be most harmful to these populations.
  • Policymakers should carefully weigh the efficacy of class-size policy against other potential uses of funds. While lower class size has a demonstrable cost, it may prove the more cost-effective policy overall.

Read the entire report below

Pb - Class Size by National Education Policy Center

Share your comment:

Failing Charter Schools Dine at the Straight A Fund Trough

The Ohio Department of Education released the list of Straight A applicants that will be moving on to the next scoring phase, as schools are pitted against each other for a chance to win $150 million dollars in the Kasich education funding lottery.

Of those applicant, 17 are charter schools (though some of these are part of a larger consortium, so the true number of charter schools is much higher) making a combined request of $31,397,903.35.

Let's take a closer look at some of these applicants.

The first on the list is Achieve Career Preparatory Academy, a drop out recovery school in Toledo, making a buzzword packed ("3-dimensional learning tools", "focus on student engagement") request for almost $200,000. Not mentioned in the request is that this dropout recovery school only graduates 14.3% of its students according to the latest ODE data. This school needs to be closed down, not handed more money via the Kasich education funding lottery.

Next on our list is the Buckeye On-Line School for Success, which is requesting close to $1 million for an IT system called "Virtualized Operations for Independent and Collaborative Education". The ironically named Buckeye On-Line School for Success is rated F for both performance and Value Add according to the latest ODE data. another example of a charter school that ought to be closed, not handed more taxpayer money.

The Lake Erie Academy is next in the spotlight, with a request for $116,000 "to improve the reading ability of K-8 students at Lake Erie Academy using the computer-based program, Read Naturally Live." This charter school has a D rating for performance and an F for value-add.

More troubling given the nature of this request, is that this charter schools reading performance has declined in each of the last 3 years. in 2010 3rd grade proficiency was 70%, in 2011 56% and in 2012 it was a paltry 25%. 8th grade reading proficiency decline from 61% in 2010 to 41% in 2012. Here we have a low performing charter school in serious decline. Rather than hand more money over so they can continue to fail in their mission, they also need to be closed down.

It should come as little surprise that one of the biggest requests comes from a charter school operated by the politically connected a David Brennan. The White Hat management ran school - Summit academy Secondary in Akron is requested in $6.2 million. As with all White Hat schools, this one is a low performer, meeting just one of its possible standards and receiving a D for performance. They want this $6.2 million to "leverage the power of technology and teacher training to show teachers how to address all student needs in an individualized way". That they are not already doing that is one indication of why these White Hat schools perform so badly.

Throughout this list of Straight A Fund charter school applicants is evidence of why Ohio's charter school experiment is failing. Already an almost $1 billion industry, it needs to be reigned in, not have more money wasted on failure, especially when so many more higher performing traditional public schools have been starved by the governors' education funding policies and forced to fight over scraps via a funding lottery.

Share your comment:

Researchers Give Failing Marks to Teacher Evaluation Systems

Via Hechinger Report.

School systems around the country are trying to use objective, quantifiable measures to identify which are the good teachers and which are the bad ones. One popular approach used in New York, Chicago and other cities, is to calculate a value-added performance measure (VAM). Essentially, you create a model that begins by calculating how much kids’ test scores, on average, increase each year. (Test score year 2 minus test score year 1). Then you give a high score to teachers who have students who post test-score gains above the average. And you give a low score to teachers whose students show smaller test-score gains. There are lots of mathematical tweaks, but the general idea is to build a model that answers this question: are the students of this particular teacher learning more or less than you expect them to? The teachers’ value-added scores are then used to figure out which teachers to train, fire or reward with bonuses.

Two academic researchers from the University of Southern California and the University of Pennsylvania looked at these value-added measures in six districts around the nation and found that there was weak to zero relationship between these new numbers and the content or quality of the teacher’s instruction.

“These results call into question the fixed and formulaic approach to teacher evaluation that’s being promoted in a lot of states right now,” said Morgan Polikoff, one of the study’s authors, in a video that explains his paper, “Instructional Alignment as a Measure of Teaching Quality,” published online in Education Evaluation and Policy Analysis on May 13, 2014. ”These measures are not yet up to the task of being put into, say, an index to make important summative decisions about teachers.”

Polikoff of the University of Southern California and Andrew Porter of the University of Pennsylvania looked at the value-added scores of 327 fourth- and eighth-grade mathematics and English language arts teachers across all six school districts included in the Measures of Effective Teaching (MET) study (New York City, Dallas, Denver, Charlotte-Mecklenburg, Memphis, and Hillsborough County, Florida). Specifically, they compared the teachers’ value added scores with how closely their instructional materials aligned with their state’s instructional standards and the content of the state tests. But teachers who were teaching the right things weren’t getting higher value-added scores.

They also looked at other measures of teacher quality, such as teacher observations and student evaluations. Similarly, teachers who won high marks from professional observers or students were also not getting higher value-added scores.

“What we’re left with is that state tests aren’t picking up what we think of as good teaching,” Polikoff said.

What’s interesting is that Polikoff’s and Porter’s research was funded by the Gates Foundation, which had been touting how teachers’ effectiveness could be estimated by their students’ progress on standardized tests. The foundation had come under fire from economists for flawed analysis. Now this new Gates Foundation’ commissioned research has proved the critics right. (The Gates Foundation is also among the funders of The Hechinger Report).

Polikoff said that the value-added measures do provide some information, but they’re meaningless if you want to use them to improve instruction. “If the things we think of as defining good instruction don’t seem to producing substantially better student achievement, then how is it that teachers will be able to use the value-added results to make instructional improvements?” he asked.

Polikoff concludes that the research community needs to develop new measures of teacher quality in order to “move the needle” on teacher performance.

You can read the entire report below

Educational Evaluation and Policy Analysis-2014-Polikoff

Share your comment:

The Hijacking of SB229

A good rundown of the changes the House made to SB229, from the Ennis, Roberts & Fischer Co law firm.

The Ohio House Education Committee has unveiled sweeping changes to Substitute Senate Bill 229 with regard to teacher and principal evaluations. The original version of SB 229, which passed the Senate unanimously on December 4th, 2013, modified frequency and composition of teacher evaluations and reduced some of the burden on school administrators. The new version of the Bill proposed by the House Education Committee, however, would modify both the OTES and OPES evaluation systems in ways that would undoubtedly place additional strain on the relatively untested evaluation systems. The proposed changes include the following:

  • Bumps student growth measures back up to 50% from the 35% proposed by the Senate, unless a district elects to use an alternative “student survey” framework (available for grades 4-12), in which case the final rating would be comprised of 40% SGM, 40% teacher performance rating, and 20% student survey results;
  • Requires that an evaluator use an average score if a teacher receives different scores on the observations and review components of the evaluations;
  • Increases SGM from three to five total possible ratings: “Most Effective”, “Above Average”, “Average”, “Below Average”, and “Least Effective”;
  • Adds new performance level rating of “Effective” that will exist in the realm between “Skilled” and “Developing”;
  • Requires that at least one formal observation of a teacher be unannounced;
  • Beginning in 2015, allows districts to evaluate “Accomplished” and “Skilled” teachers every other year, but only if the teacher’s SGM score is rated “Average” or higher (teachers must still receive one observation and a conference in the “off” year);
  • District can elect not to evaluate 1) a teacher who is on leave for 70% or more of the year, and 2)a teacher who submitted notice of retirement before Dec. 1st;
  • Teachers rated “Effective” “Developing” or “Ineffective” must be placed on an improvement plan;
  • In 2015 and beyond, districts cannot assign students to a teacher who has been rated ineffective for two or more years (but does not specify what a district should do with these teachers!);
  • A district is also prohibited from assigning a student teacher to a teacher who is “Developing” or “Ineffective” during the previous year;
  • If a teacher with at least ten years of experience receives a designation of either “Least Effective” or “Below Average” on his/her SGM rating, that teacher may be rated “Developing” only once;
  • Mandates that results of an evaluation must follow the teacher even if he/she is transferred to a new building or takes employment elsewhere;
  • Requires ODE to develop a standardized framework for assessing SGM for all non-value added grade levels and subjects by 2016;
  • By 2016, districts must administer assessments to students in each of grades K-12 for English Language Arts, Mathematics, Social Studies, and Science. Assessments must be selected by ODE and based on value-added progress dimension or vendor-developed student growth measures (may include assessments already required by law);
  • Beginning next July, evaluators must verify completion of at least one evaluation training course outlined in the bill;
  • After July 1, 2015, the State Board must ensure individuals seeking licensure as superintendent, assistant superintendent, principal, vocational director, administrative specialist, or supervisor have completed a teacher evaluator training;
  • The revised bill mandates that the State Board of Education must develop a standards based system for principals and assistant principals, which districts must conform to;
  • Third grade reading guarantee assessments must either be value-added or vendor-approved assessments;
  • ODE must provide detailed report of school performance on evaluations to general assembly, and must accept comments for improvement from districts that it passes on to general assembly;
  • Exempts from collective bargaining all amendments made by the bill to 3319.111, 3319.112, 3319.113, 3319.114, 3319.115, and 3319.117;
  • Permits a district to enter into a MOU with union that stipulates value-added progress demission rating issued for 2014-2015 will not be used when making decisions regarding dismissal, retention, tenure or compensation.

Here's the Lesgislative Service commisions comparision document so oyu cna see what changes were made from the unanimously passed Senate version

SB229 Comparison Document

Share your comment:

May 2014 Levy Results

School levies continued their normal pattern of results in the May 2014 primary, almost identically mirroring the May 2013 results.
N/R Failed Passed Pass %
New 38 27 41.5%
Renewal 8 74 90.2%
Over all 46 101 68.7%

Here's the full list of preliminary results:

District Local Result N/R Votes for Voters Against
Ada Hardin Passed Renewal 61.07 38.93
Allen East Allen Passed Renewal 67.07 32.93
Arcadia Hancock Passed Renewal 53.92 46.08
Athens Athens Passed Renewal 62.77 37.23
Aurora Portage Passed Renewal 64.78 35.22
Austintown Mahoning Failed New 34.18 65.82
Ayersville Defiance Failed New 21.71 78.29
Beaver Columbiana Passed Renewal 50.09 49.91
Bellaire Belmont Failed New 43.64 56.36
Belpre Washington Passed Renewal 71.35 28.65
Berne Union Fairfield Passed Renewal 53.64 46.36
Bethel Miami Passed New 53.10 46.90
Bettsville Seneca Failed New 47.89 52.11
Bluffton Allen Passed Renewal 70.95 29.05
Boardman Mahoning Passed Renewal 59.81 40.19
Boardman Mahoning Passed Renewal 59.80 40.20
Botkins Shelby Passed Renewal 60.95 39.05
Bristol Trumbull Failed Renewal 43.48 56.52
Brooklyn Cuyahoga Passed Renewal 53.07 46.93
Brookville Montgomery Passed New 54.62 45.38
Brunswick Medina Passed Renewal 62.91 37.09
Canal Winchester Franklin Passed Renewal 56.95 43.05
Carey Wyandot Passed Renewal 57.09 42.91
Clark-Shawnee Clark Passed New 51.27 48.73
Claymont Tuscarawas Passed Renewal 65.22 34.78
Cloverleaf Medina Passed New 56.38 43.62
Colonel Crawford Crawford Failed New 47.30 52.70
Coshocton County Coshocton Passed New 50.08 49.92
Danbury Ottawa Passed New 54.87 45.13
Danville Knox Passed Renewal 69.26 30.74
Defiance Defiance Passed New 54.71 45.29
Delphos Allen Passed Renewal 70.18 29.82
Delphos Allen Passed Renewal 69.73 30.27
Dover Tuscarawas Failed New 44.56 55.44
East Clinton Clinton Passed Renewal 62.09 37.91
Edison Jefferson Passed New 52.45 47.55
Elyria Lorain Passed Renewal 65.30 34.70
Fairfield Butler Passed New 57.87 42.13
Fairless Stark Failed New 41.14 58.86
Field Portage Failed New 46.39 53.61
Findlay Hancock Passed Renewal 66.45 33.55
Fort Loramie Shelby Passed New 74.30 25.70
Franklin Warren Passed New 52.66 47.34
Franklin Muskingum Failed New 42.04 57.96
Genoa Area Ottawa Failed New 33.74 66.26
Granville Licking Passed Renewal 63.19 36.81
Green Summit Passed Renewal 71.57 28.43
Greeneview Greene Failed New 35.90 64.10
Greenon Clark Failed New 49.15 50.85
Groveport Madison Franklin Passed New 53.37 46.63
Hopewell-Loudon Seneca Passed Renewal 61.89 38.11
Howland Trumbull Failed New 44.79 55.21
Indian Lake Logan Failed Renewal 46.89 53.11
Jackson Stark Passed Renewal 66.68 33.32
Jefferson Madison Passed New 50.17 49.83
Jefferson County JVS Jefferson Passed Renewal 61.12 38.88
Johnstown-Monroe Licking Passed New 53.76 46.24
Jonathan Alder Madison Passed New 51.97 48.03
Keystone Lorain Failed New 40.73 59.27
Kirtland Lake Passed Renewal 60.55 39.45
Lakeview Trumbull Passed Renewal 60.37 39.63
Lexington Richland Passed Renewal 71.25 28.75
Liberty Trumbull Passed Renewal 54.51 45.49
Liberty Trumbull Passed Renewal 51.01 48.99
Liberty Center Henry Passed New 56.58 43.42
Liberty Union-Thurston Fairfield Passed Renewal 50.33 49.67
Logan Elm Pickaway Passed New 55.03 44.97
Loveland Hamilton Passed New 55.45 44.55
Madison Lake Failed New 31.16 68.84
Madison-Plains Madison Passed Renewal 57.45 42.55
Manchester Summit Passed Renewal 61.78 38.22
Mathews Trumbull Failed New 41.67 58.33
McComb Hancock Passed Renewal 78.33 21.67
Mechanicsburg Champaign Failed Renewal 46.79 53.21
Middletown Butler Passed New 50.48 49.52
Millcreek-West Unity Williams Passed New 51.31 48.69
Mogadore Summit Failed New 49.10 50.90
Mohawk Wyandot Passed Renewal 56.46 43.54
Mount Gilead Morrow Failed New 46.60 53.40
National Trail Preble Passed Renewal 63.29 36.71
New Bremen Auglaize Passed Renewal 64.68 35.32
New Philadelphia Tuscarawas Failed New 49.88 50.12
Newbury Geauga Failed New 48.12 51.88
North Central Williams Passed New 63.16 36.84
North Royalton Cuyahoga Failed New 47.39 52.61
North Union Union Passed Renewal 61.88 38.12
Northridge Licking Passed Renewal 55.98 44.02
Northwest Stark Passed Renewal 53.48 46.52
Northwood Wood Passed New 50.64 49.36
Norwalk Huron Passed New 60.89 39.11
Olmsted Falls Cuyahoga Failed New 49.45 50.55
Osnaburg Stark Failed New 38.34 61.66
Parkway Mercer Passed Renewal 67.60 32.40
Parma Cuyahoga Passed Renewal 55.41 44.59
Patrick Henry Henry Passed Renewal 53.13 46.87
Perkins Erie Failed New 42.36 57.64
Perry Allen Passed Renewal 52.14 47.86
Pike-Delta-York Fulton Passed Renewal 52.36 47.64
Piqua Miami Passed Renewal 60.71 39.29
Pleasant Marion Failed New 48.05 51.95
Poland Mahoning Passed Renewal 60.77 39.23
Ravenna Portage Failed New 45.15 54.85
Ridgedale Marion Failed New 42.20 57.80
Ripley-Union-Lewis- Brown Failed New 48.53 51.47
Rittman Wayne Passed Renewal 78.12 21.88
River Valley Marion Failed New 46.57 53.43
Riverside Lake Passed Renewal 56.06 43.94
Rootstown Portage Passed Renewal 60.90 39.10
Russia Shelby Passed Renewal 72.33 27.67
Sandusky Erie Passed Renewal 69.86 30.14
Shaker Heights Cuyahoga Passed New 62.49 37.51
Sheffield-Sheffield Lake Lorain Passed Renewal 66.53 33.47
Sheffield-Sheffield Lake Lorain Passed Renewal 66.34 33.66
Sidney Shelby Passed Renewal 62.97 37.03
Southeast Portage Passed Renewal 69.80 30.20
Southern Perry Failed New 42.97 57.03
Southington Trumbull Failed New 39.27 60.73
Southwest Licking Licking Failed New 45.88 54.12
Springfield Mahoning Failed Renewal 48.38 51.62
St. Henry Consolidated Mercer Passed Renewal 71.09 28.91
St. Marys Auglaize Passed Renewal 69.55 30.45
Swanton Fulton Passed Renewal 56.31 43.69
Sylvania Lucas Failed New 47.57 52.43
Tecumseh Clark Passed Renewal 60.52 39.48
Tecumseh Clark Passed Renewal 58.85 41.15
Tiffin Seneca Passed Renewal 66.33 33.67
Tipp Miami Passed Renewal 60.33 39.67
Tri-County North Preble Passed New 59.27 40.73
Tri-Rivers JVS Marion Failed Renewal 49.69 50.31
Tri-Rivers JVS Marion Failed Renewal 37.05 62.95
Triad Champaign Failed New 36.46 63.54
Trimble Athens Passed New 51.56 48.44
Triway Wayne Passed Renewal 55.36 44.64
Tuscarawas Valley Tuscarawas Passed New 52.16 47.84
Union Belmont Failed New 41.25 58.75
Union-Scioto Ross Passed Renewal 59.01 40.99
United Columbiana Passed Renewal 65.85 34.15
Van Buren Hancock Failed New 38.19 61.81
Walnut Township Fairfield Failed New 21.98 78.02
Warren Washington Passed Renewal 63.89 36.11
Wellington Lorain Passed Renewal 60.21 39.79
Wellsville Columbiana Passed Renewal 73.09 26.91
West Branch Mahoning Failed New 29.31 70.69
West Liberty-Salem Champaign Failed Renewal 46.20 53.80
Willard Huron Passed Renewal 54.44 45.56
Woodridge Summit Passed Renewal 59.80 40.20
Zanesville Muskingum Failed Renewal 48.36 51.64

Share your comment:

Get Involved

 Email*
 First Name
 Last Name
 School District

Search