Get Updates via Email

Events Reminder

A quick reminder of an important event TODAY that we're involved with and hope to see many of our readers at.

Rally for Good Jobs and Strong Communities
Thursday, May 5th, 2011 staring at 5:00pm
On May 5th, the House will be voting for Governor Kasich's budget. Come show your support for Good Jobs and Strong Communities in Ohio.
WHERE: Ohio Statehouse, Columbus

Some Hows and Whys of Value Add Modelling

We thought it would be useful to provide a quick primer on what Value Add actually is, and how it is calculated, in somewhat explainable terms. This is a good explanation via the American Statistical Association

The principal claim made by the developers of VAM—William L. Sanders, Arnold M. Saxton, and Sandra P. Horn—is that through the analysis of changes in student test scores from one year to the next, they can objectively isolate the contributions of teachers and schools to student learning. If this claim proves to be true, VAM could become a powerful tool for both teachers’ professional development and teachers’ evaluation.

This approach represents an important divergence from the path specified by the “adequate yearly progress” provisions of the No Child Left Behind Act, for it focuses on the gain each student makes, rather than the proportion of students who attain some particular standard. VAM’s attention to individual student’s longitudinal data to measure their progress seems filled with commonsense and fairness. There are many models that fall under the general heading of VAM. One of the most widely used was developed and programmed by William Sanders and his colleagues. It was developed for use in Tennessee and has been in place there for more than a decade under the name Tennessee Value-Added Assessment System. It also has been called the “layered model” because of the way each of its annual component pieces is layered on top of another.

The model begins by representing a student’s test score in the first year, y1, as the sum of the district’s average for that grade, subject, and year, say μ1; the incremental contribution of the teacher, say θ1; and systematic and unsystematic errors, say ε1. When these pieces are put together, we obtain a simple equation for the first year:

y1 = μ1+ θ1+ ε1 (1)
or
Student’s score (1) = district average (1) + teacher effect (1) + error (1)

There are similar equations for the second, third, fourth, and fifth years, and it is instructive to look at the second year’s equation, which looks like the first except it contains a term for the teacher’s effect from the previous year:

y2 = μ2+ θ1+ θ2+ ε2 . (2)
or
Student’s score (2) = district average (2) + teacher effect (1) + teacher (2) + error (2)

To assess the value added (y2 – y1), we merely subtract equation (1) from equation (2) and note that the effect of the teacher from the first year has conveniently dropped out. While this is statistically convenient, because it leaves us with fewer parameters to estimate, does it make sense? Some have argued that although a teacher’s effect lingers beyond the year the student had her/him, that effect is likely to shrink with time.

Although such a model is less convenient to estimate, it more realistically mirrors reality. But, not surprisingly, the estimate of the size of a teacher’s effect varies depending on the choice of model. How large this choice-of-model effect is, relative to the size of the “teacher effect” is yet to be determined. Obviously, if it is large, it diminishes the practicality of the methodology.

Recent research from the Rand Corporation shows a shift from the layered model to one that estimates the size of the change of a teacher’s effect from one year to the next suggests that almost half of the teacher effect is accounted for by the choice of model.

One cannot partition student effect from teacher effect without information about how the same students perform with other teachers. In practice, using longitudinal data and obtaining measures of student performance in other years can resolve this issue. The decade of Tennessee’s experience with VAM led to a requirement of at least three years’ data. This requirement raises the concerns when (i) data are missing and (ii) the meaning of what is being tested changes with time.
The Ohio Department of Education has papers, here, that discuss the technical details of how VAM is done in Ohio.

BattleforKids.org provided us this information
Here's a brief example of both analysis that are used in Ohio. Both are from the EVAAS methodology produced by SAS:

Value-added analysis is produced in two different ways in Ohio:
1. MRM analysis (Multivariate Response Model, also known as the mean gain approach); and
2. URM analysis (Univariate Response Model, also known as the predicted mean approach).

The MRM analysis is used for the Ohio value-added results in grades 4-8 math and reading. It can onlybe used when tests are uniformly administered in consecutive grades. Through this approach, district, school and teacher level results are compared to a growth standard. The OAA assessments provide the primary data for this approach.

The URM analysis is used for expanded value-added results. Currently this analysis is provided through the Battelle for Kids' (BFK) SOAR and Ohio Value-Added High Schools (OVAHS) projects. The URM analysis is used when tests are not given in consecutive grades. This approach "pools" together districts that use of the same sequence of particular norm reference tests. In the URM analysis, prior test data are used to produce a prediction of how a student is likely to score on a particular test, given the average experience in that school. For example, results from prior OAA and TerraNovaT results are used as predictors for the ACT end-of-course exams. Differences between students' predictions and their actual/observed scores are used to produce school and teacher effects. The URM analysis is normalized each year based on the performance of other schools in the pool that year. This approach means that a comparison is made to the growth of the average school or teacher for that grade/subject in the pool.

Value add high stakes use cautioned

The American Mathematics Society just published a paper titled "Mathematical Intimidation:Driven by the Data", that discusses the issues with using Value Add in high stakes decision making, such as teacher evaluation. It's quite a short read, and well worth the effort.
Many studies by reputable scholarly groups call for caution in using VAMs for high-stakes decisions about teachers.

A RAND research report: The esti- mates from VAM modeling of achieve- ment will often be too imprecise to support some of the desired inferences [McCaffrey 2004, 96].

A policy paper from the Educational Testing Service’s Policy Information Center: VAM results should not serve as the sole or principal basis for making consequential decisions about teach- ers. There are many pitfalls to making causal attributions of teacher effective- ness on the basis of the kinds of data available from typical school districts. We still lack sufficient understanding of how seriously the different technical problems threaten the validity of such interpretations [Braun 2005, 17].

A report from a workshop of the Na- tional Academy of Education: Value- added methods involve complex sta- tistical models applied to test data of varying quality. Accordingly, there are many technical challenges to ascer- taining the degree to which the output of these models provides the desired estimates [Braun 2010]
[...]
Making policy decisions on the basis of value- added models has the potential to do even more harm than browbeating teachers. If we decide whether alternative certification is better than regular certification, whether nationally board cer- tified teachers are better than randomly selected ones, whether small schools are better than large, or whether a new curriculum is better than an old by using a flawed measure of success, we almost surely will end up making bad decisions that affect education for decades to come.

This is insidious because, while people debate the use of value-added scores to judge teachers, almost no one questions the use of test scores and value-added models to judge policy. Even people who point out the limitations of VAM ap- pear to be willing to use “student achievement” in the form of value-added scores to make such judgments. People recognize that tests are an im- perfect measure of educational success, but when sophisticated mathematics is applied, they believe the imperfections go away by some mathematical magic. But this is not magic. What really happens is that the mathematics is used to disguise the prob- lems and intimidate people into ignoring them—a modern, mathematical version of the Emperor’s New Clothes.
The entire, short paper, can be read below.

Mathematical Intimidation: Driven by the Data

The High Cost of Low Teacher Salaries

WHEN we don’t get the results we want in our military endeavors, we don’t blame the soldiers. We don’t say, “It’s these lazy soldiers and their bloated benefits plans! That’s why we haven’t done better in Afghanistan!” No, if the results aren’t there, we blame the planners. We blame the generals, the secretary of defense, the Joint Chiefs of Staff. No one contemplates blaming the men and women fighting every day in the trenches for little pay and scant recognition.

And yet in education we do just that. When we don’t like the way our students score on international standardized tests, we blame the teachers. When we don’t like the way particular schools perform, we blame the teachers and restrict their resources.

Compare this with our approach to our military: when results on the ground are not what we hoped, we think of ways to better support soldiers. We try to give them better tools, better weapons, better protection, better training. And when recruiting is down, we offer incentives.

We have a rare chance now, with many teachers near retirement, to prove we’re serious about education. The first step is to make the teaching profession more attractive to college graduates. This will take some doing.

At the moment, the average teacher’s pay is on par with that of a toll taker or bartender. Teachers make 14 percent less than professionals in other occupations that require similar levels of education. In real terms, teachers’ salaries have declined for 30 years. The average starting salary is $39,000; the average ending salary — after 25 years in the profession — is $67,000. This prices teachers out of home ownership in 32 metropolitan areas, and makes raising a family on one salary near impossible.

[readon2 url="http://www.nytimes.com/2011/05/01/opinion/01eggers.html?_r=1&pagewanted=print"]Continue reading...[/readon2]

It's National Teacher Day!

Today is National Teacher Day!

National Teacher Day

On this day we would like to thank everyone who has supported Join the Future, and in doing so helped to support our goals.

  • Advocate for great public schools
  • Continually build a strong network of community support
  • Promote policies that improve public education
  • Foster respect for public school teachers and education support professionals

White Hat Management Political Contributions

Plunderbund has an article detailing the failings of White Hat Management as a charter school management company, and in light of the news of the GOP plan to privatize public education, the political crosswinds that created the plan
The goal of this budget seems pretty obvious. White Hat has a problem with their schools and boards and sponsors so the legislature just removes them from the picture, allowing White Hat to drirectly start up schools as for-profit companies, eliminating the hassle of oversight. White Hat has a problem with people questioning how they use public funds provided by the state, so the legislature just makes those funds private as soon as they are delivered to White Hat.

Some of the language in this budget is so specifically directly at helping White Hat that it wouldn’t surprise me at all if Brennan’s lawyers write it themselves. Whether it’s true or not, Ohio’s GOP politicians appear to be paying back one of their largest campaign donors with millions of state dollars while completely disregarding the educational needs of thousands of Ohio’s students.
The whole article is worth a read.

Just to highlight the scope of Brennan's political largese, here is a list of the almost $1.6 million in political campaign contributions made by some of the main players involved with White Hat Management since 2000.

Cycle Amount Contributor Name Recipient Name
2010 $50,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2010 $50,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2010 $30,000 BRENNAN, ANN AMER OHIO REPUBLICAN PARTY - STATE CANDIDATE FUND
2010 $30,000 BRENNAN, ANN AMER OHIO REPUBLICAN PARTY - STATE CANDIDATE FUND
2010 $30,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - STATE CANDIDATE FUND
2010 $25,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2010 $25,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2010 $17,075 BRENNAN, ANN AMER REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2010 $17,075 BRENNAN, DAVID L REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2010 $17,000 BRENNAN, DAVID L OHIO HOUSE REPUBLICAN CAMPAIGN CMTE
2010 $17,000 BRENNAN, DAVID L REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2010 $17,000 BRENNAN, ANN AMER REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2010 $17,000 BRENNAN, DAVID L OHIO HOUSE REPUBLICAN CAMPAIGN CMTE
2010 $11,396 BRENNAN, ANN AMER MANDEL, JOSH
2010 $11,396 BRENNAN, DAVID L KASICH, JOHN & TAYLOR, MARY
2010 $11,395 BRENNAN, DAVID L KASICH, JOHN & TAYLOR, MARY
2010 $11,395 BRENNAN, ANN AMER HUSTED, JON
2010 $11,395 BRENNAN, ANN AMER KASICH, JOHN & TAYLOR, MARY
2010 $11,350 BRENNAN, ANN AMER WAGONER, MARK
2010 $11,350 BRENNAN, DAVID L NIEHAUS, TOM
2010 $11,350 BRENNAN, ANN AMER FABER, KEITH
2010 $11,350 BRENNAN, ANN AMER NIEHAUS, TOM
2010 $11,300 BRENNAN, ANN AMER HUSTED, JON
2010 $11,300 BRENNAN, ANN AMER MANDEL, JOSH
2010 $11,000 BRENNAN, DAVID L DEWINE, MIKE
2010 $11,000 BRENNAN, ANN AMER MORGAN, SETH
2010 $11,000 BRENNAN, DAVID L BATCHELDER, WILLIAM G
2010 $11,000 BRENNAN, ANN AMER BATCHELDER, WILLIAM G
2010 $11,000 BRENNAN, DAVID L AMSTUTZ, RON
2010 $11,000 BRENNAN, ANN AMER HUFFMAN, MATT
2010 $11,000 BRENNAN, DAVID L BATCHELDER, WILLIAM G
2010 $11,000 BRENNAN, ANN AMER AMSTUTZ, RON
2010 $6,000 BRENNAN, DAVID L YOST, DAVID A
2010 $5,500 BRENNAN, DAVID L BACON, KEVIN
2010 $5,000 BRENNAN, DAVID L YOST, DAVID A
2010 $4,000 BRENNAN, ANN AMER OHIO REPUBLICAN PARTY - STATE CANDIDATE FUND
2010 $3,125 MANNA, ANTHONY OHIO REPUBLICAN PARTY - STATE CANDIDATE FUND
2010 $250 WEBER III, JOSEPH R MANDEL, JOSH
2010 $250 RICE, MARK A MANDEL, JOSH
2010 $250 MESSER JR, QUENTIN MANDEL, JOSH
2010 $250 WEBER III, JOSEPH R LAROSE, FRANK
2010 $100 MESSER JR, QUENTIN HUSTED, JON
2008 $100,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2008 $100,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2008 $100,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2008 $100,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2008 $50,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2008 $50,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - EXECUTIVE CMTE
2008 $30,000 BRENNAN, DAVID L OHIO REPUBLICAN PARTY - STATE CANDIDATE FUND
2008 $25,000 BRENNAN, DAVID L MR Republican National Cmte
2008 $16,000 BRENNAN, DAVID L REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2008 $16,000 BRENNAN, DAVID L REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2008 $16,000 BRENNAN, ANN AMER REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2008 $16,000 BRENNAN, ANN AMER REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2008 $15,000 BRENNAN, DAVID L OHIO HOUSE REPUBLICAN CAMPAIGN CMTE
2008 $12,500 BRENNAN, DAVID L OHIO HOUSE REPUBLICAN CAMPAIGN CMTE
2008 $10,650 BRENNAN, ANN AMER HARRIS, BILL
2008 $10,000 BRENNAN, DAVID L BACON, KEVIN
2008 $10,000 BRENNAN, ANN AMER TAYLOR, MARY
2008 $10,000 BRENNAN, DAVID L TAYLOR, MARY
2008 $10,000 BRENNAN, ANN AMER BATCHELDER, WILLIAM
2008 $10,000 BRENNAN, ANN AMER BATCHELDER, WILLIAM
2008 $10,000 BRENNAN, DAVID L BATCHELDER, WILLIAM
2008 $10,000 BRENNAN, ANN AMER COLEY, BILL
2008 $10,000 BRENNAN, ANN AMER DOLAN, MATTHEW
2008 $10,000 BRENNAN, DAVID L HOTTINGER, JAY
2008 $10,000 BRENNAN, DAVID L DOLAN, MATTHEW
2008 $10,000 BRENNAN, DAVID L HUSTED, JON
2008 $10,000 BRENNAN, DAVID L PATTON, THOMAS
2008 $10,000 BRENNAN, DAVID L NIEHAUS, TOM
2008 $10,000 BRENNAN, ANN AMER CAREY JR, JOHN A
2008 $10,000 BRENNAN, ANN AMER HUSTED, JON
2008 $10,000 BRENNAN, ANN AMER JONES, SHANNON
2008 $5,000 BRENNAN, DAVID L HITE, CLIFFORD
2008 $5,000 BRENNAN, DAVID L GINTER, TIM
2008 $5,000 BRENNAN, ANN AMER HOTTINGER, JAY
2008 $2,500 BRENNAN, DAVID L SCHLICHTER, JOHN
2008 $2,300 HARRISON, E D MS George V Voinovich (R)
2008 $2,300 BRENNAN, ANN Lamar Alexander (R)
2008 $2,300 BRENNAN, DAVID Lamar Alexander (R)
2008 $2,300 BRENNEN, NANCY J George V Voinovich (R)
2008 $2,300 BRENNAN, DAVID L James D Jordan (R)
2008 $1,300 BRENNAN, DAVID L George V Voinovich (R)
2008 $1,000 HARRISON, E D MS George V Voinovich (R)
2008 $1,000 BRENNAN, DAVID L George V Voinovich (R)
2008 $1,000 BRENNAN, ANN AMER TIFFANY, TOM
2008 $1,000 BRENNAN, ANN AMER DARLING, ALBERTA
2008 $1,000 BRENNAN, ANN AMER COUGHLIN, KEVIN
2008 $500 BRENNAN, ANN AMER EGELHOFF, JO
2008 $500 BRENNAN, ANN AMER HUSTED, JON
2008 $500 BRENNAN, ANN AMER DAVIS, BRETT
2008 $500 BRENNAN, ANN AMER PETROWSKI, JERRY J
2008 $500 BRENNAN, ANN AMER MURSAU, JEFFREY L
2008 $500 BRENNAN, ANN AMER NERISON, LEE
2008 $500 BRENNAN, ANN AMER HINES, J A
2008 $500 BRENNAN, ANN AMER HELLMAN, DAN
2008 $500 BRENNAN, ANN AMER NERISON, LEE
2008 $500 BRENNAN, ANN AMER HONADEL, MARK
2008 $500 BRENNAN, ANN AMER TOWNS, DEBI
2008 $500 BRENNAN, ANN AMER RIPP, KEITH
2008 $500 BRENNAN, ANN AMER VAN ROY, KARL
2008 $500 BRENNAN, ANN AMER FRISKE, DON
2008 $500 BRENNAN, ANN AMER MURTHA, JOHN
2008 $500 BRENNAN, ANN AMER WILLIAMS, MARY
2008 $500 BRENNAN, ANN AMER DOCKERY, PAULA
2008 $400 BRENNAN, ANN AMER HUEBSCH, MIKE
2008 $200 BRENNAN, DAVID L George V Voinovich (R)
2008 $200 BRENNAN, ANN AMER VOS, ROBIN J
2008 $200 BRENNAN, ANN AMER COLE, RANDY
2006 $15,000 BRENNAN, DAVID L REPUBLICAN SENATE CAMPAIGN CMTE OF OHIO
2006 $10,000 BRENNAN, ANN GREEN, MARK A (G)
2006 $10,000 BRENNAN, DAVID L CAREY JR, JOHN A
2006 $10,000 BRENNAN, DAVID L TAYLOR, MARY
2006 $10,000 BRENNAN, DAVID L HUSTED, JON
2006 $10,000 BRENNAN, DAVE GREEN, MARK A (G)
2006 $10,000 BRENNAN, DAVID L PETERSON, JON
2006 $10,000 BRENNAN, DAVID L HARTMANN, GREG
2006 $10,000 BRENNAN, DAVID L BRADLEY, JENNETTE B
2006 $10,000 BRENNAN, DAVID L LYNCH, DAVID
2006 $7,500 BRENNAN, DAVID L CAREY JR, JOHN A
2006 $5,000 BRENNAN, DAVID L TAYLOR, MARY
2006 $5,000 BRENNAN, DAVID L SEITZ, BILL
2006 $4,000 BRENNAN, DAVID L PETERSON, JON
2006 $2,500 BRENNAN, DAVID L LUTHER, BRANT
2006 $2,500 BRENNAN, DAVID L BACON, KEVIN
2006 $2,500 BRENNAN, DAVID L MARTIN, EARL J
2006 $2,500 BRENNAN, DAVID L STIVERS, STEVE
2006 $2,500 BRENNAN, DAVID L CATES, GARY
2006 $2,500 BRENNAN, DAVID L NIEHAUS, TOM
2006 $2,500 BRENNAN, DAVID L HOPPEL, JIM
2006 $2,500 BRENNAN, DAVID L STIVERS, STEVE
2006 $2,500 BRENNAN, DAVID L HAYES, WILLIAM C
2006 $2,100 BRENNAN, DAVID L Craig Foltin (R)
2006 $2,100 BRENNAN, DAVID L Craig Foltin (R)
2006 $2,000 BRENNAN, DAVID L PATTON, THOMAS F
2006 $2,000 BRENNAN, DAVID L MARTIN, EARL J
2006 $2,000 BRENNAN, DAVID L WEBSTER, SHAWN N
2004 $2,500 BRENNAN, DAVID L HAGAN, JOHN P
2004 $2,500 BRENNAN, DAVID L SETZER, ARLENE J
2004 $2,500 BRENNAN, DAVID L CALVERT, CHARLES E
2004 $2,500 BRENNAN, DAVID L NIEHAUS, TOM
2004 $2,500 BRENNAN, DAVID L CATES, GARY
2004 $2,500 BRENNAN, DAVID L PADGETT, JOY
2004 $2,500 BRENNAN, DAVID L PETRO, JIM
2004 $2,000 BRENNAN, NANCY J MRS George W Bush (R)
2004 $2,000 BRENNAN, ANN A MRS George W Bush (R)
2004 $1,000 BRENNAN, NANCY Steven C LaTourette (R)
2004 $1,000 BRENNAN, NANCY Mike DeWine (R)
2004 $1,000 BRENNAN, NANCY Bill McCollum (R)
2004 $1,000 KROHN, MARK E MR George W Bush (R)
2004 $1,000 MORRIS, JOHN C MR George W Bush (R)
2004 $1,000 BRENNAN, NANCY J George V Voinovich (R)
2004 $1,000 BRENNAN, NANCY J George V Voinovich (R)
2004 $500 BRENNAN, DAVID SPADY, FAWN
2002 $2,500 BRENNAN, DAVID L RAUSSEN, JAMES
2002 $2,500 BRENNAN, DAVID L RAUSSEN, JAMES
2002 $1,000 BRENNAN, NANCY J George V Voinovich (R)
2002 $1,000 BRENNAN, NANCY Mike DeWine (R)
2000 $1,500 BRENNAN, DAVID L GARDNER, RANDALL
2000 $1,000 BRENNAN, NANCY J George V Voinovich (R)

Data taken from www.transparencydata.com.

(c) Join the Future