Guest Post: Thoughts about teacher evaluation

A guest post by Robert Barkley, Jr., Retired Executive Director, Ohio Education Association, Author: Quality in Education: A Primer for Collaborative Visionary Educational Leaders and Leadership In Education: A Handbook for School Superintendents and Teacher Union Presidents, Worthington, Ohio – rbarkle@columbus.rr.com

Thoughts about teacher evaluation

As it often has over the 50+ years I’ve been involved in public education, teacher evaluation is once again getting considerable attention.

And as is too often the case, many who are discussing it have little idea what they’re talking about – to put it mildly.

First, there can be no meaningful discussion of this topic unless and until the parties come to a clear and shared agreement as to what are the purpose and corollary objectives of education in the first place. Without doing so any process of evaluation establishes the educational purpose and objectives extraneously and inappropriately. Thus, in almost all cases, the discussion of teacher evaluation is entirely off base and counterproductive to say the least.

For example, I have concluded, after extensive study and discussions over many years that the fundamental purpose of education is: The purpose of education is to preserve and nurture an abiding enthusiasm for learning and an unending curiosity, and to first and foremost guide students to make sense out of their current reality.

Now one can argue with this conclusion, but the point is that for any evaluation of teacher performance, or the performance of any other worker, to be of serious consequence, such a statement of purpose must be firmly established and shared by all those evolved. Rarely have I come upon a district or school that has satisfactorily completed this first step of leading to any worthwhile evaluation system.

Second, most psychologists that I have studied I think would agree that most workers, and teachers in particular, want to do a good job. In fact, it has been long established that those who enter teaching have this intrinsic and altruistic drive to do well to an even greater extent than do those entering many other professions.

And if one accepts that premise, then top-down, punitive, and competitive evaluation will have greater negative consequences than positive ones. If that is the case, then a system of non-threatening feedback will be the most productive approach to set in place.

Over these many years the best of such approaches is one labeled “360-degree feedback.” In this system, once purpose is established an appropriate context determined, everyone in the system is provided feedback as to his or her performance from all directions. This would mean that each teacher would be provided feedback from students, colleagues, parents, support personnel, and supervisors. Each employee in that system would receive the same such feedback. This means that every principal would receive feedback from the entire faculty.

And let me emphasize the “non-threatening” part of such a system. This means that the feedback you receive is yours and yours alone. No one else would see it unless you choose to share it. The theory in all this is of course that, given a natural desire to do well and improve, we will all make appropriate changes and seek guidance when necessary.

Some would say this is a naïve and utopian approach. I have been involved in such a system. It works. And as one can easily see, there is no place for merit pay in such and system and it naturally encourages teamwork and collaboration, which are the hallmark of all successful enterprises.

Illinois' New Teacher Law: Model for Other States, or Outlier?

A good article on the collaboration that led to a near unanimous acceptance of sweeping education reforms in Illinois

Illinois Gov. Pat Quinn today signed into law a sweeping measure that has the potential to significantly reshape the teaching profession in his state by linking educators' tenure, hiring, and job security to performance, rather than to seniority.

Educators and advocates have spent months debating the importance of Senate Bill 7. Does it go far enough? Will it drive improvements in teacher quality and classroom instruction?

This much is clear: The process used to approve the measure in Illinois, a state dominated by Democrats, stands in sharp contrast to the harder-edged approach taken in Republican-dominated states, such as Wisconsin, Ohio, and Idaho by leaders who have waged public battles with teachers' unions.

That contrast could prove a defining issue in coming state and national elections. Democrats—most notably two Illinois natives, President Barack Obama and U.S. Secretary of Education Arne Duncan—have called for cooperation between unions and policymakers. Republicans like Wisconsin Gov. Scott Walker and Ohio Gov. John Kasich have battled labor groups in supporting laws they argued were necessary to bring down costs for school districts and taxpayers.

Which approach will prove more successful? We may not know for some time. Several aspects of Illinois' new law, and a separate teacher-evaluation measure approved by the state last year, will prove tricky to implement.

In Wisconsin, a measure approved by Gov. Walker and GOP lawmakers, which restricts the collective bargaining rights of teachers and many other public workers, is now being challenged in court.

In Ohio, Gov. Kasich and fellow Republicans backed a law that curbed collective bargaining powers of teachers and changed how they are paid, drawing a hostile reaction from unions. Opponents of Ohio's measure are now pushing to overturn that law through a referendum. And let's not forget Idaho, where a package of laws that phase out tenure and restrict bargaining powers also face a ballot challenge.

[readon2 url="http://blogs.edweek.org/edweek/state_edwatch/2011/06/illinois_governor_signs_teacher_law_a_model_for_other_states.html"]Continue reading...[/readon2]

When It Comes To How We Use Evidence, Is Education Reform The New Welfare Reform?

Part of our ongoing effort to bring forth interesting articles covering a range of education realted topics.

There are several similarities between the bipartisan welfare reform movement of the 1990s and the general thrust of the education reform movement happening today. For example, there is the reliance on market-based mechanisms to “cure” longstanding problems, and the unusually strong liberal-conservative alliance of the proponents. Nevertheless, while calling education reform “the new welfare reform” might be a good soundbyte, it would also take the analogy way too far.

My intention here is not to draw a direct parallel between the two movements in terms of how they approach their respective problems (poverty/unemployment and student achievement), but rather in how we evaluate their success in doing so. In other words, I am concerned that the manner in which we assess the success or failure of education reform in our public debate will proceed using the same flawed and misguided methods that were used by many for welfare reform.

[readon2 url="http://shankerblog.org/?p=2701"]Continue reading...[/readon2]

Teacher responds to Kasich request for input

Yesterday, Gov. Kasich tweeted a request for input from teachers

NBC4i ran a news report, with a response from Worthington teacher, and Central OEA President, Scott DiMauro

Scott also shared his submission to the Governor with us

I have to ask what your purpose is in moving to this system. If you're genuinely interested in improving student achievement, you need to make sure performance measures are appropriate, all educators have an equitable opportunity to earn performance-based compensation, and you start from a strong salary schedule that recognizes the two objective factors that make the biggest difference in quality teaching: training and experience.

A one-size-fits-all approach won't work, so it's critical that local control is honored through the collective bargaining process. The first step has to be to drop the misguided policy proposals from HB 153 and begin an honest dialogue with the teachers' unions you so frequently love to disparage.

Thank you for seeking our input. I hope you're really listening.

You can leave your own responses to the Governor at his special web page.

Some Hows and Whys of Value Add Modelling

We thought it would be useful to provide a quick primer on what Value Add actually is, and how it is calculated, in somewhat explainable terms. This is a good explanation via the American Statistical Association

The principal claim made by the developers of VAM—William L. Sanders, Arnold M. Saxton, and Sandra P. Horn—is that through the analysis of changes in student test scores from one year to the next, they can objectively isolate the contributions of teachers and schools to student learning. If this claim proves to be true, VAM could become a powerful tool for both teachers’ professional development and teachers’ evaluation.

This approach represents an important divergence from the path specified by the “adequate yearly progress” provisions of the No Child Left Behind Act, for it focuses on the gain each student makes, rather than the proportion of students who attain some particular standard. VAM’s attention to individual student’s longitudinal data to measure their progress seems filled with commonsense and fairness. There are many models that fall under the general heading of VAM. One of the most widely used was developed and programmed by William Sanders and his colleagues. It was developed for use in Tennessee and has been in place there for more than a decade under the name Tennessee Value-Added Assessment System. It also has been called the “layered model” because of the way each of its annual component pieces is layered on top of another.

The model begins by representing a student’s test score in the first year, y1, as the sum of the district’s average for that grade, subject, and year, say μ1; the incremental contribution of the teacher, say θ1; and systematic and unsystematic errors, say ε1. When these pieces are put together, we obtain a simple equation for the first year:

y1 = μ1+ θ1+ ε1 (1)
or
Student’s score (1) = district average (1) + teacher effect (1) + error (1)

There are similar equations for the second, third, fourth, and fifth years, and it is instructive to look at the second year’s equation, which looks like the first except it contains a term for the teacher’s effect from the previous year:

y2 = μ2+ θ1+ θ2+ ε2 . (2)
or
Student’s score (2) = district average (2) + teacher effect (1) + teacher (2) + error (2)

To assess the value added (y2 – y1), we merely subtract equation (1) from equation (2) and note that the effect of the teacher from the first year has conveniently dropped out. While this is statistically convenient, because it leaves us with fewer parameters to estimate, does it make sense? Some have argued that although a teacher’s effect lingers beyond the year the student had her/him, that effect is likely to shrink with time.

Although such a model is less convenient to estimate, it more realistically mirrors reality. But, not surprisingly, the estimate of the size of a teacher’s effect varies depending on the choice of model. How large this choice-of-model effect is, relative to the size of the “teacher effect” is yet to be determined. Obviously, if it is large, it diminishes the practicality of the methodology.

Recent research from the Rand Corporation shows a shift from the layered model to one that estimates the size of the change of a teacher’s effect from one year to the next suggests that almost half of the teacher effect is accounted for by the choice of model.

One cannot partition student effect from teacher effect without information about how the same students perform with other teachers. In practice, using longitudinal data and obtaining measures of student performance in other years can resolve this issue. The decade of Tennessee’s experience with VAM led to a requirement of at least three years’ data. This requirement raises the concerns when (i) data are missing and (ii) the meaning of what is being tested changes with time.

The Ohio Department of Education has papers, here, that discuss the technical details of how VAM is done in Ohio.

BattleforKids.org provided us this information

Here's a brief example of both analysis that are used in Ohio. Both are from the EVAAS methodology produced by SAS:

Value-added analysis is produced in two different ways in Ohio:
1. MRM analysis (Multivariate Response Model, also known as the mean gain approach); and
2. URM analysis (Univariate Response Model, also known as the predicted mean approach).

The MRM analysis is used for the Ohio value-added results in grades 4-8 math and reading. It can onlybe used when tests are uniformly administered in consecutive grades. Through this approach, district, school and teacher level results are compared to a growth standard. The OAA assessments provide the primary data for this approach.

The URM analysis is used for expanded value-added results. Currently this analysis is provided through the Battelle for Kids' (BFK) SOAR and Ohio Value-Added High Schools (OVAHS) projects. The URM analysis is used when tests are not given in consecutive grades. This approach "pools" together districts that use of the same sequence of particular norm reference tests. In the URM analysis, prior test data are used to produce a prediction of how a student is likely to score on a particular test, given the average experience in that school. For example, results from prior OAA and TerraNovaT results are used as predictors for the ACT end-of-course exams. Differences between students' predictions and their actual/observed scores are used to produce school and teacher effects. The URM analysis is normalized each year based on the performance of other schools in the pool that year. This approach means that a comparison is made to the growth of the average school or teacher for that grade/subject in the pool.