How Stable are Value-Added Estimates

Via

Highlights:

• A teacher’s value-added score in one year is partially but not fully predictive of her performance in the next.
• Value-added is unstable because true teacher performance varies and because value-added measures are subject to error.
• Two years of data does a meaningfully better job at predicting value added than does just one. A teacher’s value added in one subject is only partially predictive of her value added in another, and a teacher’s value added for one group of students is only partially predictive of her valued added for others.
• The variation of a teacher’s value added across time, subject, and student population depends in part on the model with which it is measured and the source of the data that is used.
• Year-to-year instability suggests caution when using value-added measures to make decisions for which there are no mechanisms for re-evaluation and no other sources of information.

Introduction

Value-added models measure teacher performance by the test score gains of their students, adjusted for a variety factors such as the performance of students when they enter the class. The measures are based on desired student outcomes such as math and reading scores, but they have a number of potential drawbacks. One of them is the inconsistency in estimates for the same teacher when value added is measured in a different year, or for different subjects, or for different groups of students.

Some of the differences in value added from year to year result from true differences in a teacher’s performance. Differences can also arise from classroom peer effects; the students themselves contribute to the quality of classroom life, and this contribution changes from year to year. Other differences come from the tests on which the value-added measures are based; because test scores are not perfectly accurate measures of student knowledge, it follows that they are not perfectly accurate gauges of teacher performance.

In this brief, we describe how value-added measures for individual teachers vary across time, subject, and student populations. We discuss how additional research could help educators use these measures more effectively, and we pose new questions, the answers to which depend not on empirical investigation but on human judgment. Finally, we consider how the current body of knowledge, and the gaps in that knowledge, can guide decisions about how to use value-added measures in evaluations of teacher effectiveness.

[readon2 url="http://www.carnegieknowledgenetwork.org/briefs/value-added/value-added-stability/"]Continue reading...[/readon2]

Bloomberg's brain dead brainwave

Billionaire Mayor of New York, and wannabe corporate education reformer Mike Bloomberg has suggested a radically absurd idea

he said, “you would cut the number of teachers in half but you would double the compensation of them, and you would weed out all the bad ones and just have good teachers.

“Double the class size with a better teacher is a good deal for students.”

Bloomberg's opinion is based upon a misguided and factually wrong premise, one he continues to hold to

Karen Matthews, a reporter with The Associated Press, asked, via Twitter, whether the mayor saw one teacher and 62 children as a good model. The mayor’s press secretary, Stu Loeser, shot back: “Are you asking as a journalist, advocate, or mom?”

No doubts haunt the mayor. In 2008 he insisted that class-size research was “unambiguous.”

“I don’t even understand why the subject comes up anymore,” he said, adding that all that mattered was teacher quality.

Let's examine class sizes and see if they matter. Michael C. Morrison, Ph.D. has analyzed 9,000 school districts to determine the impact on class sizes and graduation. His findings are unambiguous.

District probabilities for above average graduation performance are inversely related to district pupil-teacher ratios. As class size increases district probability for above average graduation performance decrease, controlling for district per capita income (a proxy for district socio-economic status) and district total revenue per student (a district proxy for programs and services).

Here's the graph of results

This isn't the only study of course, it's a subject that has been well and extensively researched. The National Bureau of Economic Research (NBER) found

“We find that assignment to a small class increases the probability of attending college by 2.7 percentage points, with effects more than twice as large among blacks. Among those with the lowest ex ante probability of attending college, the effect is 11 percentage points. Smaller classes increase the likelihood of earning a college degree by 1.6 percentage points and shift students towards high-earning fields such as STEM (science, technology, engineering and medicine), business and economics.”

Michael Morrison has detailed further studies on the subject, here.

As for Mayor Bloomberg, he doesn't practice what he is preaching

There’s a final oddity. Among the so-called meritocratic elite, low teacher-to-child ratios are beloved. The mayor’s daughters went to Spence, where classes hover from 10 to 15. Trinity, Dalton, Riverdale, Horace Mann: All charge $35,000 or more per year, and classes rarely exceed 12 in the lower grades.

Imagine if they packed those billionaire's kids into classrooms of 63!

ODE subject matter contact info

The Ohio Department of Education (ODE) has just released this contact information document by subject matter expertise. We thought it might be useful to share with our readers if you;re trying to find the right perosn for the the right topic. Feel free to download it, or just book mark this page.

ODE Contacts by Topic

YouTube for Teachers

We thought this was interesting enough to share. Youtube, the popular video sharing site has created a channel just for educators. You can view, share and even upload educational videos. YouTube provides teachers with ten reasons why their Youtube teachers channel could be useful in the classroom:

1. Spark Lively Discussion: Engage students by showing a video relevant to their lives. Video clips can bring in different perspectives or force students to consider a new viewpoint, helping to spark a discussion.
2. Organize all the great video content you find: Playlists are YouTube’s way of allowing you to organize videos on the site. When one video ends, the playlist plays the next video without offering ‘related videos,’ thus creating a curated environment for you students.
3. Archive your work: Capture and save projects and discussions so you can refer back to them year after year. This will also help you save time as you can assign old videos to your new students.
4. Allow students to dig deeper into a subject: Give students the option to dig deeper into a subject by creating a playlist of videos related to that concept. By creating playlists of relevant videos you allow students to pursue their interests without wasting their time searching for information (or finding potentially objectionable content).
5. Get struggling students to speed up and push strong students ahead: Videos (or playlists) can help supplement in class teaching for struggling students. Students can review them at home time so you’re not forced to teacher exclusively to the middle 50%.
6. Review for upcoming exams: Turn test review and flashcards into easy-to-watch videos. This way students can hear your explanations as they study. You can also create a “test review” video students can use to study the night before the big test.
7. Create a YouTube center in your classroom: When working in stations or centers, have students use your YouTube channel to complete an assignment, freeing you upto work with small groups of students.
8. Create quizzes to accompany videos for instant feedback: Create a Google Form that students complete after watching a video. You can use this quiz to get instant feedback on what they’re learning.
9. Create Interactive Video Quests: Use YouTube annotations to create “Choose your own adventure” style video quests. You can also create a video guide.
10. Flip your classroom: If your students watch a video of the basic concepts at home you can focus in class on applying those concepts, working collaboratively with their classmates rather than simply listening to you lecture.

You can find our more here.

What teachers are telling the Governor: Day 3

Day 3 in our on going series of publishing comments provided to the Governor on his request for input on teacher evaluation and merit pay.

Subject: Hmm, how should teachers be paid?

Here's why a merit-based system for teachers has it's flaws:
- Improvements in a student's organization, self-esteem, confidence, social skills, behaviors, etc. are hard to measure.
A teacher may take a challenging social group, and improve things listed above, but still struggle with test scores.
Should they really be penalized for not meeting the test "quota" despite improving a child's life skills?
- Merit-systems have proven to lead to corruption & the abandoning of meaningful lessons for test-taking drills.
Turning a public service into a for-profit business is unfair to the already endless struggles faced by students.
- Who evaluates the teacher? ..The principal? How is there any certainty that he/she will objectively evaluate the teacher free of non-teaching-related criteria?
-Cronyism.
-Competition between teachers who are expected to collaborate for a common good (the student.)
The current process DOES have an evaluation system in place. When teachers perform unsatisfactorily, they can be assigned a "Peer Assistance Review" mentor who observes the struggling/ineffective teacher and provides support & feedback for getting him/her "back on track."

Teachers, like students, each have elements of their personalities & skills that shine above others. If we're adjusting our practices to meet the same performance criteria, how sad that many students will miss out on some of the unique talents that some teachers might be reluctant to share, or that some pioneering teachers might be unwilling to stray from the norm & try new practices.

The fact is, teachers, like a private-sector worker who personally approaches his/her boss about a raise, have already collectively agreed on what they consider a fair wage. Many of the restrictions in SB5 that limit teacher resources & put no limits on class sizes will ultimately make it even more difficult to fulfill whatever criteria is considered. Again, trying to equate successful "bottom-line" business tactics to motivating children with an endless number of variables is definitely NOT in the best interest of the teaching & learning process..
Thanks,

A lot of comments express frustration about fairness and competency of people making education decisions, such as the following

Subject: Idea

Dear Mr. Kasich,
The idea I have about "paying teachers based on performance," is that the system we have in place now works just fine. As a long-term educator with a masters degree and 3 licenses, why shouldn't I make more money than a teacher right out of college with a bachelor's degree? I have put thousands of dollars and years into getting my education and licenses. I should have tenure and job security. I should have a good paycheck and retirement. I should have good benefits now and in my retirement.

What about paying Charter and Private schools based on how they do on the Ohio Achievement Tests? Right, they don't even have to take it, so I think that would be a good place for YOU to start.

Your thoughts on changing our education system are insulting, and just show how little you know about our education system in general. The State of Ohio has the responsibility to educate our children, and our public education system does it the best. Maybe you need to review your responsibilities as a governor and provide more money to our public education system that is doing well; and would do even better if half of our money wasn't dumped into Private and Chartered schools.

Subject: Ideas for New Pay System

Remove ALL politicans from making ANY of the decisions for pay. Policitians have already made a mockery of the teaching profession and have absolutely shown no respect for educators. Obviously the policitians who voted for this nonsense have never taught one minute otherwise they would understand their are way to many variables to even suggest merit.

If this nonsense continues, ONLY educators should make the decision NOT policitans.

Finally for today, there are one or two comments that are supportive of the governor's efforts, even if their suggestions are, shall we say, "different"

Subject: Innovative Teacher Salary Idea

Hi Governor Kasich!

Depending on what amount each school district spends per pupil, allow the students $1000 to interview the teachers, and "hire" the one they want. Teachers may present the students with their educational portfolio - past student's test scores, how the teacher sets up the class, what the teacher's expectations of the student are, etc. The students then put all their money into the pot for that teacher, and if they reach their goals for the year - AYP, test scores, attendance - the teacher get's their students' bonuses. Much like "The Apprentice" with The Donald - the teacher who performs the best gets the best bonus. The students feel ownership of their teacher, and the teacher only gets beyond their base salary for bringing their class to victory. Also, the best teachers can take more than just 24 students - up to 50 students - thereby increasing their chance for a bonus (but also getting more kids in front of the best teachers) The teachers who are not "hired" by the students, have the smallest class sizes, the least chance at bonuses, and eventually are weeded out.

I think you're awesome! Keep up the good work!

HB 153 Teacher Retesting Provision Facts

House Bill 153 contains a provision for retesting teachers in the lowest ranked 10 percent of all public schools (Sec. 3319.58 below).

Sec. 3319.58
(A) As used in this section, "core subject area" has the same meaning as in section 3319.074 of the Revised Code.

(B) Each year, the board of education of each city, exempted village, and local school district, governing authority of each community school established under Chapter 3314. of the Revised Code, and governing body of each STEM school established under Chapter 3326. of the Revised Code with a building ranked in the lowest ten per cent of all public school buildings according to performance index score, under section 3302.21 of the Revised Code, shall require each classroom teacher teaching in a core subject area in such a building to register for and take all written examinations prescribed by the state board of education for licensure to teach that core subject area and the grade level to which the teacher is assigned under section 3319.22 of the Revised Code. However, if a teacher who takes a prescribed examination under this division passes that examination and provides proof of that passage to the teacher's employer, the teacher shall not be required to take the examination again for three years, regardless of the performance index score ranking of the building in which the teacher teaches. No teacher shall be responsible for the cost of taking an examination under this division.

(C) Each district board of education, each community school governing authority, and each STEM school governing body may use the results of a teacher's examinations required under division (B) of this section in developing and revising professional development plans and in deciding whether or not to continue employing the teacher in accordance with the provisions of this chapter or Chapter 3314. or 3326. of the Revised Code.

However, no decision to terminate or not to renew a teacher's employment contract shall be made solely on the basis of the results of a teacher's examination under this section until and unless the teacher has not attained a passing score on the same required examination for at least three consecutive administrations of that examination.

The Facts:

• According to ODE, this law will not take effect until the 2012-2013 school year because the ranking system to determine the lowest 10% of districts is not required to be in place until September 2012.
• Under 3319.074, the core subject areas are defined as follows:
  “Core subject area” means reading and English language arts, mathematics, science, foreign language, government, economics, fine arts, history, and geography.
• The “current examinations prescribed by the state board of education for licensure to teach that core subject area” are:
  a) Praxis II content exam(s) AND Principles of Learning and Teaching for all content areas except world languages
  b) American Council on the Teaching of Foreign Languages (ACTFL/LTI) exams (2) for world languages AND Praxis II Principles of Learning and Teaching
  c) Information about these exams and links to the testing companies can be found at the ODE website here.
  
• Every teacher would have to take a minimum of two exams
• There are only seven Praxis II testing dates per year, and not all tests are administered on all dates. Certain tests are only offered three or four times per year. It may take multiple days for teachers to take all required tests.
• Tests are administered primarily via paper and pencil at testing sites throughout the state. Some tests are moving to computer-based administration, but still must be taken at a designated testing center. Testing sites would likely not be able to handle the thousands of teachers who would be required to take tests beginning in 2012.
• Fees for taking the Praxis exams are as follows:
  a) $50 registration fee charged once per testing year
  b) Praxis II computer-based tests range from $50-150 per test
  c) Praxis II paper-delivered tests range from $65-90 with most being $80
  d) The average cost for a teacher taking ONE content test and the Principles of Learning and Teaching is $230. Teachers with multiple certificates/licenses teaching in more than one core area will cost more.
  e) World language teachers will have to take the Praxis II PLT ($50 registration + $90 test = $140) in addition to the ACTFL Oral Proficiency Interview ($134) and Writing Proficiency Test ($65) for a total of $339
  
• By using this year’s performance index scores and identifying the lowest 10% of buildings and then identifying teachers in those buildings who taught in the core areas last year, some sources estimate that over 6,000 teachers would need to be retested.
• By law, teachers are not responsible for the cost of taking these exams, so districts in the lowest 10% will be put under the additional financial burden or retesting their teachers.

This new law is bad because...

• Requiring teachers in core subjects in the lowest 10% of buildings to be retested places a huge financial burden on districts already struggling with budget cuts in these tough economic times.
• Retesting teachers wastes planning and preparation time and takes the focus off of the classroom and students when we need to focus on the essentials-- a high quality education for all Ohio students.
• Testing teachers does not help them improve their performance. Ongoing formative feedback that addresses the complexities of the teaching profession and individualized support allows teachers to improve their performance.
• The tests required for licensure are not designed to diagnose problems teaching performance and do not reflect the complexity of interacting with diverse students. They are only valid to measure knowledge of specific subjects that new K–12 educators will teach, as well as general and subject-specific teaching skills and knowledge.
• Requiring that all teachers in core areas be tested in identified buildings is unfair. Master Teachers, National Board Certified teachers and other teachers who have demonstrated practice at an advanced or accomplished level would be forced to take the tests by law. And, teachers who were never required to take these exams for certification or licensure will now be forced to take them by law.
• Testing centers will not be able to handle the huge increase in the number of test-takers this law requires.
• This retesting provision drains money away from districts and gives it to large testing corporations.