Dispatch's Misguided Missives on Testing.

From our mailbag. Dublin teacher Kevin Griffin writes:

On Monday the Dispatch Editorial Board once again wrote about education and showed that they just don’t get it. Or maybe they have seen that 15 other states have either delayed common core implementation or pulled out of the common core completely. Their dim-witted conclusion that we must have high stakes standardized tests to rank teachers and schools shows that they miss every important aspect of the testing conversation as it relates to the most important of the stakeholders, the children.

Parents should have the right to opt their child out of any state mandated standardized test. After all, isn’t the educational choice movement about (a-hem) choice. They may do this, not because of the common core, but because of the high stakes the politicians have attached to these tests.

Because of the high stakes attached to these tests teachers are teaching useless test taking skills as opposed to relevant content area. This goes completely against the “career readiness” we hear so much about from the corporate education reformers.

Since tests are being used in teacher evaluations, and will ultimately be used to rank teachers, the testing climate has changed the school environment. Teachers, either by choice or by administrative mandate, are teaching test taking strategies, giving an abundance of practice tests, or spending time teaching-to-the-test to raise scores, not because they believe it is what is best for the children.

Schools have changed their entire structure, removing curriculum and courses like music, art, and health to focus on what’s on the test. We are no longer teaching the whole child, but rather what the testing companies like Pearson and PARCC have decided is important enough to test.

The next problem with the tests is their reliability. The tests are being used in a “value-added model” (VAM) to evaluate teachers. The theory is that this complex and secretive formula can measure how much “value” a teacher adds to student learning. However an April 8th, 2014 study released by the American Statistical Association (ASA) denouncing VAM to evaluate teachers states “Ideally, tests should fully measure student achievement with respect to the curriculum objectives and content standards adopted by the state, in both breadth and depth. In practice, no test meets this stringent standard.”

The ASA report also points out that test scores do not measure a students level of creativity and that outside factors such as class size and the number of high needs students in a classroom can also have an effect of students test scores.

The Dispatch’s opinion piece also fails to recognize that the number one way to predict a student’s standardized test score is to look at their poverty level. Education policy expert Dr. Diane Ravitch has written that she can easily predict a schools’ testing outcomes based on their zip code. 50% of Ohio’s school age children qualify for free or reduced lunch. Why is it that hunger, sickness, and a stable home life are being dismissed as non-factors?

Why would a parent choose to opt out their child out of standardized tests? Maybe it’s because they don’t want the testing companies to have so much data on their kids. Maybe they know the tests are unreliable. Maybe they don’t believe the testing companies should be in charge of our classrooms. Maybe they trust that the teachers know what is best for their children without a bubblesheet exam. Maybe they know that teachers don’t want or need all these tests. Or maybe they just want their child to be well rounded and happy and to actually enjoy school.

Share your comment:

One of many nails in the VAM coffin….

Following up on the previous post, this via Better Living Through Mathematics

If you’ve read the American Statistical Association’s position on the dangers of evaluating teacher performance based on the “Value-Added Model,” you’re probably wondering how they arrived at this very sobering conclusion. As Albert Einstein was alleged to have stated, “Not everything that counts is countable, and not everything that is countable counts.” In this case, AMSTAT took that advice to heart and so strongly inveighed against VAM that they essentially labeled it a form of statistical malpractice.

In this post, I’m going to examine one of the studies that no doubt had a profound impact on the members of AMSTAT that led them to this radical (but self-evident) conclusion. In 2012, the researcher C. Kirabo Jackson at Northwestern University published a “working paper” for the National Bureau of Economic Research, a private, nonprofit, nonpartisan research organization dedicated to promoting a greater understanding of how the economy works (I’m quoting here from their website.) The paper, entitled “Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from 9th Grade Teachers in North Carolina” questions the legitimacy of evaluating a teacher based on his/her students’ test scores. Actually, it is less about “questioning” and more about “decimating” and “annihilating” the practice of VAM.

I downloaded the paper and have been reading it for the past few days. Jackson clearly has done his homework, and this paper is extremely dense in statistical analysis which is rooted in data collected by the National Educational Longitudinal Study 1988, which began with 8th graders who were surveyed on a range of educational issues as described below:

On the questionnaire, students reported on a range of topics including: school, work, and home experiences; educational resources and support; the role in education of their parents and peers; neighborhood characteristics; educational and occupational aspirations; and other student perceptions. Additional topics included self-reports on smoking, alcohol and drug use and extracurricular activities. For the three in-school waves of data collection (when most were eighth-graders, sophomores, or seniors), achievement tests in reading, social studies, mathematics and science were administered in addition to the student questionnaire.

To further enrich the data, students’ teachers, parents, and school administrators were also surveyed. Coursework and grades from students’ high school and postsecondary transcripts are also available in the restricted use dataset – although some composite variables have been made available in the public use file.

The survey was followed up in 1990, 1992, 1994 and 2000, which means that it began when students were just about to begin their high school career, and then followed up when they were in 10th and 12th grades, and followed them through post-high school, college and postgraduate life. It is one of the most statistically valid sample sets of educational outcomes available.

What should be noted is that Jackson is not an educational researcher, per se. Jackson was trained in economics at Harvard and Yale and is an Associate Professor of Human Development and Social Policy. His interest is in optimizing measurement systems, not taking positions on either side of the standardized testing debate. Although this paper should reek with indignation and anger, it makes it’s case using almost understated tone and is filled with careful phrasing like “more than half of teachers who would improve long run outcomes may not be identified using test scores alone,” and “one might worry that test-based accountability may induce teachers to divert effort away from improving students’ non-cognitive skills in order to improve test scores.”

But lets get to the meat of the matter, because this paper is 42 pages long and incorporates mind-boggling statistical techniques that account for every variable one might want to filter out to answer the question: are test scores enough to judge the effectiveness of a teacher? Jackson’s unequivocal conclusion: no, not even remotely.

The first thing Jackson does is review a model that divides the results of education into two dimensions: the cognitive effects, which can be measured by test results, and the non-cognitive effects, which are understood to be socio-behavioral outcomes, which when combined, determine adult outcomes. To paraphrase the old Charlie the Tuna commercial, it’s more than whether we want adults that test good – we also want them to be good adults. Clearly, Jackson is aiming a little higher than those who would believe that test scores are the end result of “good teaching.” He’s focusing on what “non-cognitive” effects a teacher can have on a student, which includes things like diminishing their rates of truancy and suspensions, improving their grades (which are different from test scores) and helping increase the likelihood that they will attend college.

Which poses the less than obvious question: if teachers have an effect on both cognitive and non-cognitive outcomes, are they correlated or independent? That is, if a teacher is effective in raising test scores, will that lead to less truancy, fewer suspensions, better grades and less grade retention? Even more interesting is the idea that teachers could be more effective on one scale while being low on the other: is it possible for a teacher to be very effective at improving a student’s non-cognitive functioning while not having an effect on his/her test scores?

By page 4, Jackson’s paper starts to draw blood: using the results of the NELS 1988, Jackson concludes that a standard deviation increase in non-cognitive ability in 8th grade is associated with fewer arrests and suspensions, more college-going and better wages than the same standard deviation improvement in test scores. It’s almost as if Jackson is telling us, “hey, 8th grade teachers: want to improve your students future life? Spend less time on test prep and more time helping them show up at school, staying out of trouble and improving their actual grades.”

This alone would be enough of a takeaway, but this incredibly dense paper continues to hammer away at any thought that test scores are meaningful in any way: in the same paragraph, Jackson states that a teacher’s effect on college-going and wages may be as much as three times larger than predicted based on test scores alone. HFS! Oh, and just to make things more interesting, it is followed by this statement: “As such, more than half of teachers who would improve long run outcomes may not be identified using test scores alone.”

To summarize, we’re only in the middle of page 4 of this paper, and we’ve already learned the following:

a) Teachers have an effect on both cognitive skills of their students, and non-cognitive skills of their students. The first leads to higher test scores, the second leads to more college going, fewer arrests and better wages.

b) In 8th grade, non-cognitive achievement is a better predictor of college going and higher wages, as well as fewer arrests and suspensions, than test scores.

c) A teacher’s effect on these “non-cognitive” outcomes is as much as 300% greater than can be measured using test scores.

But wait, there’s more!

Okay, I’m only below the middle of page 4, and already I’ve read three conclusions that essentially kill off any legitimacy to judging a teacher’s effectiveness based on test scores, and the good stuff has even gotten started!

What Jackson is up to in his paper is something bigger, way bigger: it would be possible to argue at this point that somehow cognitive and non-cognitive skills, while both responsible in some part to positive adult outcomes, are still correlated; that is, if you improve the test scores, the other non-cognitive stuff will come along as a bonus. This is where Jackson goes for the jugular, and, as is typical of research papers, he essentially “buries the lead.”

“This paper presents the first evidence that teachers have meaningful effects on non cognitive outcomes that are strongly associate with adult outcomes and are not correlated with test scores.” (Emphasis mine, italics his, by the way.)

Share your comment:

Nations Top Statisticians Question Use of Value-Add

The American Statistical Association (ASA) the nationas leading group of statisticians recently released a statement of Valuee-add, that calls into question the whole premise of using this statistical measure for the purposes of evaluating teachers job performance. Their comments below:

  1. VAMs are complex statistical models, and high-level statistical expertise is needed to
    develop the models and [emphasis added] interpret their results.
  2. Estimates from VAMs should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model. These limitations are particularly relevant if VAMs are used for high-stakes purposes.
  3. VAMs are generally based on standardized test scores, and do not directly measure
    potential teacher contributions toward other student outcomes.
  4. VAMs typically measure correlation, not causation: Effects – positive or negative –
    attributed to a teacher may actually be caused by other factors that are not captured in the model.
  5. Under some conditions, VAM scores and rankings can change substantially when a
    different model or test is used, and a thorough analysis should be undertaken to
    evaluate the sensitivity of estimates to different models.
  6. VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools.
  7. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.
  8. Attaching too much importance to a single item of quantitative information is counter-productive—in fact, it can be detrimental to the goal of improving quality.
  9. When used appropriately, VAMs may provide quantitative information that is relevant for improving education processes…[but only if used for descriptive/description purposes]. Otherwise, using VAM scores to improve education requires that they provide meaningful information about a teacher’s ability to promote student learning…[and they just do not do this at this point, as there is no research evidence to support this ideal].
  10. A decision to use VAMs for teacher evaluations might change the way the tests are viewed and lead to changes in the school environment. For example, more classroom time might be spent on test preparation and on specific content from the test at the exclusion of content that may lead to better long-term learning gains or motivation for students. Certain schools may be hard to staff if there is a perception that it is harder for teachers to achieve good VAM scores when working in them. Overreliance on VAM scores may foster a competitive environment, discouraging collaboration and efforts to improve the educational system as a whole.

You can read their whole document below.

ASA Statement on Using Value-Added Models for Educational Assessment

Share your comment:

Living in John Kasich's Ohio

Via Ohioans united

One of the best things we can do for every child is to see that he or she has a quality education. That’s something that will put them on a path to a bright future. Properly-funded public schools are essential if we are to ensure that all children – no matter where they live or who their parents are- receive the education they deserve. That’s why it is so disheartening to see Gov. Kasich shifting millions and millions of dollars from public schools into failing charter schools.

And remember – every dollar given to a failing charter school is a dollar less for traditional public schools. Since taking office, Kasich has cut more than half a billion dollars in state funding for public schools. The result is over 90% of public schools have had to make cuts, including teachers and course offerings.

To me, as a teacher and a mom, it is the height of irresponsibility for the governor to be taking money out of a system that provides a quality education to Ohio’s children and shift the funds into schools that are failing at an alarming rate. Ohio’s kids deserve better.

That’s why I’ve joined with Ohioans United to get the word out about Gov. Kasich’s harmful education policy. Watch the ad below and share it with your friends and family to make sure they know the truth about John Kasich’s record.

Share your comment:

So-Called Right to Work Is Still Wrong for Workers

Via HuffPo

The course is even more tenuous for ALEC and its supporters in the other states, with Ohio looking at a possible state constitutional amendment being placed on the November ballot while proposed measures in Maine and Pennsylvania appear to be the least likely to be approved.

There are many reasons for this. Despite what corporate cronies claim, so-called right-to-work doesn't create an environment that is good for workers or companies. In fact, a recent quality of life report released by Politico found the bottom five states -- Mississippi, Louisiana, Arkansas, Tennessee and Alabama -- are right-to-work states. Meanwhile, four out of five with the highest quality of living -- New Hampshire, Minnesota, Vermont and Massachusetts -- are free bargaining states.

Right-to-work is a ruse. These laws depress wages, resulting in workers making about $1,500 less than those living in non-RTW states. They are also more likely not to receive health insurance and more likely to work in a dangerous workplace. In addition, it is proven not to be a deciding factor in where businesses locate.

Lawmakers must resist the cheap corporate rhetoric pushed by ALEC and others that makes right-to-work seem like a solution. It isn't. All it seems to help create is less pay, less freedom in the workplace and maybe most important of all, a smaller middle class. If that's seen by some as progress, government should be taking a pass on it.

Share your comment:

Teacher Supply and Demand in Ohio

The Ohio Education Research Center has just published an interesting report titled "2013 Teacher Supply and Demand in Ohio". Among its findings are:
Retirement rule changes have already resulted in major declines in teacher stocks. On average in Ohio, 6,000 teachers retire each year according to STRS annual reports. It is anticipated that high levels of retirement among teachers and administrators with 35+ years of service will continue. Retirement rates will likely level off and fall after July 2015 reflecting tighter eligibility requirements and less generous benefits. Mid-career teachers and administrators will find themselves “locked into” the system with less attractive retirement options making departures less likely. This is due to substantial pension wealth losses created by leaving the system early, especially if inflation accelerates. If low interest rates continue, further benefit reductions may be necessary.

The supply of new teacher license holders in Ohio varies across grade ranges and subject areas. Over a quarter of all new teachers licensed in Ohio in 2012 were in early childhood or prekindergarten through 3rd grade indicating a disproportionately large supply of early childhood teachers relative to teachers in the 4-12 grade range. Relatively few new teachers are trained in math and science compared to those trained in language arts and social studies.

There is a shift moving students and teachers from private schools to community schools. Private school enrollments and number of teachers are shrinking, while the number of community schools is growing. The first community schools were created in Ohio in 1998. Today they comprise slightly less than 10% of Ohio’s entire educational system. The number of people holding licenses for highly-skilled administrative jobs outnumbers the actual positions in Ohio. There are roughly five people who hold a superintendent license for every superintendent job, three people who hold a principal license for every principal position, and about two people who hold a financial license for every treasurer position. This oversupply has caused inflation adjusted administrative pay to fall over time.

Many people trained as teachers never become licensed. One sixth of graduates with a bachelor’s degree or master’s degree in education were never licensed to teach in Ohio within five years of graduation.
Teacher supply is a big problem. Not only are many Ed graduates not becoming teachers, but many teachers quit within their first 5 years. Couple this with the uneven distribution of teachers across grades and subjects and one can see it is a problem that needs addressed. Instead what we have from law makers are policies designed to drive out even more teachers from the profession, and make the profession as unattractive to prospective new educators as possible. Provisions included in the Ohio House's substitute SB229 bill are obvious examples of this. The sub. bill added language that “prohibits a school district from assigning students to a teacher who has been rated “ineffective” for two consecutive school years”. the measure of being ineffective of course being the highly flawed and unstable OTES.

The report also notes that poor quality charter schools are driving out private schools. No doubt many parents are see charter schools as a fully funded private schooling system, rather than an alternative to traditional public schools. Given how charter schools are secretly operating their finances and increasingly being run by for-profit management companies, this isn't an erroneous assumption. What is erroneous however is the quality of the education being delivered by most charter schools in Ohio.

The financial mismanagement of charter schools has become so prevalent, even the dysfunctional Ohio General Assembly are beginning to notice
Democrats in the Ohio House and Senate pledged yesterday to introduce legislation that would require more transparency and accountability for operators and sponsors of charter schools.

Sen. Joe Schiavoni, of Boardman, and Rep. John Patrick Carney, of Columbus, said they will introduce companion bills in coming days. If approved, the legislation would provide the public with details about how privately operated charter schools spend the millions in tax dollars they receive each year.

“In 1998, there were only 15 charter schools in Ohio. Those 15 charters received $11 million in state funding. Today we spend $900 million on nearly 400 charter schools,” Schiavoni, the Senate minority leader, said at a Statehouse news conference.

“The growing problem is we don’t know how most of these taxpayer dollars are being spent.”

You can read the entire Ohio Education Research Center report here.

Share your comment:

Get Involved

 First Name
 Last Name
 School District