AP LANGUAGE AND COMPOSITIONDEBATE #6 SOURCES: TEACHER EVALUATIONS

Source A

(Association for Supervision and Curriculum Development)

What Research Says About… / Using Value-Added Measures to Evaluate TeachersJane L. DavidEven the staunchest advocates of performance-based pay don't think it's fair to judge teachers' effectiveness solely on the basis of end-of-year test scores, without regard to where the teachers' students started at the beginning of the year. Can value-added measures, which show students' growth from one year to the next, solve this problem?

What's the Idea?

The claim for value-added measures is that they capture how much students learn during the school year, thereby

putting teachers on a more level playing field as they aim for tenure or additional pay.

What's the Reality?

End-of-year test scores do not show how much students learned that year in that class, so measures that take into account where students started are surely an improvement. However, such measures of growth are only a starting point. Making judgments about individual teachers requires sophisticated analyses to sort out how much growth is probably caused by the teacher and how much is caused by other factors. For example, students who are frequently absent tend to have lower scores regardless of the quality of their teacher, so it is vital to take into account how many days students are present. Thus, to be fair and to provide trustworthy estimates of teacher effectiveness, value-added measures require complicated formulas that take into account as many influences on student achievement as possible.

What's the Research?

A growing number of researchers are studying whether value-added measures can do a good job of measuring the contribution of teachers to test score growth. Here I summarize a handful of analyses that shed light on two questions.

How Fair Are Value-Added Measures?The trustworthiness of a value-added measure depends on how it is defined and calculated. Koretz (2008) argues that measuring the value added by the teacher requires knowing not only how much students have learned in a given year, but also the rates at which those particular students learn. Students reading well above grade level, for example, would be expected to learn faster than struggling readers. Value-added measures should take these differences into account.Rothstein (2008) worries that test score gains are biased because students are not randomly assigned to teachers. For example, comparing teachers whose classrooms are treated as dumping grounds for troubled students with teachers whose classrooms contain the best-behaved students will favor the latter.

RAND researchers examined whether giving students different tests would lead to different conclusions about teacher effectiveness (Lockwood et al., 2006). They calculated value-added ratings of middle school teachers in a large school district on the basis of their students' end-of-year scores from one year to the next on two different math subtests. They found large differences in teachers' apparent effectiveness depending on which subtest was used. The researchers concluded that if judgments about teacher effectiveness vary simply on the basis of the test selected, administrators should use caution in interpreting the meaning of results from value-added measures.Researchers have also identified other threats to the trustworthiness of value-added measures. Goldhaber and Hansen (2008) looked at the stability of such measures over time: Do value-added analyses identify the same teachers as effective every year? Using a large data set from North Carolina, they found that estimates of teacher effectiveness were not the same across years in reading or math. Other researchers (for example, Koretz, 2008) question whether it is even possible to compare gains from one year to the next using tests that do not measure the same content.

Are Value-Added Measures More Accurate Than Traditional Evaluations?Traditional methods for evaluating teacher effectiveness have their own problems—for example, infrequent or poor classroom observations or administrator bias. In fact, the persistently subjective nature of these more traditional evaluations is what fuels the current enthusiasm among policymakers for basing teacher evaluation on "objective" test scores.Do value-added measures do a better job of judging teacher effectiveness than traditional teacher evaluations do? Researchers have looked at this question by comparing results from the two approaches.When Jacob and Lefgren (2008) looked at 201 teachers in 2nd through 6th grade, they found a strong relationship between principals' evaluations and value-added ratings (based on student math and reading scores) of the same teachers. The researchers then asked which method did a better job of predicting how the teachers' future classes would score. They found that either method was fairly accurate in predicting which teachers would be in the top and bottom 20 percent the following year in terms of their students' test scores. Although value-added measures did a slightly better job of predicting future test scores, adding principal ratings increased the accuracy of these predictions.Studies of teacher evaluation systems in Cincinnati, Ohio, and Washoe County, Nevada, also found that value-added measures and well-done evaluations based on principal observations produced similar results (Milanowski, Kimball, & White, 2004).

What's One to Do?

From the federal government to foundations, the pressure is on to use student test score gains to evaluate teachers. Yet doing so in a credible and fair way is a complex and expensive undertaking with no guarantee that intended improvements in teaching and learning will result. What's more, it is not clear that value-added measures yield better information than more traditional teacher evaluation practices do.The complexity and uncertainty of measuring student achievement growth and deciding how much responsibility for gains to attribute to the teacher argue against using such measures for high-stakes decisions about individuals. To protect teachers from erroneous and harmful judgments, a consensus is emerging that we need multiple measures that tap evidence of good teaching practices as well as a variety of student outcomes, including but not limited to standardized test score gains. According to a recent study (Coggshall, Ott, & Lasagna, 2010), most teachers support such a multiple-measures approach.Investing in expensive data analysis systems may not be as important as investing in ways of measuring teacher effectiveness that can identify the specific supports teachers need to improve their practice.

References

Coggshall, J. G., Ott. A., & Lasagna, M. (2010). Convergence and contradictions in teachers' perceptions of policy reform ideas. (Retaining Teacher Talent, Report No. 3). Naperville, IL: Learning Point Associates and New York: Public Agenda. Available:www.learningpt.org/expertise/educatorquality/genY/CommunicatingReform/index.php

Goldhaber, D., & Hansen, M. (2008). Is this just a bad class? Assessing the stability of measured teacher performance. (Working paper #2008-5). Seattle: Center on Reinventing Public Education, University of Washington. Available: www.crpe.org/cs/crpe/view/csr_pubs/249

Jacob, B. A., & Lefgren, L. (2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics, 26(1), 101–136. Available:http://ideas.repec.org/a/ucp/jlabec/v26y2008p101-136.html

Koretz, D. (2008, Fall). A measured approach: Value-added models are a promising improvement, but no one measure can evaluate teacher performance. American Educator, 18–27, 39.

Lockwood, J. R., McCaffrey, D. F., Hamilton, L. S., Stecher, B., Le, V-N., & Martinez, F. (2006). The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Santa Monica, CA: RAND Corporation. Available:www.rand.org/pubs/reports/2009/RAND_RP1269.pdf

Milanowski, A., Kimball, S., & White, B. (2004). The relationship between standards-based teacher evaluation scores and student achievement: Replication and extensions at three sites. Madison: Consortium for Policy Research in Education, University of Wisconsin. Available:http://cpre.wceruw.org/papers/3site_long_TE_SA_AERA04TE.pdf

Rothstein, J. (2008). Student sorting and bias in value added estimation: Selection on observables and unobservables. (Working Paper No. 170). Princeton, NJ: Princeton University and Cambridge, MA: National Bureau of Economic Research.

Source B

(The Washington Post)

The fundamental flaws of ‘value added’ teacher evaluationBY VALERIE STRAUSS December 23, 2012 at 2:00 pm

Evaluating teachers by the test scores of their students has been perhaps the most controversial education reform of the year because while it has been pushed in a majority of states with the support of the Obama administration, assessment experts have warned against the practice for a variety of reasons. Here Jack Jennings, found and former president of the non-profit Center on Education Policy explains the problem. This appeared on Huffington Post.

By Jack Jennings

American tourists are often amused when traveling on the London “tube” to hear the announcement at each station to “mind the gap.” This attention-getting advice is meant to warn passengers exiting the subway car to step over the space between the car and the platform.

American education has its own gap, and it might be helpful if we repeatedly heard public announcements to “mind” it. This gap is the distance between what policymakers are putting in place and what research has found.

The most controversial example of this gap between policy and research relates to the current fight to change the methods used to evaluate teachers’ classroom performance. The Obama Administration, charitable foundations, conservative critics of teacher unions, and various others are encouraging state governments to revise teacher evaluation systems to consider the impact of individual teachers on their students’ achievement. The states are responding. According to the National Council on Teacher Quality, 36 states and the District of Columbia have made policy changes in teacher evaluation since 2009, and 30 states now require these evaluations to include objective evidence of student learning.

The stakes are high because teachers could lose their jobs if they have low ratings on these new evaluations. Their salaries and promotions could also be affected, as well as their standing among their peers.

Proponents of change rightfully argue that current teacher evaluation systems are inadequate. Often, these involve a short “walk in” visit by the principal or another type of cursory review. Clearly, better ways of evaluating teachers must be found.The problem comes with many of the alternatives being proposed or implemented, especially those that rely heavily on tests. In particular, the Obama administration, through its Race to the Top competitive grants and its waivers of No Child Left Behind Act (NCLB) requirements, is putting pressure on states to incorporate student test scores as a significant component of any new teacher evaluation system.To fulfill the conditions for receiving Race to the Top grants or No Child Left Behindwaivers, states are often turning to the reading and math tests used for NCLB accountability. Since those tests are already administered to almost all students in grades 3 through 8 and once in high school, it’s understandable that states would choose to use them for teacher evaluation purposes.

The common sense rationale for linking teacher evaluations to student test scores is to hold teachers accountable for how much their students are learning. The favorite way of measuring gains, or lack thereof, in student learning is through “value-added” models, which seek to determine what each teacher has added to the educational achievement of each of his or her students.

Even though it seemingly makes sense to look at individual gains attributable to particular teachers, this method is fundamentally flawed due to the nature of current state tests, as well as the methods used to assign students to teachers and other reasons. These tests were not designed to be used in that way, and various aspects of their administration make this use improper.

In a briefing paper prepared for the National Academy of Education (NAE) and the American Educational Research Association, Linda Darling-Hammond and three other distinguished authors reached the following conclusion: “With respect to value-added measures of student achievement tied to individual teachers, current research suggests that high-stakes, individual-level decisions, as well as comparisons across highly dissimilar schools or student populations should be avoided.” The paper goes on to saythat “in general, such measures should be used only in a low-stakes fashion when they are part of an integrated analysis of what the teacher is doing and who is being taught.” (Disclaimer: Although I am a member of NAE, I did not research or write that paper.)

The paper highlights three specific problems with using value-added models to evaluate teacher effectiveness, especially for such important decisions as teacher employment or compensation:

1. Value-added models of teacher effectiveness are highly unstable. Teachers’ ratings differ substantially from class to class and from year to year, as well as from one test to another.

2. Teachers’ value-added ratings are significantly affected by differences in the students who are assigned to them, even when models try to control for prior achievement and student demographic variables. In particular, teachers with large numbers of new English learners and others with special needs have been found to show lower gains than the same teachers who are teaching other students.

3. Value-added ratings cannot disentangle the many influences on student progress. These include home, school and student factors that influence student learning gains and that matter more than the individual teacher in explaining changes in test scores.

Cautions about value-added testing have also been expressed by a group of testing and policy experts assembled by the Economic Policy Institute. This group concluded that “[w]hile there are good reasons for concern about the current system of teacher evaluation, there are also good reasons to be concerned about claims that measuring teachers’ effectiveness largely by student test scores will lead to improved student achievement.”

In a similar vein, W. James Popham, professor emeritus at UCLA and test design expert, has concluded that the use of students’ test scores to evaluate teachers “runs counter to the most important commandment of educational testing — the need for sufficient validity evidence.”

Despite this strong advice based on research, states are pushing ahead to incorporate value-added models into their teacher evaluation systems. In Louisiana and Florida, two states that have made sweeping changes, the state legislatures eliminated teacher tenure and instituted systems that rely to a substantial degree on test scores to determine employment and salary. Many other states are not far behind.Why have politicians and others decided to ignore the research and use defective systems to make major decisions about retaining teachers or determining their pay? Why are we not “minding the gap”?

Possibly, proponents of change felt they had to push hard to eliminate a defective system. In addition, some research, including an ongoing study of measures of effective teaching supported by the Gates Foundation, gives credence to the use of student achievement measures when combined with other measures, such as teacher observations and student feedback, as part of an effective teacher evaluation system. It is also possible that the researchers who raise serious concerns about the emphasis being placed on test score measures have not effectively stated or publicized their objections. Regardless of reason, there is trouble ahead.

In Florida an administrative order already temporarily stopped that system. In that state as well as others, groups are planning to litigate as soon as a teacher is fired or teacher salaries are lowered based on the results of a value-added model. Research findings will likely be used to discredit the value-added approach.

The shame of all this is that there is another way. As the National Academy paper points out, some other tools to measure teacher effectiveness are more stable and sophisticated. These include assessments of the actual performance of teachers and on-the-job evaluations, both of which rely on professional judgments.Are states not fully embracing such options because they are more complex and have higher training costs than simply using test scores? If so, they are being very short-sighted.We have ignored the advice to mind the gap for too long. The way we educate our children and treat our teachers should be based on facts, not on impulses.

Source C

(Excerpt from the Michigan Revised School Code Act 451, updated 2009)

380.1249 Performance evaluation system for teachers and school administrators; requirements; governor's council on educator effectiveness; recommendations on evaluation processes; compliance with subsection (2) or (3) not required; effect of collective bargaining agreement; effectiveness label.

Sec. 1249.

(1) Not later than September 1, 2011, and subject to subsection (9), with the involvement of teachers and school administrators, the board of a school district or intermediate school district or board of directors of a public school academy shall adopt and implement for all teachers and school administrators a rigorous, transparent, and fair performance evaluation system that does all of the following:

(a) Evaluates the teacher's or school administrator's job performance at least annually while providing timely and constructive feedback.

(b) Establishes clear approaches to measuring student growth and provides teachers and school administrators with relevant data on student growth.

(c) Evaluates a teacher's or school administrator's job performance, using multiple rating categories that take into account data on student growth as a significant factor. For these purposes, student growth shall be measured by national, state, or local assessments and other objective criteria. If the performance evaluation system implemented by a school district, intermediate school district, or public school academy under this section does not already include the rating of teachers as highly effective, effective, minimally effective, and ineffective, then the school district, intermediate school district, or public school academy shall revise the performance evaluation system within 60 days after the effective date of the amendatory act that added this sentence to ensure that it rates teachers as highly effective, effective, minimally effective, or ineffective.

(d) Uses the evaluations, at a minimum, to inform decisions regarding all of the following:

(i) The effectiveness of teachers and school administrators, ensuring that they are given ample opportunities for improvement.

(ii) Promotion, retention, and development of teachers and school administrators, including providing relevant coaching, instruction support, or professional development.

(iii) Whether to grant tenure or full certification, or both, to teachers and school administrators using rigorous standards and streamlined, transparent, and fair procedures.

(iv) Removing ineffective tenured and untenured teachers and school administrators after they have had ample opportunities to improve, and ensuring that these decisions are made using rigorous standards and streamlined, transparent, and fair procedures.

(2) Beginning with the 2013-2014 school year, the board of a school district or intermediate school district or board of directors of a public school academy shall ensure that the performance evaluation system for teachers meets all of the following:

(a) The performance evaluation system shall include at least an annual year-end evaluation for all teachers. An annual year-end evaluation shall meet all of the following:

(i) For the annual year-end evaluation for the 2013-2014 school year, at least 25% of the annual year-end evaluation shall be based on student growth and assessment data. For the annual year-end evaluation for the 2014-2015 school year, at least 40% of the annual year-end evaluation shall be based on student growth and assessment data. Beginning with the annual year-end evaluation for the 2015-2016 school year, at least 50% of the annual year-end evaluation shall be based on student growth and assessment data. All student growth and assessment data shall be measured using the student growth assessment tool that is required under legislation enacted by the legislature under subsection (6) after review of the recommendations contained in the report of the governor's council on educator effectiveness submitted under subsection (5).

(ii) If there are student growth and assessment data available for a teacher for at least 3 school years, the annual year-end evaluation shall be based on the student growth and assessment data for the most recent 3-consecutive-school-year period. If there are not student growth and assessment data available for a teacher for at least 3 school years, the annual year-end evaluation shall be based on all student growth and assessment data that are available for the teacher.

(iii) The annual year-end evaluation shall include specific performance goals that will assist in improving effectiveness for the next school year and are developed by the school administrator or his or her designee conducting the evaluation, in consultation with the teacher, and any recommended training identified by the school administrator or designee, in consultation with the teacher, that would assist the teacher in meeting these goals. For a teacher described in subdivision (b), the school administrator or designee shall develop, in consultation with the teacher, an individualized development plan that includes these goals and training and is designed to assist the teacher to improve his or her effectiveness.

Source D

(The Washington Post)

How one great teacher was wronged by flawed evaluation system

BY VALERIE STRAUSS September 8, 2013 at 10:30 am

Principal Carol Burris of South Side High School in New York has for some time been chronicling the consequences of standardized test-driven reform in her state (here, andhere and here, for example). Burris was named New York’s 2013 High School Principal of the Year by the School Administrators Association of New York and the National Association of Secondary School Principals, and in 2010,  tapped as the 2010 New York State Outstanding Educator by the School Administrators Association of New York State. She is the co-author of the New York Principals letter of concern regarding the evaluation of teachers by student test scores. It has been signed by more than 1,535 New

York principals and more than 6,500 teachers, parents, professors, administrators and citizens. You can read the letter by clicking here.  

In this new post, Burris tells the story of a New York state teacher who was just unfairly smacked by the state’s flawed new teacher and principal evaluation system, known asAPPR, which in part uses student standardized test scores to evaluate educators. The method isn’t reliable or valid, as Burris shows here.

By Carol Burris

Jenn is a teacher of middle-school students.  Her school is in a small city district that has limited resources.  The majority of kids in the school receive free or reduced priced lunch and about 40% are black or Latino.  Many are English language learners. Lots of them are homeless.

After learning that she was rated less than effective because of her students’ standardized test scores, she wrote to Diane Ravitch,   who posted her letter on her blog. She wrote:

“I’m actually questioning whether I can teach for the next 20 years. It’s all I’ve ever wanted to do, but this APPR garbage is effectively forcing out some of the best teachers I’ve worked with. I may be next.”

I contacted Jenn to better understand her story.  I encountered the kind of teacher I love to hire.  She has never imagined herself as anything but a teacher—teaching is not a stepping stone to a career in law or business.  She does all the extras. She comes in early and leaves late. She coaches. She understands that she must be a counselor, a nurse and a surrogate parent to her students—the most at-risk students in the seventh-grade.  Jenn is their support teacher for English Language Arts.

She is valued by her principal who gave her 58 out of 60 points on the measure of teaching behaviors—instruction, lesson plans, professional obligations, understanding of child development, communication with parents—all of the things that matter and that Jenn can truly control.

And then came the test score measures.  The grade-level teachers and the principal had to create a local measure of student performance.  They chose a group measure based on reading growth on a standardized test.  They were required to set targets from a pre-test given in the winter to a post-test given in the spring.  The targets were a guess on the part of the teachers and principal.  How could they not be? The team was shooting in the dark—making predictions without any long-term data.  Such measures can never be reliable or valid.

The state of Massachusetts requires that measures of student learning be piloted and that teachers be evaluated not by one set of scores, but rather by trends over time.  That state’s evaluation model will not be fully implemented for several years because they are building it using phase-in and revision. But New York does not believe in research or caution.  New York is the state where the powerful insist that teachers “perform,” as though they were trained circus seals.  There is no time for a pilot in the Empire State.  Our students and we must jump, as our chancellor advises, “into the deep end of the pool.” In New York, our commissioner warns that we can never let the perfect be the enemy of the good.  We don’t even let nonsense be the enemy of the good.  And so Jenn hoped that she and her colleagues made a reasonable gamble when they set those targets.

Many of the seventh-grade students did not take the standardized reading test seriously. Middle schoolers are savvy—they knew the test didn’t count. So they quickly filled in the bubbles as teachers watched in horror.  Luckily, enough students took their time so that their teachers were able to get 10/20 points on that local measure of learning, which put Jenn in the Effective range.

The final piece in her evaluation was her score from new Common Core-aligned tests that   the state gave to students this past spring. The tests were far too difficult for Jenn’s Academic Intervention Services (AIS) students.  They were too long.  The reading passages were dense and many of the questions were confusing. We know that only about 1 in 5 students across the state, who are like the majority of Jenn’s students, scored proficient on the Common Core tests.  Even more importantly, we know that about half

of all students like Jenn’s scored in level 1—below Basic. These are the students who, overwhelmed by frustration, give up or guess. The test did not measure their learning—it measured noise.

So Jenn’s students’ scores, along with all the other seventh-grade scores on the Common Core tests, were put in a regression model and the statisticians cranked the model, and they entered their covariates and set confidence levels and scratched their heads over Beta reports and did all kinds of things that make most folks’ eyes glaze over.  And all of that cranking and computing finally spit out all of the teachers’ and principals’ places on the modified bell curve.  Jenn got 5 points out of 20 on her state growth score along with the label, Developing.

When all of the points were added up, it did not matter that she received 58/60 points in the most important category of all, which is based on New York’s teaching standards. And it did not matter that she was Effective in the local measure of student learning. 5+10+58 = 73 which meant that Jenn was two points short of being an Effective teacher. Jenn was labeled, Developing, and put on a mandated improvement plan.

This seven-year dedicated teacher feels humiliated.  She knows that parents will know and possibly lose confidence in her. She is angry because the label is unfair. She will be under scrutiny for a year. Time she would spend on her students and her lessons will be wasted in meetings and improvement plan measurement. The joy of teaching is gone. It has been replaced by discouragement and fear.

Her principal also knows it is not fair—she gave Jenn 58/60 points.  Over time, however, she may begin to doubt her own judgment—the scores may influence how she rates teachers. After all, principals get a growth score too, and the teachers with low scores will become a threat to principals’ own job security over time.  Those who created this system put Machiavelli to shame.

Jenn is not alone.  There are hundreds, if not thousands, of good teachers and principals across the state who are receiving poor ratings they do not deserve based on a flawed model and flawed tests.  Slowly, stories will come out as they gain the courage to speak out.  There will be others who suffer in silence, and still others who leave the profession in disgust.  None of this is good for children.

During the July 2013 hearing of the Governor’s New Education Reform Commission, David Steiner, the previous New York State Commissioner of Education, said,

“There is a risk, and I want to be honest about this, that very, very, mature, effective teachers are saying you are treating me like a kid.  In the name of getting rid of the bottom 5 percent, we risk losing the top 5 percent….We do not want to infantilize the profession in order to save it.” Steiner directed those remarks to his former deputy, now state commissioner, John. B. King.  Did King understand what his former mentor was trying to tell him?  Because he did not respond to Steiner’s observation, we do not know.

John King told districts to use caution when using this year’s scores to evaluate teachers and principals. He claimed that the tests did not negatively impact teacher’s accountability ratings. Perhaps he should ask Jenn if she agrees. We already know that 1 in 4 teachers and principals moved down at least one growth score category from last year — hardly the hallmark of a reliable system.

There is much that King and the Board of Regents can do. They can ask the governor to pass legislation so that the evaluations remain private.  They can request that teachers like Jenn, who are more than effective in the eyes of their principals, be spared an improvement plan this year.  I hold no hope, however, that John King will do that. He lives in “the fierce urgency of now.”  But for Jenn and her students, now quickly becomes tomorrow. The risk that David Steiner explained is real.  We need to make sure that we have our best teachers tomorrow and not lose them in the deep end of the pool.

Source E

(The New York Times)

AUGUST 27, 2009, 1:01 PM

SAT Scores and Family IncomeBy CATHERINE RAMPELL

Much has been written about the relationship between SAT scores and test-takers’ family income. Generally speaking, the wealthier a student’s family is, the higher the SAT score.

Let’s take a look at how income correlated with scores this year. About two-thirds of test-takers voluntarily report their family incomes when they sit down to take the SAT. Using this information, the College Board breaks down the average scores for 10 income groups, each in a $20,000 range.

First, here are the individual test sections:

Source: College Board

Source: College Board

Source: College Board

Here are all three test sections next to each other (zoomed in on the vertical axis, so you can see the variation among income groups a little more clearly):

Source: College Board

A few observations:

There’s a very strong positive correlation between income and test scores. (For the math geeks out there, the R2 for each testaverage/income range chart is about 0.95.)

On every test section, moving up an income category was associated with an average score boost of over 12 points.

Moving from the second-highest income group and the highest income group seemed to show the biggest score boost. However, keep in mind the top income category is uncapped, so it includes a much broader spectrum of families by wealth.

Source F