Sunday, September 23, 2012

The Problem with Using Test-Scores as Performance Assessment Metrics

I've been hearing a lot about Education Reform plans lately, most recently with the Chicago strike in which teachers are objecting to using test scores as a performance metric. 

In general, the conservative view seems to feel that it is fair and proper to use test scores to award teacher compensation. I am not expert enough to say what the liberal view is, though they increasingly seem to be blaming "the schools" (see Waiting for Superman).

I think that using test scores as performance assessments for public school teachers is myopic, unless highly effective risk adjustment measures are employed i.e. ones that acknowledge reality of differenees in student starting points.

 To explain why, I thought I would examine my old Middle School (grades 6-8), which now appears to only serve grades 7 and 8.  When I was there, the school clearly had problems and the achievement gap was a source of constant debate. A glance at its performance report suggests these problems have only intensified. The test score data is available here, stratified by gender, ethnicity, grade, and free/reduced lunch status: although some of the data seems to not be displaying correctly.

The site is tough to navigate so I won't get into the specifics: most of what you need to know can be found in the first couple pages of this school improvement plan. From the plan:

- "There has been an overall decline in the performance of all students in 
Grade 8 with a variance of 5% points from 2005 to 2008." 
- "There is a 25% point gap in problem solving which is a decline from past years" (i.e. mathematics scores have decreased by that amount (I think?))

However, it is important to note that the mild decline in overall mathematics scores (and stagnant Language Arts scores) mask the deep stratification gender and economic criteria. The racial/economic achievement gap is striking, with the major change being declining White student scores: 

For Language Arts

"However, there is still a 26% point gap compared to White students over the course of five years in LA's. In the areas of reading comprehension and writing process, the achievement gap has dropped from 34-38% to 22% in the past five years, only because White student achievement has dropped 
10% over that time period.   No systematic improvement is evident."
-  "While the achievement gap of Hispanic students as compared to White students 
has declined between 8-10% points, it is mostly due to the decline in 
White student achievement levels."
-"The gap between Free & Reduced Lunch students compared to Paid 
Lunch students shows no reduction and is around 25% - 30% points in 
all three areas of Language Arts.  Only 51-54% of Free & Reduced 
Lunch students master the three LA assessments compared to 77-79% 
for Paid Lunch students."

For Mathematics, the story is basically the same:

"The achievement gap in mathematics continues to be around 20-27% 
points for Black students compared to White students. Our Hispanic and 
Free/Reduced lunch subgroups have approximately the same percentage 
point achievement gap for the years 2005-2008."  

------

Clearly, things are regressing rather than improving for Fall Creek Valley Middle School, although the decline is fairly modest in overall score averages. Most strikingly, the achievement gap shows no signs of improving. 

How do we improve this, then? Can teachers be incented to improve these scores by tying compensation to performance? Personally, I think this a highly myopic response to a broader social movement. The true source of Fall Creek's test performance decline is the rapid demographic changes Lawrence is facing, a classic white flight movement. From the report:
 
"There has been a dramatic change in demographics at our school over the past 
12 years. . .White students have declined as a percent of total school population from 76.9% 
in 1996 to 41.4% in 2008.  Black student population has increased as a 
percentage from 19.4% to 44.4% during the same period.  Multi-racial student 
population has increased from 0.3% to 6.7% during the same period and 
Hispanic students have increased from 1.3% to 6.4% during the same period. 
Free and Reduced Lunch students have increased from 14.3% in 1996-97 to 
42.3% in 2007-08 The percentage of free and reduced lunch students had 
increased to over 48% during the 2008-09 school year [Lydgate -- i.e. three times the 1997 free/reduced lunch percent]"

Thus, Fall Creek Valley teachers are fighting an up-hill demographic value as the percentage of high-risk minority/low-income students in their class increase every year, resulting in the decline in overall achievement rates and a persistant achievement gap. 

In fact, the focus on changes in school rate is completely misguided. Lawrence is simply housing a higher percentage of poor performing Indiana students, while more high performing students are shifting to outer metro area schools. I am skeptical that any real overall changes are being observed, itself hard to quantify likely due to changes to the test. 

Indeed, you cannot compare the task Fall Creek teachers face to their counter-parts in the nearby suburbs of Hamilton County: their job might as have a completely different title. Where market rewards for performance may produce some quality in a free-market business environment (a debate for another post, perhaps), public services are different in that they cannot turn away high-risk students as a private school might. While teacher benefits and unions may be flawed, performance assessment is not a reasonable answer unless extremely effective risk-adjustment techniques are employed, something I am highly doubtful will quantify the true separation in starting-point between a suburban and urban/urbanizing public school environment. 

Education reformers are looking for a magic bullet that solves student performance and achieves public sector savings (Superman, if you will). In this regard, they will only be continually disappointed.

- Lydgate 

No comments:

Post a Comment