The National Education Association issued a preliminary response Monday, Jan. 22, to a review of the beginning of a merit pay experiment for teachers in Little Rock elementary schools.
The NEA response, reprinted verbatim below, questions the impartiality of a study financed by the same organization financing the pay experiment. It also notes the limited scope of the review, the absence of some underlying data on which the University of Arkansas's conclusions were based and suggested that a press release on the report hyped the more limited findings of the researchers themselves.
The NEA response:
Synopsis and Review
“Evaluation of Year One of the Achievement Challenge Pilot Project in the Little Rock Public School District” by Joshua H. Barnett, Gary W. Ritter, Marcus A. Winters, and Jay P. Greene
This is the first installment of a multi-year study of elementary schools in the Little Rock School District that are participating in the Achievement Challenge Pilot Project (ACPP), a merit pay program. ACPP pays classroom teachers and school employees cash bonuses based on gains made by students on norm-referenced test scores for math, reading and language. The project started with funding from the Public Education Foundation of Little Rock (the Walton Family Foundation is among its donors), which continues as its major source of support. According to sources at the Arkansas Education Association, this study has the same financial backer, which raises the concern of how credible it is for the same entity that underwrites the merit pay program to be responsible for evaluating its effectiveness. Surely, an evaluation of the program could benefit from an independent analysis.
The ACPP program currently operates in five of Little Rock’s 34 elementary schools, with three of the schools beginning in 2006-07. The five schools all have high percentages of minority students and students who qualify for free and reduced lunches, as well as low academic achievement. This study encompasses two ACPP schools, one which started in 2004-05 and one that started in 2005-06. Three “comparison schools” were also included in the study.
The authors undertook two analyses:
1) a statistical analysis (fixed effects model) of changes in norm-referenced test scores for fourth and fifth graders. Due to “data limitations”, only math scores were used and only one of the two ACPP schools (along with all three comparison schools) was included in the study.
2) A survey of teachers, asking them about behaviors and attitudes related that the authors considered related to merit pay. Teachers in the two ACPP schools and three comparison schools were surveyed.
The Study’s Findings
The authors contend that:
1. Their statistical model (a “fixed-effects” model) estimated that “schools where the ACPP operated in 2005-06 showed an improvement of 3.5 normal curve equivalent points. For the average student, this gain represents an improvement of nearly 7 percentile points.”
2. The comparison of teacher surveys in the ACPP vs. non-ACPP schools suggested that that they:
- were no more innovative than comparison teachers.
- were no more likely to work harder than comparison teachers.
- were more satisfied with their salaries than comparison teachers.
- reported no more counterproductive competition than comparison teachers.
- their work environment became more positive than comparison teachers.
- were less likely than comparison teachers to agree that low-performing students were a “burden” in the classroom.
- were more likely than comparison students to report that the academic performance of their students had improved over the past year.
Overall, the authors concluded that “while the results from this first year study suggest positive impacts of the ACPP, we believe the second year study with five schools involved in the ACPP will greatly assist in expanding on and explaining the first year findings.”
While headline of the press release announcing the study boldly states that says the “Report Finds Merit Pay Has Positive Effects,” upon closer examination there are several reasons to question whether the reported findings live up to the hype. Overall, it is important to keep in mind that these are preliminary findings based on incomplete data from the first year of a multi-year study. At best, it is premature to be calling ACPP a success in achieving any of its stated goals. More probably, it is irresponsible to make any claims about the effectiveness of ACPP on the basis of this study.
1. Modest Effects
One of the authors described the findings as “modest” and while encouraging to those who carried out the study, they “certainly wouldn't say we're thrilled by any stretch.” (Associated Press, 1/16/07)
2. Narrow Scope
The study is very limited in its scope. Due to “data limitations,” only math scores were included in the analysis (p.6). Reading and language test scores were omitted. Further, “due to the limits of available data,” the estimates are based on students in only one of the two ACPP schools. (p.7). Thus, “even though nearly 2,000 students attend the schools involved in the evaluation, approximately 500 students were part of the student achievement evaluation.” (p.12) These data limitations alone make the results highly questionable.
It is difficult for other trained statisticians to further evaluate the results of the study of the impact of ACPP on test scores because the specifications, equations, coefficients, and significance tests are not shown in the report and are only partially and briefly discussed.
It is also worth noting that there is no discussion of each school’s status in school improvement plans under No Child Left Behind (NCLB) guidelines, which could be a very significant factor in affecting the behavior and attitudes of teachers in each of the schools. While the percentages of students rated as “proficient” in math and literacy tests were considered as criteria in selecting the comparison schools, the authors did not make clear the distinction between norm-reference tests (which were used in their analysis) and criteria-referenced tests that are used as benchmarks for determining school improvement status. Thus, according to sources at the Arkansas Education Association, a paradoxical situation exists where teachers in ACDD schools earned bonuses based on norm-referenced tests while their same schools were put on improvement plans based on the results of criteria-related tests.
3. Lack of causality
The comparison of survey data of teachers in schools with and without ACPP does not take into account any other factors that might explain differences between the two groups. While the authors conduct simple significance tests on averages for survey answers for schools in and out of the ACPP program, this alone is not sufficient to infer that these differences can be attributed to ACPP.
For example, the authors also imply that greater “satisfaction with salaries” can be attributed to the presence of merit pay while it may actually just be an effect of higher pay. It is not possible for teachers in ACPP schools to have pay deducted because of low test scores. It is only possible to receive a bonus on top of regular pay for improved test scores. Though the actual percentage of teachers receiving bonuses was not reported, it was reported that over was $200,000 in bonuses was divvied up in one ACPP school and nearly $230,000 was awarded in the other school in 2005-06. No additional pay was awarded in any of the comparison schools. Thus, any differences in pay satisfaction might be more attributable to higher pay than any incentive effect.
Setting aside this overall lack of causality, it is interesting to note that while the authors hypothesize that merit pay may “motivate current teachers to improve the performance of students through additional effort and innovation, where teachers work to learn and implement new effective teaching strategies,” their survey found no difference between the two groups of teachers in how “innovative” they were or in reporting that they worked “harder.” The authors do not discuss this apparent lack of support for key behavioral theories about why merit pay would lead to higher student achievement in the first place.