Blog reader Jake da Snake, a professional educator, checks in today with commentary on advertising in today's Democrat-Gazette from the eStem charter school. The ad includes three graphs purporting to show sharp gains in scores on standardized tests by eStem students in just three months of enrollment there.
Perhaps so. I'll reiterate that stats we compiled show that eStem opened with a student body disproportionately composed (compared with the surrounding LR school district) of students already demonstrating successful academic performance. They are continuing to do so. Good for them and the school, if not unanticipated.
But that's not Jake's concern. His concern is with presentation of the numbers, in what he indicates is a statistically dishonest fashion. Read on. You won't find a similar analysis of these numbers in the D-G, a leading advocate of eStem financially and editorially.
eStem 2008 to 2009 Annual Report
The NWEA test used at eStem is used by over 2,200 school districts in the testing of over 3 million students. It is recognized as a reliable and valid measure of academic growth. NEA and other educational organizations tout the fact that NWEA believes that testing should take place more than once a year in order to provide a more accurate picture of progress.
Problems with eStem advertisement:
The primary problem with eStem’s advertisement is the use of bad graphs which exacerbate errors of scale. This is called a Zoom Graph.
The Zoom Graph is a graph that does not start at (0,0) thus making small changes seem big. Useful in the study of trends, but often passed off as actual changes. Such graphs are very popular in financial circles (and educational circles, as well).
By reducing the range of starting and ending scale numbers on the graph, one can make the graph appear much steeper. The actual percentages of improvement are as follows: (1) Elementary – 4.1%; (2) Middle – 1.2%; (3) High – 1.1%. However, looking at the graphs, it appears as if one is climbing Mt. Everest as one goes up these few percentage points.
The graphs are distortions.
* Graphs should not provide a distorted picture of the values they portray.
* Distortion can be either deliberate or accidental.
* (Of course, it can be useful to know how to produce a graph which bends the truth.)
Ed Tufte of Yale University has defined a “lie factor” as a measure of the amount of distortion in a graph.
* The lie factor is defined to be:
Lie Factor = size of effect shown in graphic divided by the size of effect shown in data
* If the lie factor of a graph is greater than 1, the graph is exaggerating the size of the effect.
The Lie Factor for these graphs is several times greater than 1.
The secondary problem is minor but is one that goes against NWEA recommendations. All graphs use a dotted line to represent “Expected Growth.” NWEA argues that this is not proper terminology and that one should use “Typical Growth.”
Finally, no information is given as to how these scores compare to other schools using the NWEA.