WILL Blog | WILL Responds to Criticism from Colorado Group
Not surprisingly, a University of Colorado group that is universally critical of studies with positive results for school choice has found fault with a recent WILL study, “Bang for the Buck.” The study, which we co-authored, analyzed the public schools in Milwaukee that produced the best student outcomes per tax dollars spent. After controlling for variables such as students’ socio-economic status, we found that independent public charter schools (and non-instrumentalities) are much more efficient with taxpayer money.
In response, Dr. Casey Cobb of the National Education Policy Center in Colorado wrote a critique of the study, which has been touted by opponents of school choice, like WEAC.
Unfortunately, Dr. Cobb’s critique reflects a serious lack of understanding of Wisconsin charter schools and the data used in our analysis (which is publically available from DPI). We could have explained this to him but he never contacted us.
Below we address each of Dr. Cobb’s critiques:
- “Test scores do not comprehensively represent the purposes of schools.”
We agree that test scores are an incomplete measure of a school’s purpose. Instead of test scores, Dr. Cobb suggests that we should have looked at other variables, such as student success in extracurricular activities and graduation rates. First, it is very hard, if not impossible, to quantify “extracurricular” activities. Second, even if such an analysis could be done, that does not invalidate the use of the Badger Exam or WKCE as a measure of school achievement. A point that Dr. Cobb, ironically, makes by citing research that uses test scores to measure achievement.
- “The report does not address threats to the validity of its assumption that there is uniform financial accounting across schools and types.”
Dr. Cobb gives no actual evidence to support his claim that charter schools share services with Milwaukee Public Schools.
In reality, non-instrumentality charter schools in Milwaukee are forced to pay MPS for most of the services that Dr. Cobb speculates might be shared. According to Sean Roberts of Milwaukee Charter School Advocates, “special education, food service, transportation and other district services aren’t free for non-instrumentality schools. They pay MPS for them—on top of the per-pupil “skim.” The skim – not referenced by Dr. Cobb – is the roughly $2,000 per student that MPS receives for students attending non-instrumentality charter schools. It is the difference between what MPS pays the charter schools and what MPS receives from the state.
Says Roberts, “in some cases, non-instrumentality charters may benefit from the broader purchasing power of its authorizing district, but those savings are often offset by the administrative and overhead costs from the district—sometimes upward of 50%.”
- “The analytic description of the study is incomplete, making interpretation difficult.”
If Dr. Cobb had trouble understanding our study, we would have been happy to answer any questions and walk him through the report. But he never contacted us, which is the standard practice among academics making an honest attempt to analyze a report.
Perhaps due to his, admitted, issues interpreting our study, Dr. Cobb cites to research that critiques multiple regression alone as a means of assessing efficiency. However, this is not the method utilized in the paper (a two-stage analytic technique where the coefficients from multiple regression analysis are converted to efficiency scores which are then compared in a difference-of-means test). Dr. Cobb suggests the use of a stochastic frontier model rather than the method we used. However, due to the intricacies of school finance, this is a statistical impossibility that would introduce perfect collinearity into the model. Even these models have come under increasing scrutiny for improperly modeling the skewness of the data. The bottom line is that there are inherent benefits and drawbacks to every modeling technique. We believe we have chosen the best method given the available data and modeling assumptions.
- ‘”Autonomy” is never really defined—it is just used as a loose term implying independence—so autonomous behavior is assumed by virtue of their charter status. The report then makes strong but unmeasured claims about the superior “efficiency” of charter schools based on their having this greater autonomy.’
Given that autonomy is defined on the first page of the Executive Summary (and pg 6 of the full report), this is simply not true. We define autonomy on a number of dimensions including the charter school authorizer, who controls the hiring of staff, and the extent to which curriculum is determined by the school district. It is hard to dispute– though Dr. Cobb tries – that an instrumentality charter school that is controlled by Milwaukee Public Schools (MPS) has less autonomy than an independent charter school, which has no affiliation with MPS.
Based on this greater level of autonomy, we hypothesize that these schools will be more efficient. The remainder of the paper is focused on measuring efficiency differences. Therefore, it is wrong to say that these are “unmeasured claims.”
- “While the report’s analysis controls for some school demographic characteristics, it does not appear to adjust for selection effects; effects that could prove fatal to their conclusions.”
Selection bias was accounted for as extensively as possible – given the lack of an appropriate instrumental variable – through the inclusion of a number of control variables that are related to parental involvement and student performance. These variables include the percentage of students receiving free-and-reduced lunch, the demographic makeup of the student body of the school, and the percentage of students in the school who are English-language learners.
The National Education Policy Center has an extensive track record of finding fault with any and all research in support of the idea that poor and minority children in Milwaukee, and elsewhere, deserve the opportunity to choose their own school. Given the limited utility of Dr. Cobb’s suggestions, it appears that this critique is simply designed to undermine a study that found support for school choice.
Dr. Will Flanders, Education Research Director
CJ Szafir, Vice President of Policy