全球智库信息集成服务系统

This report contains corrections.

How bad are school vouchers for students? Far worse than most people imagine. Indeed, according to the analysis conducted by the authors of this report, the use of school vouchers—which provide families with public dollars to spend on private schools—is equivalent to missing out on more than one-third of a year of classroom learning. In other words, this analysis found that the overall effect of the D.C. voucher program on students is the same as missing 68 days of school.

This analysis builds on a large body of voucher program evaluations in Louisiana, Indiana, Ohio, and Washington, D.C., all of which show that students attending participating private schools perform significantly worse than their peers in public schools—especially in math.¹ A recent, rigorous evaluation of the D.C. Opportunity Scholarship Program from the U.S. Department of Education reaffirms these findings, reporting that D.C. students attending voucher schools performed significantly worse than they would have in their original public school.²

The analysis is timely given President Donald Trump and Education Secretary Betsy DeVos’ main education priority: to privatize education by creating and expanding voucher programs nationwide. In the Trump budget released in February, the president has suggested doubling investment in vouchers.³* But while President Trump and Secretary DeVos often assert that research backs their proposals, the evidence is lacking.

In order to add necessary context to the recent voucher research—and the debate over the budget—the authors compare the negative outcomes of one of these voucher programs—the D.C. Opportunity Scholarship Program—to other factors that negatively affect student achievement. That analysis also finds that the effect of vouchers on student achievement is larger than the following in-school factors: exposure to violent crime at school, feeling unsafe in school, high teacher turnover, and teacher absenteeism.⁴

To be clear, the far-reaching negative effects of factors such as feeling unsafe in school cannot be overstated. For example, there is a large body of work that discusses the negative impact of exposure to violent crime on children’s well-being, including academic performance.⁵ Certainly, many of these factors are serious and are known to have a negative impact on multiple areas of child development. However, the comparisons made in this report focus only on how each in-school factor—violence at school, feeling unsafe, teacher turnover, and teacher absenteeism—affects school achievement.

Further, using the Center for Research on Education Outcomes (CREDO) formula, the authors of this report also found that the overall effect of the D.C. voucher program on students is the equivalent of 68 fewer days of schooling than they otherwise would have received had they remained in their traditional public school. In other words, the students who participated in the D.C. voucher program lost more than one-third of a year of learning.⁶ To be clear, translating this effect into days of learning is an approximation intended to help assess relative impact. In this case, 68 days lost is clearly substantial lost ground for students participating in the D.C. voucher program.

Importantly, the authors’ comparisons rely on both a selection of rigorous, recent voucher studies with test score impacts and one rigorous quasi-experimental study for each comparison topic. That being said, using alternative studies could lead to different results. The methods for selecting both specific voucher studies and comparison studies to add context are detailed in this report’s Appendix. Finally, the evaluation of the D.C. voucher program so far assesses one year of achievement data and future findings may be different. However, these findings are very similar to those from other longer-term voucher studies, which are described in this report.

Caveats aside, the negative achievement outcomes of voucher programs uncovered by recent studies have not deterred the Trump administration;⁷ its stance on vouchers exemplifies a larger tendency to a dismiss evidence that does not align with the administration’s priorities.⁸ This dismissal of evidence plays an unfortunate role in both undermining the nation’s public schools and confusing parents who seek access to high-quality public school options.

Earlier studies of voucher programs appeared to show some promise. Many of those evaluations—in New York City, Dayton, Ohio, and Washington, D.C., as well as in the states of Florida, Minnesota, and Louisiana—reported a modest increase or neutral impact on student achievement and graduation rates.⁹ The findings of some of these studies, however, have more recently been called into question as methodological flaws were discovered when adding additional years or replicating the study.¹⁰ As a result, recent voucher program evaluations employ more rigorous research methods such as experimental and quasi-experimental designs and refine their use of certain variables. These design changes and data additions allow for more stable and definitive interpretations of the results.

More recent evaluations of voucher programs in three states—Indiana, Louisiana, and Ohio—have all come to similar conclusions and show that voucher programs have negative or neutral effects on student achievement.¹¹ Importantly, all impacts described in this report are relative to public schools. Unlike other experimental designs where participants may receive a placebo or the intended intervention, the intervention in these cases is a private school voucher and the comparison is a traditional public school. Therefore, on the whole, these results show that relative to their peers in public schools, students in voucher programs are losing ground.

The latest evaluation examined the outcomes of students using vouchers in Indiana for two, three, or four years between the 2011-12 and 2014-15 school years. The Indiana study has the largest sample size—and the largest voucher program—across all studies examined in this report. More than 34,000 Indiana students received vouchers in the 2016-17 school year.¹² The study used a matching methodology to compare the test scores of students who transferred to participating voucher schools with similar students who remained in public schools. It found that students who used vouchers did not see academic gains in their new schools and that they performed worse, on average, than their matched peers in the public schools that they left.

Notably, the study also found variation in voucher impacts depending on the sample of students investigated and how long those students stayed in the program—those who stayed longer experienced fewer negative impacts than those who only stayed for two years. However, in English language arts, students with disabilities in voucher programs experience an average learning loss each year when compared with students without special education identification.¹³

The most recent evaluation of the Louisiana voucher program, from the 2011-12 to 2013-14 school years, also showed a negative impact.¹⁴ The Louisiana program expanded statewide in 2012 after serving only students in New Orleans since 2008.¹⁵ In the 2015-16 school year, the program served more than 7,000 students.¹⁶ Using an experimental design, the evaluation found that, on average, participating voucher students’ performance in both reading and math dropped for two years and then, according to one model with a restricted sample size, saw performance rebound to end up remaining somewhat level to public school students’ performance in math after three years.¹⁷ However, the using an expanded sample size, the second model in the same paper found that student performance after three years continued to suffer from negative impacts. Ultimately, the model showing continued negative performance is more relevant and accurate, as it is based on a substantially larger sample size. Other researchers agree and assert that the continued negative findings across all three years are the most accurate results to highlight.¹⁸

The Louisiana findings are significant and show declines that are the equivalent of the average math student—at the 50th percentile—dropping to the 34th percentile after three years of participation in the Louisiana voucher program. The researchers of this evaluation find that this large effect is driven by even larger significant, negative effects for students who started in earlier grades—first through third grades—and somewhat smaller nonsignificant, negative effects for students who started in later grades: fourth through sixth.¹⁹

Researchers have studied vouchers in Ohio as well. Also a statewide program, Ohio gave vouchers to more than 18,000 students in the 2013-14 school year.²⁰ The state has since expanded the program to grant vouchers to up to 60,000 qualifying students.²¹ Unlike other voucher programs, the Ohio program is targeted to only students attending low-performing public schools. The findings in Ohio, which used a quasi-experimental design and compared students who are similar across a multitude of factors, are similarly negative for students in math and reading after attending a voucher school. This study is the only one out of the four voucher programs recently evaluated that finds significant negative impacts in both math and reading.²²

In addition to these four state-based studies of voucher program impacts on test scores, some recent studies do show positive effects on graduation rates, parent satisfaction, community college enrollment, and other nonachievement-based outcomes, but it is unclear if these outcomes are lasting and valid.²³ For example, research shows that nationally, graduation rates for students in public schools and peers participating in voucher programs equalize after adjusting for extended graduation rates.²⁴ Some critics suggest that private schools may graduate students who have not successfully completed the full program.²⁵ Also, in regard to parent satisfaction, while some studies do show greater satisfaction among parents whose children participate in voucher programs, the most recent evaluation of the D.C. voucher program shows that any increase in parent or student school satisfaction is not statistically significant.²⁶

Finally, there is some evidence that the performance of students in public schools increased after the creation of voucher programs—as was the case in Florida and Ohio during the implementation of voucher programs in those states.²⁷ However, it is unclear why these modest increases occurred.²⁸ Nonetheless, these recent rigorous voucher evaluations showing substantive academic losses for students who do participate in voucher programs serve as a heavy counterpoint to these modest results for students remaining in public schools. 

This paper focuses on the most recent evaluation of the D.C. Opportunity Scholarship Program, which is the only voucher program funded and authorized by federal law and uses a randomized control trial design.²⁹

Enacted in 2004, the D.C. voucher program provides publicly funded vouchers for low-income families who have a school-age child and reside in Washington, D.C. Eligible families receive vouchers through a lottery process. Families can enter the lottery if they meet program eligibility—D.C. residency and a household income at or below 185 percent of the federal poverty line. The district gives priority based on a few factors, including attendance in a low-performing public school and siblings in the program. If a student is selected, the family can apply and enroll in private school or decline and remain in a public school.³⁰ About 70 percent of families who are offered a voucher choose to enroll in a private school.³¹ The D.C. voucher program accepts about 600 students per year.³² Once you receive a voucher through the lottery, you are eligible to continue with the program until graduating high school.

Vouchers grant up to $8,653 for elementary and middle school annual tuition, and up to $12,981 for high school tuition.³³ If the tuition or other fees exceeds the voucher amount, parents must cover the difference.³⁴

In the spring of 2017, the Institute of Education Sciences (IES), the independent research arm of the Education Department, published a study that found that D.C. students who used a voucher scored 0.12 standard deviations lower in math than students who were not offered a voucher and remained in a public school.³⁵ The evaluation assessed the outcomes of students from the 2012, 2013, and 2014 lotteries. This 0.12 score drop is the equivalent of an average student in the 50th percentile dropping to the 45th percentile after participating in the D.C. voucher program for one year.³⁶ According to the IES study, participating in the D.C. voucher program had no statistically significant effect on reading achievement.³⁷

These results mean that in math,³⁸ the 1,166 participating voucher students fell behind their peers who remained in public schools, despite an annual federal investment of $60 million in the D.C. voucher program. Both groups of students performed similarly in reading.³⁹ Notably, this study only analyzes one year of impact data; however, the results are in sync with the recent longer-term findings discussed above in Louisiana, Indiana, and Ohio.

IES’s study of the D.C. voucher program is an important addition to existing voucher research. While there are voucher programs available in 13 other states,⁴⁰ the D.C. Opportunity Scholarship Program is the only voucher program with a congressionally mandated evaluation.⁴¹ IES worked in partnership with Washington, D.C., to structure the voucher program as a randomized controlled trial so that researchers could use the most rigorous scientific research method—an experimental design—to evaluate outcomes.⁴² Researchers call experimental designs the gold standard because they leverage random, lottery-based assignment of students to either the treatment or control group—voucher schools or public schools, respectively—to more accurately assess the difference between the two groups of students.⁴³ While these types of studies can be more expensive and cumbersome to implement, they largely provide methodologically unassailable results.

A week after IES published the evaluation, which reported the negative effects of the D.C. voucher program, the Republican-led Congress passed a bipartisan budget deal with a prohibition to the use of the experimental design evaluation method in any future federally funded studies of the D.C. voucher program.⁴⁴ Instead, IES must use “an acceptable quasi-experimental research design.”⁴⁵

This change does not affect the ongoing D.C. voucher evaluations that are already in progress—namely, to explore the long-term effects of D.C. vouchers, including graduation rates. All future evaluations of the federally funded D.C. voucher program will therefore employ methodologies with more limitations, making it more difficult to definitively interpret results. As a consequence, the findings of the most recent evaluation of the D.C. voucher program using the experimental method is now even more important to consider in the ongoing voucher program debate.

The authors of all four recent voucher studies—in Washington, D.C., Louisiana, Indiana, and Ohio—have suggested some theories for the decline in student achievement, such as capacity and loss of instructional time.

The researchers in the district tested three theories about the negative impacts and found that only one of them may explain some of the learning loss experienced in voucher schools.⁴⁶ More specifically, they investigated the possibilities that control group students were in higher-performing public schools; that voucher schools continue to offer less instructional time than public schools; and finally, that student mobility—the fact that a student is moving and adjusting to a new school—may be the underlying cause of the negative impact.⁴⁷

Of these three, only the instructional time factor proved to be a likely cause. As in the 2010 evaluation, the researchers found that private schools offer less instructional time than public schools. On average, private schools offer 65.5 minutes less per week in reading instruction and 48.3 minutes less per week in math instruction.⁴⁸ More quality instructional time is linked to higher student achievement.⁴⁹ Therefore, with each additional year that students are enrolled in the voucher program, they lose even more instructional time.

In Indiana, Louisiana, and Ohio, researchers speculated that participating private schools may lack the immediate capacity and resources to educate students who are academically behind, who are English-language learners, or who have disabilities.⁵⁰ This potential lack of capacity is of particular concern for the participating populations in all four contexts, because the students who tend to use vouchers are more likely to be behind academically. In Indiana, they are also more likely to be English-language learners.⁵¹

Finally, many public schools who lost students to vouchers are on an upward performance trajectory.⁵² Therefore, part of the learning loss experienced by voucher students compared with the learning of their public school peers could be attributable to the fact that those public schools are on an upward performance trajectory. This is consistent with a recent study that found, on average, that over the past two decades, public schools’ performance has caught up to or exceeded the performance of private schools.⁵³

To put the negative effects of vouchers in perspective, the authors of this report compared the effect of the D.C. voucher program on student achievement in math with the effects of other negative in-school factors—feeling unsafe in school, violent crime at school, teacher turnover, and teacher absenteeism.⁵⁴ Beyond school, the authors even found that vouchers have a greater negative impact on student achievement than smoking, according to one study. While this context is compelling, comparing and understanding the impacts of smoking are beyond the scope of this analysis.⁵⁵

As part of the study, the authors show each negative impact as the equivalent of a specific drop for a student who was average in math—in the 50th percentile—to a lower rank in the same distribution. This conversion uses the effect size—in the case of the D.C. voucher program, -0.12 standard deviations—to find the equivalent percentile drop from the 50th percentile on a normal curve.⁵⁶

The comparable results are meaningful and startling. The impact of attending a participating D.C. voucher school on math achievement is a larger decrease than all other factors that the authors reviewed. A public school student who is average in math—in the 50th percentile—declines to the 45th percentile after participating in the D.C. voucher program for one year.

An average math student who experiences high teacher turnover drops from the 50th percentile to the 48th. Feeling unsafe in school, experiencing at least 10 teacher absences, and exposure to school crime have the same equivalent effect, dropping to the 49th percentile.

A decline in percentile points means that students are substantially falling behind relative to their performance in public school or relative to their similar public school peers, depending on the study. All the losses associated with these different detrimental interventions are significant. And yet, this comparison shows that students attending a voucher school experienced the largest relative learning loss across all five studies of negative impacts.

Further, using CREDO’s days of learning conversion, this standard deviation unit decrease is the equivalent of losing 68 days of learning—more than one-third of an entire school year.⁵⁷ This conversion assumes that students learn, on average, the equivalent of one-quarter of a standard deviation unit—0.25—per year.

It is important to note that each study that the authors used only evaluates the impact of one factor in student achievement, when in reality, students are exposed to a variety of environmental and other factors, many of which have the potential to affect test scores. Therefore, it is possible—and likely—that students are exposed to more than one of the negative factors. For example, a student who witnesses violence in school may also feel unsafe in school. Further, when comparing these effects with the impact of participating in a voucher programs, it is also possible that students in both public schools and voucher programs experience additional impacts from some of the factors described in these comparison studies. However, there is no overwhelming evidence that would suggest bias in this comparison; there is no reason to believe that these known factors—teacher turnover, absenteeism, feeling unsafe in school, and exposure to violent crime at school— would occur more in the private sector versus the public sector or vice versa⁵⁸.

As noted above, this analysis uses only one rigorous, well-regarded study on each topic. Therefore, using alternative studies could lead to different results. The methods for selecting both specific voucher studies and comparison studies to add context are detailed in the Appendix. Selection criteria included rigorous methods, recent results, and studies that are representative of their field. The statistical significance; negative or positive sign direction; and magnitude are similar to other rigorous studies addressing these topic areas. In some cases, other similar studies may find an effect in reading but not in math, for example, but maintained the same sign, as well as similar significance and magnitude. This comparison focuses on math impacts only to allow for an aligned analysis of effects. Further, the evaluation of the D.C. voucher program only assessed one year of achievement data; however, these are similar to the longer-term impacts found in the other three recent rigorous studies compared here.

Ultimately, this comparison shows that the negative impact of the D.C. voucher program exceeds the impact of other factors which society deems harmful. 

The wave of voucher studies with negative findings in recent years have put some advocates in a bind. As studies consistently show negative outcomes, voucher proponents have backtracked or tried to spin the findings. For example, many voucher proponents point to other measures of voucher effectiveness, such as increased parent satisfaction.⁵⁹ The U.S. Conference of Catholic Bishops, a strong proponent for expanding voucher programs as a way to increase enrollment in parochial schools, wr