In a nutshell

What do they do? The Children’s Scholarship Fund provides partial scholarships to low-income children in grades K-8, to help them enroll in private schools. These scholarships cover between 25% and 75% of private school tuition (depending on the family's income). The program currently serves more than 25,000 students in the United States, with 5,480 scholarship students in New York City.

Does it work? Having reviewed a variety of academic studies, we do not feel that there is empirical support for CSF's approach of granting partial-tuition scholarships.

What do you get for your dollar? CSF spends approximately $1200 per student, overall; we estimate that the total cost of a child's private-school education (paid for not only by the scholarship, but by the participating family and by the private school's own donors) is comparable to the cost of a public-school education (in the neighborhood of $12,000-$15,000 per student per year in NYC).

The Details

What do they do?

CSF offers partial-tuition scholarships to low-income students, helping to finance their attendance at private school for grades K-8. The program is open (on a first-come, first-served basis) to all who meet basic income requirements and remain in good standing at their choice of school.

Application and eligibility

CSF awards scholarships on a first-come, first-served basis (Attachment A-1 Pg 3) to students in grades K-8. Families must provide proof of income and enrollment in a private school to be eligible for assistance (Attachment A-1 Pg 3), but there are no academic requirements, and students maintain their scholarships through eighth grade provided that the family continues to pay its portion of tuition and the child maintains good standing at the school s/he attends (Attachment A-1 Pg 4).

Students may attend any accredited school; most attend private and parochial schools (Attachment A-1 Pg 2). We have very little specific information about the children enrolled in CSF and the schools they attend; unlike many of the charities we examine, CSF is relatively "hands-off" (in that its interaction with families is limited to verifying eligibility and providing funding). CSF did provide a list of schools attended by scholarship recipients (Attachment B-13).

Size of scholarships

Awards cover between 25% and 75% of a student’s tuition, and are calculated based on family size and income: families at or below 100% of the federal poverty line receive 75% of their child's tuition, those at up to 185% of the poverty line receive 50% assistance, and those at up to 270% of the poverty line receive 25% assistance (Attachment A-1 Pg 2). In Attachment A-1, CSF New York states that its average scholarship size is $1,488, with an average family share of $1,397 from an average family income of $22,440 (though it also states that the average tuition is $2,915 - we aren't sure why there is an apparent $30 discrepancy).

It seems important to us that scholarship size be carefully calibrated to families' needs. Insufficient scholarships would fail to make private school affordable; on the other hand, over-generous criteria would presumably subsidize families more than necessary, rather than serving as many students as possible. Unfortunately, we have very little to go on in assessing whether scholarship size is well calibrated to income, as CSF did not provide us with the details of how scholarship size is determined. In conversation, CSF's New York Executive Director indicated that CSF monitors the percentage of students that continue in the scholarship program from each year, and seeks to maintain that percentage at 80% or above.

We gain some indication of scholarships' impact on affordability from Attachment B-4, an evaluation of a program similar to CSF's (discussed more below) for which scholarship recipients were chosen by lottery (rather than through a first-come first-served approach). Attachment B-4 Pg 2 states that approximately 75% of scholarship recipients ended up enrolling in private school, compared to 11% of non-recipients.

Does it work?

We find the CSF model to have a great deal of intuitive appeal: anecdotally, we'd guess that the low-income children most likely to be eligible (and interested) have seriously deficient public schools, and would have no trouble identifying private schools with better staff and learning environments. We would therefore expect that children who receive CSF scholarships - allowing them to attend private schools for up to 9 years - would be much better positioned to perform academically. However, from what we've seen, available empirical evidence broadly contradicts this intuition.

CSF and similar programs have been analyzed in a range of studies that awarded scholarships to interested students using randomized lotteries, then followed both the students who had been awarded scholarships and the students who hadn't. There is some dispute over exactly what the results indicate: some researchers point to a small impact on academic performance, found only with African-American students, while others dispute even this impact and argue that the impact of scholarships is essentially negligible. We feel that the latter case is slightly stronger.

We believe there are several possible explanations for why this intuitively appealing program may not have the desired effects, and ultimately would not bet on it as a way of significantly improving students' academic performance.

Studies on academic impact

We reviewed academic research on U.S. voucher programs structured similarly to CSF, including four studies of privately funded voucher programs (in New York, Washington D.C., Charlotte, Dayton) and two publicly funded programs (in Milwaukee and Cleveland). Each study attempted to measure the impact that vouchers or scholarships had on participants' academic performance, as measured by standardized test scores. We found the literature reviews prepared by the RAND Corporation (Attachment B-1 Pgs 79-94) and the Economic Policy Institute (Attachment B-2 Pgs 5-20) particularly informative.

New York City

Researchers followed the academic performance of about 1,300 students who were randomly selected to receive offers of partial-tuition scholarships, as well as 1,300 students who did not receive such offers. All such students entered grades K-4 in 1997; their performance was followed over the course of three years (Attachment B-4 Abstract).

This is the only relevant study we know of for which all raw data is publicly available, making challenges by other researchers possible - and different researchers did reach different interpretations. Specifically:

• Paul Peterson and William Howell (then at Harvard) and David Myers and Daniel Mayer (researchers at Mathematica Policy Research) conducted the initial analysis. They found no statistically significant impact on academic performance (as measured by standardized test scores) for the group of students as a whole, but a moderate and statistically significant impact (between 4.1 and 8.2 percentile points on the Iowa Test of Basic Skills) on the academic performance of African-American students (Attachment B-3).
• Two researchers at Princeton University, Alan Krueger and Pei Zhu, argued that the apparent impact on African-American students was no longer statistically significant if another group of students (those for whom pre-experiment scores were unavailable) were included in the analysis and the definition of African-American was expanded to include mixed race black/hispanic students (Attachment B-4).
• Peterson and Howell responded to Krueger and Zhu, conceding that both proposed adjustments would point to weaker effects, but challenging the validity of these adjustments and observing that even with Krueger and Zhu's definition of African-American, 2 of the 5 classes studied still showed statistically significant improvement on test scores (Attachment B-5).
• Reviewing the new analysis presented by Krueger and Zhu, Myers and Mayer concluded that "one must remain cautious when interpreting the findings for African Americans" (Attachment B-6 Pg 5).

We don't find Peterson's and Howell's case very convincing, given both its high sensitivity to the exact methodological approach taken (as Krueger and Zhu demonstrated) and its reliance on after-the-fact decisions to focus on subgroups of the population.

Other experiments

The New York City experiment is the only one we know of that produced a strong data set with few methodological concerns and followup data available for most participants. The other major studies we have investigated are summarized below; we believe their results are inconclusive and add relatively little to our understanding of scholarships' effects.

Howell and Peterson designed and conducted experiments in Washington D.C., Dayton, and Charlotte (Attachment B-8 Pg 6), with similar structure to the New York City experiment. The results they found were broadly consistent with their own findings from New York: no statistically significant impact on the academic performance of the sample as a whole, and moderate statistically significant impacts (around 3-9 percentile points, though a few groups saw smaller or even negative impacts) on the academic performance of African-American students (Attachment B-8 Pg 13, Attachment B-9 Pg 8). However, the quality of these experiments is lower than that of the New York experiment, due to low participation rates in follow-up surveys. 67% of students in the New York experiment participated in follow up exams (Attachment B-3 Pg 43); by contrast, only half of students participated in the follow up exams in Dayton and D.C., and about 40% participated in Charlotte (Attachment B-2 Pg 17). Although we don't know whether students who left the study differed systematically from those who stayed, and thus can't tell whether or in which direction their attrition biased the results, large levels of attrition pose a significant threat of bias to an initially randomized study. This concern is prominent in other comparative discussions of these experiments, including the analysis done by Evidence Based Programs. In addition, because data has not been made public for the non-New York studies, other researchers haven not had the opportunity to critique the results as Krueger and Zhu did for New York City.

Milwaukee conducted a publicly funded voucher experiment. Cecilia Rouse of Princeton and Jay P. Greene of the Manhattan Institute found positive impacts on academic performance for students who received vouchers; John Witte of the University of Wisconsin responded with a critique of their methodology, observing that only 48% of the students lotteried out of the voucher program stayed in the public school system, and there is reason to believe that the remaining group included a disproportionate number of low performing and learning-disabled students (those who could not, or would not, gain admittance to a private school). Witte's re-analysis found a small negative impact of receiving a voucher (Attachment B-2 Pgs 7-8).

We do not discuss the Cleveland experiment (Attachment B-10 Pg 4); this was not a lottery-based experiment, and there is little data on background characteristics of students. It's therefore highly unclear to what extent the two groups of students (recipient and non-recipients of scholarships) were otherwise comparable.

Impact beyond academic performance

Some researchers have attempted to evaluate the impact of voucher/scholarship programs beyond academic achievement, looking at the difference in altruistic behavior by students (and their parents) who are awarded vouchers vs. students and parents who are not. While we are sympathetic to the idea that such non-academic impact is important, the analysis we've seen on this topic - submitted by CSF - does not, to us, offer a compelling case for the impact of scholarships.

The study CSF submitted (Attachment B-14) conducted an exercise in which students were given money and then offered the opportunity to donate to a specified charity; students who chose to donate more were taken to be demonstrating greater altruism. The study found that students who (by lottery) had been awarded scholarships for private schools rated as more altruistic than students who had not been awarded such scholarships (Attachment B-14 Pg 4). The researchers used similar experiments to gauge students' generosity toward their peers, and the generosity of their parents, finding no statistically significant effects for either (Attachment B-14 Pg 4). Furthermore, the two groups of students (those who had been awarded scholarships and those who had not) do not appear otherwise comparable: those who were awarded scholarships had significantly higher incomes (to begin with) than those who were not (Attachment B-14 Pg 30).

Interpreting the evidence

While the idea of providing scholarships to low-income students is intuitively appealing, the following factors may prevent it from significantly improving academic outcomes:

• Families who choose to use scholarships may be more motivated than other families to begin with, such that they would find other ways of pursuing a better education even in the absence of these scholarships. This possibility seems especially strong when the scholarships are partial rather than full (as is the case for CSF as well as for the voucher experiments discussed above), meaning that only families willing to pay extra for education participate.
• If disadvantaged children are already academically far behind by the time they are eligible for these scholarships, it's possible that a private school - focused largely on serving less disadvantaged children - is not the appropriate remedy. Schools such as those run by KIPP primarily serve disadvantaged children, and may thus be better positioned to build a curriculum around their particular needs and help them catch up.

Most of the empirical evidence we discuss above is highly ambiguous, due to methodological concerns; however, the New York City Voucher Experiment appears to have produced a relatively clean and bias-free data set, and it implies that the impact of scholarships on academic performance is small (and limited to subgroups of the population) if it exists at all.

What does it cost?

CSF expects to spend approximately $30 million to provide scholarships to 25,000 children during the 2008 school year (Attachment D-1). Average tuition at a school attended by a CSF scholarship recipient is about $3,000 (Attachment A-1 Pg 3).

However, the actual cost of educating a student at schools that CSF scholarship recipients attend is likely significantly higher than the tuition, as private schools may rely on fundraising of their own to supplement revenue from tuition. We don’t have data on costs for the schools in which CSF students enroll (these schools are religious institutions and are not required to file their financials publicly), but to get a ballpark figure, we looked at the financials of St. Aloysius, a parochial school in New York City that is also one of our applicants. St. Aloysius expects total expenses of $3.25 million in 2008 and serves approximately 300 students, which comes out to overall expenses of ~$11,000/student. This in the same ballpark as public school per-student costs ($13,755 per student in NY, according to the US Census Bureau).

The organization

Financials. All data is taken from CSF's Form 990s (available via GuideStar), Attachment A-2 Pg 5, and Attachment D-1.

In 2006, the latest year for which a Form 990 is available, CSF held $8,017,644 in assets (or approximately 40% of that year's expenditures).

Board of directors. CSF has a fifty-member National Board of Advisers that includes businessmen, prominent politicians (from both sides of the aisle), educators, and non-profit leaders. The exact role of the Board in the management of the organization is unclear to us.

Conclusion

On its face, we find the CSF model intuitively compelling; however, the empirical evidence shows much weaker effects than we would expect. Ultimately, we have more confidence in KIPP, whose fundamentally different approach is supported by what we feel is a stronger (although less rigorous) empirical case.

Attachments

A. Application and response

• Attachment A-1: Round 1 Application Response
• Attachment A-2: Application - Round 2
• Attachment A-3: Round 2 Application Response

B. Program related attachments

• Attachment B-1: RAND School Voucher Literature Review
• Attachment B-2: EPI School Voucher Literature Review
• Attachment B-3: “School Choice in New York City” – Howell and Peterson (2002)
• Attachment B-4: “Another Look at the New York City Voucher Experiment” – Krueger and Zhu (2003)
• Attachment B-5: “Efficiency, Bias, and Classification Schemes” – Peterson and Howell (2003)
• Attachment B-6: “Comments on ‘Another Look at the New York City Voucher Experiment’” – Myers and Mayer/Mathematica Research (2003)
• Attachment B-7: “An Evaluation of the Children’s Scholarship Fund” – Peterson (2001)
• Attachment B-8: “Test Score Effects of School Vouchers in Dayton, New York, and Washington D.C.” – Howell, Wolf, Peterson, and Campbell (2000)
• Attachment B-9: “The Effect of School Choice: An Evaluation of the Charlotte Children’s Scholarship Fund” – Greene (2000)
• Attachment B-10: Jay P Greene Literature Review (2000)
• Attachment B-11: Complete List of CSF studies
• Attachment B-12: “Baseline Findings for an Evaluation of the Children’s Scholarship Fund(CSF) in Los Angeles” – Quigley (2006)March8-06FINALQuigley
• Attachment B-13: List of Public Schools From Which Scholarship Recipients Came and Private Schools In Which They Enrolled
• Attachment B-14: Bettinger, Eric and Robert Slonim. (2004). Using Experimental Economics to Measure the Effects of a Natural Educational Experiment on Altruism. NBER Working Paper 11725. [CSF submitted this in hardcopy. No ungated copy available.]

C. Organization related attachments

D. Financials

• Attachment D-1: Budget For Fiscal Year 2008
• Attachment D-2: Expected Major Donations in 2008 (withheld due to confidentiality request)