We invited 113 organizations to apply in this cause, using the process described here. 50 completed our Round 1 application; 54 did not apply (the remaining 9 applied in other causes).
Informally speaking (we will publish better support of what follows when we put up our full research on the cause), the achievement gap is a thorny problem, notorious for resisting well-funded and well-intentioned interventions. We therefore looked for charities with rigorous evidence that they've improved academic outcomes for disadvantaged children. This means giving data not only on their participants' academic performance, but using some appropriate "comparison group" to attempt to get at how participants would likely do without the benefit of the program.
We used the following principles in naming finalists:
Focus on data reflecting academic performance, such as graduation rates, college enrollment rates, attendance, grade promotion, and test scores. While data like this is very far from perfect in indicating how well children are learning and what is happening in terms of their life opportunities, we feel that if a charity's activities are indeed leading to improved academic ability and understanding (especially math and reading), this should be reflected in improved performance on hard measures, as well as other benefits. We are looking for an organization that can strongly demonstrate its impact.
We have not seen strong evidence linking factors such as self-esteem to academic performance, and so while improving children's self-esteem may be valuable, we are not convinced that it is a way to help close the achievement gap. Similarly, we do not believe that survey data showing self-reported attitude change is convincing evidence of improved academic abilities and performance.
Look not just for improvement, but for improvement above and beyond what might be expected without the organization's help. It isn't enough to see that test scores improved over some time period; NYC test scores have improved citywide over time, and this could be due to a variety of reasons that have nothing to do with nonprofit interventions. To us, evidence that a program worked means evidence that the participants outperformed some comparable "comparison group."
Look carefully for selection bias. We believe that child and family motivation are extremely important in education. Many studies using a "comparison group" are comparing voluntary participants in a program to non-participants, and the differences they pick up could therefore be driven by differences in motivation.
Our strongest applicants showed or referred us to evidence that is at least attempting to assess the impact of their program above and beyond a tendency to attract more motivated students; some accomplished this through randomization, others by selecting their clients in a way that seems unlikely to involve selection bias; others by comparing changes in test scores between their populations and "control groups" (this method is more problematic than using randomized design, but less so than simply looking at whether their students outperform others).
Require evidence that a program has worked before. Some applicants submitted extremely rigorous, methodologically strong studies showing practically no difference between their participants and a control group. We believe that this sort of self-evaluation and documentation is valuable, but we are looking for proven, effective, scalable ways of helping people, not just strong research techniques.
Learning through an Expanded Arts Program (LEAP) promotes a specific in-school curriculum, and attached a study showing that children randomly selected to receive this curriculum tested better than those who had been randomly selected not to.
Children's Scholarship Fund provides scholarships for private school to K-8 students, and attached a study on very similar programs that also employed randomization (although it showed better results only for African-American children; we plan to investigate this study further).
Teach For America, which trains and subsidizes recent college graduates to teach in disadvantaged districts, attached a broad study of effects at several of its sites, finding that its teachers' students perform about the same on reading and better on math.
KIPP, Achievement First, and Replications Inc. all focus on creating new schools, and all examined the changes in their students' test scores versus those of students in nearby districts (a far more problematic comparison than the randomization-based ones above, but still one that attempts to look at program effects beyond its selection of students).
New Visions for Public Schools, a school support organization, looked at a series of schools it created by matching its students to students in similar districts with similar academic records, and found consistently improved attendance and grade promotion rates (though test score outcomes were mixed).
Student Sponsor Partners claims to seek out "academically below-average" children and pay their way through private high school - we need more detail on its selection process, but if it is avoiding bias, then its results (substantially higher graduation rates, and enrollment in better colleges, compared to kids from the same middle schools) are impressive.
The LEDA Scholars program, an extremely selective program that attempts to raise college expectations for top disadvantaged/minority students, attached a study showing much better results for its scholars than for those who made its final round (although we are now in the process of investigating just how different these two groups were prior to LEDA's intervention).
Harlem Center for Education and Double Discovery Zone both run the Federal Talent Search program, which, as HCE pointed out, has been shown preliminarily to increase college matriculation rates compared to a control group matched on observable characteristics including past academic record (see this study).
The St. Aloysius School runs all the way from preschool through 8th grade, with high school support - an intensity of intervention that we haven't seen in any other program - and claims a 98% graduation rate. Despite little information on outcomes, we found the model compelling enough to investigate further.
Other submissions either had no data on academic outcomes, or did not provide what we see as necessary context for this data (i.e., a comparison group that there's at least some reason to believe is appropriate to compare), or showed little to no apparent impact of their program. It's possible that these applicants have the information we want, and didn't send it due to misinterpretations of our Round 1 application, time constraints, or other reasons. But due to time constraints of our own, we opted to focus on the applicants who seemed most promising.
As a note, no afterschool/summer programs demonstrated improved academic outcomes either for their own program or for a very similar one, and we haven't encountered any independent evidence that afterschool/summer programs in general (whether academically or recreationally focused) have consistent impact on academic performance.
