Our Impacts
We are data nerds. We believe, as a matter of integrity and intellectual honesty, that organizations claiming to help others should support those claims with convincing quantitative evidence of impact. We rely on such evidence in law, public policy, medicine, engineering, and other disciplines that affect our daily lives. We should hold well-intentioned charities to the same standard.
The data we've collected since 2014 show partner and control villages on divergent development paths, with the former outperforming the latter. Below we graph partner v. control villages over time in terms of changes in a overall development score and categorical scores for health, education, business, lifestyle, agriculture, and livestock.
Do these graphs reveal statistically significant impacts? Find out below.
Our most recent impact evaluation occurred in early 2019. Our approach wasn't fancy or expensive. Instead, it tested a foundational assumption that financing village-led projects without any sort of preconceived development agenda would allow local communities to illuminate development pathways obscured from the outside. In other words, we funded a bunch of projects, collected a bunch of development indicators, and tested whether anything had changed.
Did communitites improve and, if so, how? Here's what we found. For starters, while control villages exhibited a 19% increase in overall development score over four years, partner villages posted a whopping 73% increase during the same timeframe. This result was statistically significant at the 1% level. The table below reveals which of the 25 development data metrics we use contributed to this development surge by treatment villages.
| Metric | Boys in nursery | Girls in nursery | Goat assets | Waterborne illness | Infant deaths | Agri biz | Other biz | Homes w/ metal roofs |
|---|---|---|---|---|---|---|---|---|
| % change | +50% | +67% | +83% | -64% | -100% | +81% | +100% | +50% |
| impact per village | +7 boys | +12 girls | +45 goats | -109 cases | -2 deaths | +13 biz | +8 biz | +13 homes |
| p-value | p<0.05 | p<0.01 | p<0.01 | p<0.01 | p<0.05 | p<0.01 | p<0.01 | p<0.05 |
| total impact (all villages) | +133 boys | +228 girls | +855 goats | -2071 cases | -38 deaths | +247 biz | +152 biz | +247 homes |
Using community surveys spanning 2014 (baseline) to 2018, across 41 projects and 56 villages, we tracked the following 25 development indicators pertaining to health, education, business, lifestyle, agriculture, and livestock, both in partner villages and control villages that want to partner with Village X. We chose these indicators because they are easy to collect and indicative of village development trends. We then applied a difference-in-differences analysis to detect our model’s impacts, controlling for village population, number of households, and surrounding district.
Project profiles on this website have graphs showing how development scores for a given village change over time, including an overall village development score and scores for each of the sub-categories set forth above. These scores are not precision instruments. Instead, they capture village development trends over time.
Want to take a deeper dive? Checkout our dataset here. It's part of our 100% tranparency guarantee.
We are not satisfied with our impact evaluation. It's a good start, but we can do better. In particular, we could track how projects we finance affect individual household spending within a village. How do families modify their spending in response to village-led projects? Do the projects make families wealthier? We could check our data against development trends revealed by satellite imagery.
Answering these questions requires overcoming two challenges: (1) our current portfolio of partner villages is not large enough (we need at least 100 per year); and (2) we do not possess a large monitoring and evaluation budget. We plan to scale our operations to overcome the first challenge and to partner with academic researchers to overcome the second one.
