We are data nerds. We believe, as a matter of integrity and intellectual honesty, that organizations claiming to help others should support those claims with convincing quantitative evidence of impact. We rely on such evidence in law, public policy, medicine, engineering, and other diciplines that affect our daily lives. We should hold well-intentioned charities to the same standard.
We calculate these dimensional scores from 17 development indicators collected per village (more on that below). The data show partner and control villages on divergent development paths. Both partner and control villages performed poorly during two bad agricultural years (2015 and 2016). Yet, starting in 2015, partner villages outperformed control villages in terms of waterborne illness, education, lifestyle, livestock, and business. In 2017, partner villages started outperforming control villages in terms of agricultural output. Do these descriptive statistics contain statistically significant impacts? Find out below.
Our most recent impact evaluation occurred in early 2018. Our approach wasn't fancy or expensive (more on that below). Instead, it tested a foundational assumption that financing village-led projects without any sort of preconceived development agenda would allow local communities to illuminate development pathways obscured from the outside. In other words, we funded a bunch of projects, collected a bunch of development indicators, and tested whether anything had changed. Did communitites improve and, if so, how? Here's what we found.
|Metric||Boys in nursery||Girls in nursery||Girls in secondary||Goats||Waterborne illness||Non-ag businesses||Homes w/metal roofs|
|actual change||+11 boys||+14 girls||+4 girls||+23 goats||-133 cases||+11 businesses||+17 homes|
Using community surveys spanning 2014 (baseline) to 2017, across 27 projects and 39 villages, we tracked the following 17 development indicators pertaining to health, education, business, lifestyle, agriculture, and livestock, both in partner villages and control villages that want to partner with Village X. We chose these indicators because they are easy to collect and indicative of village development trends. We then applied a difference-in-differences analysis to detect our model’s impacts, controlling for village population and number of households.
Project profiles on this website for repeat village parters (those that have completed at least one project) have graphs showing how development scores for a given village change over time, including an overall village development score and scores for each of the dimensions set forth above. These scores are not precision instruments. Instead, they capture village development trends over time.
Want to take a deeper dive? Checkout our dataset here. It's part of our 100% tranparency guarantee.
We are not satisfied with our impact evaluation. It's a good start, but we can do better. In particular, we could track how projects we finance affect individual household spending within a village. How do families modify their spending in response to village-led projets? Do the projects make families wealthier?
Answering these questions requires overcoming two challenges: (1) our current portfolio of partner villages is not large enough (we need at least 300 per year); and (2) we do not possess a large monitoring and evaluation budget. We plan to scale our operations to overcome the first challenge and to partner with academic researchers to overcome the second one.