In the fall of 2013, while living in Kitale—a town in Western Kenya—I remember reading through several job market papers posted in blog form on the World Bank’s Development Impact blog. The experience, in part, inspired me to pursue graduate studies in development economics.
It is about time I wrote about this topic on this blog.
The Freakonomics podcast ran an episode last week entitled, “Is the Protestant Work Ethic Real?” The majority of the episode focused on research by Gharad Bryan, James Choi, and Dean Karlan evaluating the effects of a faith-based development program implemented by International Care Ministries in the Philippines. I wrote about this paper, back in March, but this podcast brings up a couple additional points worthy of discussion.
Seemingly everyone knows about ‘conflict minerals’. They are probably in our phones and laptops—even some of our jewelry is made out of them. Almost everyone wants to make ‘conflict minerals’ a thing of the past. Despite having little to do with Wall Street financial reform, Section 1502 of the Dodd-Frank Act implemented a reporting requirement on all publicly traded companies in the US. Modeled after the “Kimberly Process”, the international regulation on ‘blood diamonds’, the Dodd-Frank Act requires companies to report annually on audits investigating the presence of ‘conflict minerals’ in their supply chain.
Religion, spiritual practices, and faith are easily observable factors in the daily lives of people almost anywhere in the world. This leads many to speculate and theorize about the role of religion in driving economic and social outcomes. Positive correlations abound between religiosity and a host of factors that may influence economic success. Correlation, however, does not imply causation and pinning down the real causal relationship is complicated by the fact that people tend to choose their religion. Therefore, observing that people of faith experience different economic outcomes fails to account for the fact that unobservable personal characteristics may cause both religiosity and economic outcomes.
Earlier this year I read Tim Ogden’s book entitled, “Experimental Conversations: Perspectives on Randomized Trials in Development Economics“*. Including interviews from the “randomistas” (e.g. Michael Kremer, Ester Duflo, Abijit Bannerjee, Dean Karlan), “skeptics” (e.g. Angus Deaton), and folks not typically associated with RCTs (e.g. Tyler Cowen), it was an interesting book to read. One of the “dogs that didn’t bark” in the book was the statement that RCTs are the gold standard.
Last week I was introduced to a new podcast organized by the Accord Research Alliance (a group of faith-based development organizations) focusing on the intersection of faith-based development and impact evaluation. This is much overdue service and one that I hope survives the test of time.
The Soundcloud page for the podcast is here.
I’m happy to report that a paper stemming from my fieldwork in Kenya is (finally) being published in the Faith & Economics journal. Although F&E is a relatively niche journal, it is my first peer-reviewed research publication in an academic journal and reports on work I performed before I even began my MS program. The paper is titled: “Learning Toward Transformation: Evaluating Material, Social, and Spiritual Change in Western Kenya, here is the abstract:
In the year following my graduation from college, and before starting up graduate studies at Michigan State, I lived and worked in Western Kenya. I was working with the Africa Theological Seminary, which was developing a new economic development program and wanted some evaluation work done on this new program.
When I arrived in Kitale, Kenya and realized that my first task was to convince a bunch of theologians and missionaries of some fundamental needs for a rigorous and credible program evaluation. We needed (1) a control group, (2) two survey rounds, and (3) random assignment – in order to achieve the so-called gold standard of program impact evaluation… the randomized control trial.
Ultimately, I was unsuccessful in arguing for random assignment, which (in hindsight) was probably for the best. But I was able to sufficiently argue for a control group and two survey rounds, in the form of a baseline and endline survey. With these features in place, we were able to implement a difference-in-differences evaluation methodology.
Now, the program I was evaluating was being implemented by the Africa Theological Seminary. This being the case, their goals were a little more involved than the typical secular development program. Not only did they want to inspire social and economic change, they also wanted to generate positive spiritual change. Many faith-based development programs (at least in theory) hold these desires. In fact, the desire is so popular it even has it’s own name: “transformational development”. Many well-known faith-based organizations – such as World Vision, Compassion International, World Renew, Hope International, and Partners Worldwide – make explicit their goals of so-called transformational development.
Unfortunately, the ‘credibility revolution‘ has not fully caught on within the faith-based development community. So there is a severe lack of knowledge of what works and what doesn’t in achieving the goals of “transformational development”. In my view, this is quite a shame, because there are a lot of things that the secular development community could learn from the faith-based development community (and vice versa) if only some faith-based organizations engaged in some rigorous program evaluations.
This is where my working paper comes into the fold. I report some of the findings from the Africa Theological Seminary program evaluation as well as develop a simple evaluation framework for transformational development programs. Here is the abstract of the paper:
Transformational development—integrating the three goals of positive material, social, and spiritual change—is a popular concept among organizations and individuals who work in faith-based development. Despite broad agreement about the theory of transformational development, very little research has been produced that empirically investigates how to best achieve these goals. By using a difference-in-differences empirical strategy this study provides an example of a simple, yet rigorous, evaluation approach for measuring real progress toward meeting the goals of material, social, spiritual change. While providing detailed explanations of the evaluation methodology, this study presents early impacts of a church-based business skills training program in Western Kenya. In this study I find little evidence of impacts of material and social change, which is not surprising given the evaluation took place over only 18 months. I do, however, find statistically significant and positive evidence of spiritual changes, specifically in attitudes towards faith integration in work and business.
The paper is currently out for review, but I’d love to hear feedback or critiques or comments.
Over the last several months many non-profits and NGOs I keep tabs on and follow have been releasing their annual reports. Due to the growing empirical rigor required by donor agencies, many of these organizations have been using the word “impact” in these reports. Or maybe it’s just because “impact” is a buzz word at the moment. (I don’t know.) Unfortunately, most of these organizations are misusing the word “impact” in their annual reports. So, I’d like to take a moment to provide a clear definition of the word “impact”. To do this, I’ll use some figures I made several years ago for a presentation I was never able to give.
Let’s say that you have some primary outcome – say, jobs or household income or calories consumed or maize yield – at some point in time your program or NGO institutes some sort of intervention – say, a training program or subsidized input coupon or asset/cash transfer. Notice that for all the primary outcomes I suggest here it is “good” when these things rise. Outcome variables – like child mortality or malaria incidence – could also be considered where it is “good” when these things fall.
You collect data on your primary outcome and, let’s say, you find that after the intervention your primary outcome variable began to increase (or increased in the rate at which it was increasing before the intervention). What is the impact of the intervention of your program or NGO?
Now let’s say you find that after the intervention your primary outcome variable began to decrease (or decreased the rate at which it was moving before the intervention). What is the impact of the intervention of your program or NGO?
In both these cases, we actually don’t know the real impact of the program. This is because “impact” (correctly defined) = “outcomes of participants” – “outcomes of participants in a world where the intervention does not exist”. Now, in the real world (compared to a lab setting) it is impossible to find out what would have happened to participants if the intervention did not exist. In the real world a true “counterfactual” does not exist. In the real world, an approximate of this counterfactual must be created by constructing a comparison group – a group of individuals who mimic those individuals who are participating in the program as closely as possible.
So let’s see how collecting data on this proxy counterfactual would allow us to understand the estimated impact of your program or NGO’s intervention. To demonstrate the need for a counterfactual (a comparison group or sometimes called a control group), consider the following four cases:
Case 1: Here the impact is positive because in the absence of the intervention individuals would have remained on the same trajectory.
Case 2: Here the impact is negative because in the absence of the intervention individuals experienced a large boost in the primary outcome. In this case the intervention was instituted at roughly the same time as some other major change in the lives of individuals in the area of interest and those participating in the intervention have been effectively held back.
Case 3: In this case the impact is positive because in the absence of the intervention individuals actually ended up falling farther behind than those who participated in the intervention. For this case think of an intervention that was instituted at the same time as some negative and wide-ranging economic shock, like a drought or some other harsh weather event.
Case 4: In this case the impact is negative because in the absence of the intervention individuals continued on their upward trajectory in the primary outcome.
For matters of completeness, there are other possibilities to consider. For example, it could be that participants of your program and the counterfactual follow the same exact trend in the primary outcome. In this case, the program has zero impact.
Measuring impact is important (and likely increasingly so in the world of development funding), but it is potentially misleading to simply report program “outcomes” and claim them to be “impacts”. To claim something to be an “impact” you must have subtracted away the counterfactual. If a counterfactual has not been subtracted you simply have an “outcome”.