‘Learning Toward Transformation’: A working paper from my time in Kenya

In the year following my graduation from college, and before starting up graduate studies at Michigan State, I lived and worked in Western Kenya. I was working with the Africa Theological Seminary, which was developing a new economic development program and wanted some evaluation work done on this new program.

When I arrived in Kitale, Kenya and realized that my first task was to convince a bunch of theologians and missionaries of some fundamental needs for a rigorous and credible program evaluation. We needed (1) a control group, (2) two survey rounds, and (3) random assignment – in order to achieve the so-called gold standard of program impact evaluation… the randomized control trial.

Ultimately, I was unsuccessful in arguing for random assignment, which (in hindsight) was probably for the best. But I was able to sufficiently argue for a control group and two survey rounds, in the form of a baseline and endline survey. With these features in place, we were able to implement a difference-in-differences evaluation methodology.

Now, the program I was evaluating was being implemented by the Africa Theological Seminary. This being the case, their goals were a little more involved than the typical secular development program. Not only did they want to inspire social and economic change, they also wanted to generate positive spiritual change. Many faith-based development programs (at least in theory) hold these desires. In fact, the desire is so popular it even has it’s own name: “transformational development”. Many well-known faith-based organizations – such as World Vision, Compassion International, World Renew, Hope International, and Partners Worldwide – make explicit their goals of so-called transformational development.

Unfortunately, the ‘credibility revolution‘ has not fully caught on within the faith-based development community. So there is a severe lack of knowledge of what works and what doesn’t in achieving the goals of “transformational development”. In my view, this is quite a shame, because there are a lot of things that the secular development community could learn from the faith-based development community (and vice versa) if only some faith-based organizations engaged in some rigorous program evaluations.

This is where my working paper comes into the fold. I report some of the findings from the Africa Theological Seminary program evaluation as well as develop a simple evaluation framework for transformational development programs. Here is the abstract of the paper:

Transformational development—integrating the three goals of positive material, social, and spiritual change—is a popular concept among organizations and individuals who work in faith-based development. Despite broad agreement about the theory of transformational development, very little research has been produced that empirically investigates how to best achieve these goals. By using a difference-in-differences empirical strategy this study provides an example of a simple, yet rigorous, evaluation approach for measuring real progress toward meeting the goals of material, social, spiritual change. While providing detailed explanations of the evaluation methodology, this study presents early impacts of a church-based business skills training program in Western Kenya. In this study I find little evidence of impacts of material and social change, which is not surprising given the evaluation took place over only 18 months. I do, however, find statistically significant and positive evidence of spiritual changes, specifically in attitudes towards faith integration in work and business.

The paper is currently out for review, but I’d love to hear feedback or critiques or comments.