Evaluating ‘Transformational Development’ – Forthcoming in F&E

I’m happy to report that a paper stemming from my fieldwork in Kenya is (finally) being published in the Faith & Economics journal. Although F&E is a relatively niche journal, it is my first peer-reviewed research publication in an academic journal and reports on work I performed before I even began my MS program. The paper is titled: “Learning Toward Transformation: Evaluating Material, Social, and Spiritual Change in Western Kenya, here is the abstract:

Transformational development—integrating the three goals of positive material, social, and spiritual change—is a popular concept among organizations and individuals who work in faith-based development. Despite broad agreement about the theory of transformational development, very little research empirically investigates how to best achieve these goals. By using a difference-in-differences empirical strategy this study provides an example of a simple, yet rigorous, evaluation approach for measuring real progress toward meeting the goals of material, social, and spiritual change. Although this study only finds marginally significant effects after one year, average impact estimates are relatively large. Thus, while economically meaningful impacts may be present, this study encounters difficulty with statistical identification due to a relatively small sample size. Nevertheless, this paper provides a useful example, with suggestions for improvement, for future research evaluating transformational development programs.

Ever since Wayne Bragg (1984) coined the term ‘transformational development’, the three goals positive material, social, and spiritual change have played a central role in the design of many faith-based development organization’s programs.* As the abstract mentions, however, rarely have programs that aim to achieve transformational development been evaluated in any sort of rigorous manner. This is potentially problematic because there are many very fundamental questions that could be answered by well-designed empirical research.

For example: Do these three goals complement each other? In line with Weber’s hypothesis does an integrated view of faith in the material world lead to improved material outcomes? Or are there implicit trade-offs between spiritual and material development? Does improved material well-being make people less spiritually faithful? Answers to these questions are not only of interest to organizations who aim to implement transformational development (such as World Vision, Compassion International, etc.), but also to secular development organizations who are confronted with tricky decisions about how to manage the role of faith in the lives of those they aim to help.

In this paper I aim to encourage faith-based development organizations to undertake rigorous evaluations of their programs by providing an example of an impact evaluation of the Discipling Marketplace Leaders program. This program was developed by the Africa Theological Seminary and facilitated business skills trainings through local churches across three cities in Western Kenya. As part of the fieldwork, I collected data before and after the program on both program participants and non-participants. Careful work selecting non-participants allowed for a difference-in-differences empirical strategy and estimation of causal impacts.

Beyond reporting results of the study I come to two broad conclusions. First, empirically evaluating spiritual change is possible, albeit in an approximate manner. It is no longer appropriate for Christian (or any other religion for that matter) aid and development organizations to refrain from rigorous evaluation because, “spiritual outcomes are immeasurable”. Although this statement may indeed be axiomatically true, I present methods in this paper for approximating spiritual changes that can be plausibly attributed to transformational development programs.

Second, this paper highlights the need for a priori power calculations so to understand the necessary sample size needed to detect effects. This evaluation suffers dramatically from a relatively small sample size driven primarily by resource constraints. The administrators of the Discipling Marketplace Leaders program felt a need to learn from rigorous analysis on their relatively untested program design. This attitude is fervently applauded by me, but a clear lesson from this paper is without a sufficient sample size (and sampling strategy) more rigorous evaluations hardly provide better policy-relevant lessons than less rigorous evaluations. At the end of the day, this paper only finds marginally significant impacts even in the presence of relatively large (and potentially meaningful) average impact estimates.

I’ll end this post with two shameless plugs. First, the Discipling Marketplace Leaders program is run by excellent leadership who have shown unbelievable humility to evaluate the impact of their program compared to the counterfactual world where they didn’t exist. If you would like to support their work, either financially or otherwise, follow this link and click ‘donate’. Second, working on this paper has increased my interest in evaluating ‘transformational development’ programs. If you are connected to a development program aiming to achieve the three goals of positive material, social, and spiritual change, send me a note. I’d love to talk about collaborating.

*For more on transformational development see the work of Bryant Myers.