What is “Impact”?

Over the last several months many non-profits and NGOs I keep tabs on and follow have been releasing their annual reports. Due to the growing empirical rigor required by donor agencies, many of these organizations have been using the word “impact” in these reports. Or maybe it’s just because “impact” is a buzz word at the moment. (I don’t know.) Unfortunately, most of these organizations are misusing the word “impact” in their annual reports. So, I’d like to take a moment to provide a clear definition of the word “impact”. To do this, I’ll use some figures I made several years ago for a presentation I was never able to give.

Let’s say that you have some primary outcome – say, jobs or household income or calories consumed or maize yield – at some point in time your program or NGO institutes some sort of intervention – say, a training program or subsidized input coupon or asset/cash transfer. Notice that for all the primary outcomes I suggest here it is “good” when these things rise. Outcome variables – like child mortality or malaria incidence – could also be considered where it is “good” when these things fall.

You collect data on your primary outcome and, let’s say, you find that after the intervention your primary outcome variable began to increase (or increased in the rate at which it was increasing before the intervention). What is the impact of the intervention of your program or NGO?

Now let’s say you find that after the intervention your primary outcome variable began to decrease (or decreased the rate at which it was moving before the intervention). What is the impact of the intervention of your program or NGO?

In both these cases, we actually don’t know the real impact of the program. This is because “impact” (correctly defined) = “outcomes of participants” – “outcomes of participants in a world where the intervention does not exist”. Now, in the real world (compared to a lab setting) it is impossible to find out what would have happened to participants if the intervention did not exist. In the real world a true “counterfactual” does not exist.  In the real world, an approximate of this counterfactual must be created by constructing a comparison group – a group of individuals who mimic those individuals who are participating in the program as closely as possible.

So let’s see how collecting data on this proxy counterfactual would allow us to understand the estimated impact of your program or NGO’s intervention. To demonstrate the need for a counterfactual (a comparison group or sometimes called a control group), consider the following four cases:

Case 1: Here the impact is positive because in the absence of the intervention individuals would have remained on the same trajectory.

Case 2: Here the impact is negative because in the absence of the intervention individuals experienced a large boost in the primary outcome. In this case the intervention was instituted at roughly the same time as some other major change in the lives of individuals in the area of interest and those participating in the intervention have been effectively held back.

Case 3: In this case the impact is positive because in the absence of the intervention individuals actually ended up falling farther behind than those who participated in the intervention. For this case think of an intervention that was instituted at the same time as some negative and wide-ranging economic shock, like a drought or some other harsh weather event.

Case 4: In this case the impact is negative because in the absence of the intervention individuals continued on their upward trajectory in the primary outcome.

For matters of completeness, there are other possibilities to consider. For example, it could be that participants of your program and the counterfactual follow the same exact trend in the primary outcome. In this case, the program has zero impact.

Measuring impact is important (and likely increasingly so in the world of development funding), but it is potentially misleading to simply report program “outcomes” and claim them to be “impacts”. To claim something to be an “impact” you must have subtracted away the counterfactual. If a counterfactual has not been subtracted you simply have an “outcome”.