Why We Do Rigorous Evaluation

Over the past week there has been an interesting debate between activists and academics in the comments section of this article, entitled 10 Reasons Fair Trade Coffee Doesn’t Work by Bruce Wydick.

As the title of the piece hints, Bruce isn’t the biggest fan of Fair Trade coffee. As you can probably imagine, this opinion is not generally accepted. One commenter, Dean’s Beans Organic Coffee Company, (perhaps sarcastically), invites Bruce (who is releasing a book on the global coffee industry) to “see what real fair trade looks like”. Wydick responds,

Thanks for your reply, and it is a great illustration of why we need to do serious academic research and not rely on observations and anecdotes. The differences you observe may partly be caused by membership in a fair trade cooperative. More likely they are due to a selection bias where villages and groups of people who are well-organized and forward thinking do things like join fair trade coops. In other words, many of the differences we observe in communities are due to causal factors other than the program. The reason we do serious research rather than rely on anecdotes and stories is to separate what appears to be true from what is true, on average. There are a number of ways to statistically separate and identify causal effects that researchers to find out what true impacts are. (If you want to explore these, I would suggest with Poor Economics, by Esther Duflo and Abhijit Banerjee as a start.) What is true, on average, and corroborated by paper after paper is that fair trade coffee has minuscule impacts compared to other worthy types of development programs, and it is sad because many people as yourself are unaware of how poorly fair trade coffee performs as an approach to free people from poverty.

This was responded to with misunderstanding and naiveté. Wydick goes on to clarify,

Appreciate the comments people have posted on the article, and I want to respond to a few more of them.

Several people have downplayed the results of “academic studies” in favor of hearing individual voices of producers and others involved with fair trade. When evaluating the merits of a program or an issue, individual voices are important, but not as conclusive as rigorous research that seeks to find the “average” of what the voices (and data) are saying. For example, if any of us had a child who was suffering from a rare form of cancer, we would make our treatment choice based on the evidence the best academic research gives in an evaluation of each type of treatment. And we clearly wouldn’t base such a critical decision on how a certain individual felt who had had the treatment, or, especially, the opinions of those in the pharmaceutical companies who sell the drug, who of course have a conflict of interest in how the treatment is perceived by the public. Development interventions are just as serious as cancer treatments because we are dealing with the lives and livelihoods of people often living very close to subsistence, where bad interventions can prove very costly. They have to be rigorously evaluated to see if their claims hold up.

Academics are kind of in this referee role in a number of areas, and many people do not like their findings. They may conflict with strongly held prior beliefs, identities, and institutions in which they have a personal investment. The case of global warming is another example where academics clearly show a consensus about its existence and causes. Yet there are many who feel comfortable countering the evidence presented in a high-quality study in the journal Science with “how darn cold it was in Chicago last week.” Others will deny global warming because they don’t want to change their habits, and so denying that it exists justifies inaction.

In the case of fair trade coffee, the central purpose of the system is to increase the income of impoverished coffee growers. This is an effort entered into with the best of motives. But research overwhelmingly shows it does not do this well, if at all. Since this is the case, rather than take a defensive posture toward the findings of academic studies, it would be refreshing to see the fair trade coffee industry begin to act more constructively and begin to correct the flaws in the system.

Obviously, I side with Wydick in this debate. I encourage you, however, to read the original article and the ongoing debate in the comments section and decide for yourself.