Bad Research is Bad Research.

I want to highlight a recent post by AidLeap on Why Programme Monitoring is so Bad. It brings up an important point and sheds light on an issue I’ve personally experienced.

A manager in a field programme that I evaluated recently showed me the glowing findings from his latest monitoring trip – based on a total sample size of two farmers. When I queried the small sample size, he looked shocked that I was asking. “It’s OK”, he explained, “We’re not aiming for scientific rigour in our monitoring.”

I regularly hear variants of this phrase, ranging from the whiny (“We’re not trying to prove anything”), to the pseudo-scientific (“We don’t need to achieve 95% confidence level!”) It’s typically used as an excuse for poor monitoring practices; justifying anything from miniscule samples, to biased questions, to only interviewing male community leaders.

I’ll second hearing the statement “We don’t need to achieve 95% confidence level”. I’ve also heard “Well, when we really want to do a deeper evaluation we’ll use a comparison group”.

There seems to be a sort of bias with folks who work for organizations and agencies that actually do stuff against spending resources (money and time mostly) finding out if their work actually works. In principle this has changed as even the smallest of organizations have a “Data Analysis Intern”or a “M&E Fellow”. In practice, however this monitoring and evaluation is typically pretty terrible.

This is likely due to a lack of rigorous (usually double-blind) peer-review that is the norm in academia. As well as the standards and goals of the various institutions. But that doesn’t change the reality.

This is unfortunate because if we think the work of these organizations is worth doing, then it is certainly worth doing well. Most of the time we just think the work is worth doing… and then stop there. Nobody tests whether the reality actually matches the theory.

Evaluating the effectiveness of something, especially when it comes to human livelihoods is important, no matter who you are. As such, performing evaluations as a veteran practitioner, a data analysis intern, or an M&E fellow is no different than performing evaluations as a professional scientist (economist, sociologist, anthropologist, agronomist, ect.) or tenured university professor. Bad research is just bad research. Full stop.

[If you’ve read this far, and haven’t already please read the original AidLeap blog post.]