By not measuring impact, NGOs are abusing their power

That’s the title of a recent article over on The Guardian’s “Secret aid worker” blog.

Needless to say, endless evaluation is an impossible idealism. Funding is tight so why should NGOs invest precious revenue in costly impact evaluation when nobody is paying attention and flimsy statistics can be passed off as watertight findings? Eloquent websites and lengthy annual reports allude to responsible practices but often offer little substance. Many NGOs state a focus on “breaking the cycle of poverty” and report “a lasting impact” on families’ lives without any evidence to back up such claims.

Other NGOs may assert that X% of families have crossed the poverty line over the past couple of years. Often these figures are based on passing the World Bank’s $1.90-a-day threshold or using tools such as the Progress Out of Poverty index (PPI). However, neither NGOs, nor the families they serve, exist in a vacuum. Yes, these families may no longer technically be impoverished, but attributing an extra dollar a day to your NGO’s recent microloan or an extra couple of points on the PPI to your workshops on family planning is inaccurate and unprofessional. Correlation is not causality.

Then there’s this:

Frustratingly, while our own evaluation gap grows, more focus is put on the experience of volunteers. Interestingly, they are surveyed at the end of their placement, unlike the families who leave our programme. Positive feedback provides a flashy quote to excite future volunteers, while negative feedback is largely ignored.

There is no substitute for a robust impact evaluation of your programme. This does not have to be done annually; mixed-methodology research, where you compare a group working with your NGO with a group that is not, can be achieved in a relatively small time frame.

Finally:

None of this is to question the goodwill of many NGOs but goodwill is not, nor will ever be, sufficient. NGOs that are unable or unwilling to provide strong evidence of the impact they are having are, at best, a considerable waste of time and money. What seems a lot more certain is that many NGOs are continuing to provide noble career paths and selfless volunteer placements for the more fortunate, while simultaneously servicing the local population with untested and meagre programmes.

Any change of approach will have to originate from within the international NGO community, otherwise for as long as this status quo rumbles on, so will the ugly questions of neo-colonialism and white privilege that often circulate around this kind of work.

Yikes! As Paul Niehaus said on Twitter, “This took courage to write…“.

But, I have to agree with the point the author is trying to make. The majority of the international NGO industry is built and supported by a system that is devoid of rigorous evaluation. It’s largely predicated on feelings rather than facts, simpleton good intentions, and hubris rather than humility. This is not to say that every NGO or individual who works in the aid and development industry doesn’t care if their work is worth the time or money. I’ve worked with several great individuals who were willing to put the value of their work on the line by measuring impact.

Also, there is change in the wind. A new bill is actually moving through congress (!) that could (maybe) change all this for the better. The Foreign Aid Transparency and Accountability Act would drastically increase the standard of evaluation within the aid industry. (Well, just within USAID, but they are the largest aid and development donor in the world, so it’s a good place to start.)

Additionally, yours truly will be spending this upcoming summer in Washington DC at the U.S. Global Development Lab within USAID working with the Office of Evaluation and Impact Assessment. I’m not sure I’ll have a huge impact, but I’m determined to make it worth the time and money.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s