26 June 2010

Aid in the UK: Ring Fenced or Wrung Out? Expanding the Evaluability of Aid

With most government departments in the UK facing 20-25% cuts over the next year or so, DFID employees are sheltering behind a funding ringed fence. They must be pinching themselves at their good fortune. But their privileged position also forces them to be as penny-pinching as possible--if not, other departments will do the pinching.

But how sturdy is the fence and are there any gaps through which resources can leak out?

The fence is coming under increased fire. The recent article in the Daily Mail was particularly scathing. These attacks, in turn, generate stock responses about aid's all-conquering properties.

In an era where the aid debate risks becoming ever more polarised we are in need of nuance. But for busy UK policymakers nuance is about as welcome as England having to face Germany in the round of 16 (I suspect this may not hold for German policymakers).

For most aid policymakers the three tests for aid spending seem to be:
  • can aid deliver the output or outcome?
  • can these outputs or outcomes make a long term difference?
  • can impact be demonstrated in a relatively short time period?
You can see the potential contradictions. It is difficult to show the short term impact of things that can make a long term difference (e.g. investments in innovation in science and technology). Things that aid can deliver (e.g. more children in school) may not make a long term difference (e.g. if class sizes are so large that the average student's ability to learn diminishes). Things that aid can deliver (e.g. stronger health systems) can present problems for the demonstration of impact (e.g. in terms of lives saved).

What can the development community do to make these contradictions more visible but less severe?

One key contribution is to inject some nuance into the impact evaluation debate. There are 5 dimensions of this debate that are woefully under-represented. A strengthening of them would make it easier for policymakers to contemplate more nuance:

Question the questions
The effort required to deliver a convincing answer often dwarfs the effort put into thinking about whether it was the right question to answer in the first place (e.g. why has no-one asked whether conditional cash transfers make sense in sub-Saharan Africa? Most of the effort has been spent on answering if cash transfers have an impact on hunger. Of course they do.). This means consulting with a wide range of stakeholders about the questions worth asking (see "this book fills a much needed gap").

Deepen the evaluation story
The "randomistas" and even the quasi-experimentalists often answer the question "does it have an impact?", sometimes "for who?", but rarely "why?". This requires innovative issue-driven blends of quant and qual.

Broaden the portfolio of what an be evaluated
It will always be more challenging to evaluate policy changes that generate consequences that are indirect and lagged, for which indicators are not yet developed, and for which all the pathways of change are not yet fully understood. But we must help to expand the set of things that are "evaluable" by developing methods to consider these dimensions.
Refine impact
Your idea of success is probably not the same as mine. Add in experiential, cultural and value differences and the gap may widen. Getting multiple views of success from different stakeholders will help to home in on what really counts and help uncover unintended consequence land mines further down the road.
Increase the genuine portability of evaluations
Most methodologically rigorous analyses--whether quant/qual or some blend--have, by definition, high degrees of internal validity. They are fit for purpose in the context in which they are applied. But by building in more variation and diversity - either within the context or across it by, say, systematic case study meta analyses--we can make plausible guesses about how portable an intervention is and the associated risks of assuming it is.
In short the development research community must nuance the impact debate, not disown it.
If we do the latter, then aid will only be used for things where it is easiest to demonstrate impact and the rest will be wrung out of DFID, despite the ringed fence.

We need to expand the radius of evaluability if we are to help protect the parts of the aid spend that may do the most good.