26 June 2011

Evaluation Literacy: Reading the Systematic Review Tea Leaves

Lots of systematic reviews are emerging from the various 3ie/DFID initiatives. As a researcher, I find them a fantastic resource. As a policymaker, I'm not so sure. It seems to me they require a huge amount of evaluation literacy.

Why? Well, it's not as if we have 20 studies on microfinance that have a similar design and similar outcome indicators and were all run in South Asia, or 20 agricultural interventions that tried to improve nutrition in the same way using the same indicator of nutrition. Getting the centre of gravity of the review is not easy, because the reviews are comparing African apples with Asian oranges and some of the fruits are bigger and more nourishing than others.

In any case, one of my colleagues at IDS, Emilie Wilson attended the big 3ie conference last week in Mexico and this is a brief report from her with links, on evaluation literacy.

Evaluation Literacy by Emilie Wilson, IDS

Donors want to see value for money. Researchers want to apply credible approaches to measuring the impact of development interventions. Politicians want to be re-elected on the back of successful social programmes. A match made in heaven?

Last week, I had the privilege of attending a 3ie conference in Cuernavaca, Mexico, on impact evaluation, entitled “Mind the Gap: from evidence to policy impact”. At IDS, I am lucky to be both working at the coal-face of policy influence using research communication and engaged with action research on how communication of research brings about change.

Wearing both those hats, I engaged in the conference wanting to learn more about the truism that “policy influence is a complex and nonlinear process”. And the beauty of attending these events is that faceless “policymakers” become Felipe Kast, Planning Minister for Chile, Gonzalo Hernandez-Licona, Executive Secretary for the National Council for Evaluation of Social Development Policy (CONEVAL) in Mexico and Ruth Levine from the Hewlett Foundation. Real people with real problems (to resolve).

The conference pre-clinics, thematic and special sessions broadly divided into three areas:

1. how to do impact evaluations (methods)

2. what have impact evaluations already told us (case studies and findings on a wide range of issues including agriculture, health, and social protection)

3. how can we share with those in decision-making positions the news about what works and what doesn’t in development interventions

I focused on this last area, attending two excellent sessions on “Donor priorities for evaluations” and “Perspectives from policymakers”.

Presentations were clear and insightful (see especially “How to influence policy” and “Factors that Help or Hinder the Use of Evidence in Policymaking” by Iqbal Dhaliwal, global head of Policy for the Jameel Poverty Action Lab (J-PAL)), donors and policymakers were frank and humble, and the audience did not shy from asking challenging questions.

Some take-aways for me include:

· building the bridge between evidence and policy is a two-way process: researchers should ensure that evidence is substantial, timely and policy-relevant; policymakers need to be ‘evaluation literate’ and understand the value of using evidence in policy

· there is an important role to be played by ‘intermediaries’ – those who synthesise, repackage, ‘translate’ – making research and evidence relevant and usable beyond academia

We are often told that policymakers are constrained by time and money. Surely this assumption was challenged by the presence of so many at this conference, which required both time (including 30 hour journeys across the world) and money. Perhaps Esther Duflo, who spoke at the opening plenary, was right to talk of “fake urgency” and warn that rushing to gain time would eventually waste time. If we don’t learn lessons now, we’ll make the same mistakes again in the future.

21 June 2011

Distracted from Distraction by Distraction

Is the internet rewiring our brains towards power browsing and away from deep processing of information that "underpins inductive analysis, critical thinking, imagination and reflection?" This is the argument made by Nicholas Carr in his book, "The Shallows: How the internet is changing the way we read, think and remember".

As someone who routinely did his school homework in front of the TV and still likes to write with music blaring in the background, you would not think I would worry much about this. And after reading this book, I'm still not sure if I do.

The book's premise is:

1. reading deeply=thinking deeply

2. the internet is designed to "seize our attention, only to scatter it" and is "by design, an interruption system"

3. due to the neurophysiological plasticity of our brain, it is being shaped by the internet into making us good at being "hunter-gatherers in the electronic forest" rather than "cultivators of personal knowledge" and

4. this is undermining our ability to "make our own associations, draw our own inferences and analogies, and fostering our own ideas".

The book is rather dense (I know, you're thinking I have lost my capacity for "deep processing"--if I ever had it) but it is authoritative, introducing the reader to a lot of behavioural psychology research about how we read web pages (in F-shapes, scanning the first couple of rows of text, dropping down and scanning half a row and then dropping down some more lines only to lose interest) and the impact of longer online stories on our attention spans (on average, for every 100 extra words published we only read 18 of them).

To me, it all seems about balance. I enjoy the hunter gatherer aspects of the net. Yes, I have to be aware of the costs of increased efficiency of searching (e.g. the tendency to be directed to what everyone else is directed to), but there are many areas in which I have no deep personal knowledge but for which I want to get a quick sense of what people I trust are thinking, and the net is invaluable for this. But perhaps I am comfortable with this because I do have areas in which I have deeper personal knowledge--hunger, malnutrition, agriculture, food policy, statistical methods etc. I spent many years pounding through texts and papers while learning my trade in order to do this.

For me, as a father, the most worrying aspect of the internet is the effect it is having on the knowledge aquisition habits of the school students of today. Will they have the opportunities to develop deep personal knowledge of a field? Or will they be consigned to being merely a cadre of superbrowsers?

For those of us in development, this book again brings me back to the need to experiment with and evaluate how research is framed, communicated and used; how we define research quality; whether knowledge that is co-constructed can avoid the trap of being fractured and how the internet can be used to diversify knowledge sources rather than promote bandwaggoning of the research idea du jour.

All in all an interesting read, but one that requires you to turn off your emails and RSS feeds (but not the one for this site).

17 June 2011

Keeping up with the Jones’ (and the Banda’s): Results for Good Change

Yesterday IDS and ODI hosted a meeting with the Secretary of State Andrew Mitchell on the Results for Change agenda, chaired by IDS Board Chair Richard Manning. In all there were 25 people present from a range of research and development partners.

The Results for Change agenda has two key features (a) intensifying the focus on generating development outcomes, and (b) putting more scrutiny on whether this is being done cost-effectively. The agenda is aimed at reassuring the UK electorate that their aid is well spent. But it is also about wanting to support “good change” to make the world a better place. By “good change” I mean change where it is most needed and for whom it is most needed, change that is transforming and enduring and, vitally, change that does no harm.

The meeting focused on how the results agenda could be shaped to support the delivery of this good change. The Secretary of State’s speech at the Royal College of Pathologists demonstrated that he and his team are well aware of the potential disconnects between results and good change. For example, a narrow interpretation of “results” could mean that we evaluate things that are easy to evaluate, diverting attention away from things that are potentially more transforming and enduring but more awkward to evaluate.

My takeaways from the meeting:

1. Innovation. There is a desperate need for innovation in this space. First, on issues. For example, what is the best way of evaluating conflict prevention efforts or the efforts of businesses to achieve development outcomes as well as profits? In other words how do we push and pull good evaluations into the more difficult spaces? Second, how do we get more stakeholder voice into the evaluation process? Will this incentivise learning, make failure harder to ignore and force more listening? Third, how do we communicate results to policymakers, to the UK public (referred to as “Mrs Jones” by several participants) and to citizens in places where DFID works (ditto, “Mrs Banda”). The gap between the technical findings and how they are communicated to these different stakeholders is clearly large.

2. Simplification. Owen Barder noted that the outcomes focus, while a much needed emphasis, was in danger of overburdening organisations using aid. The introduction of a new reporting structure without a paring down of the existing structures risks diverting creative energy away from the achievement of the outcomes. Can the input and output tracking systems be simplified? If not, then outcomes will be at risk.

3. Accountability. How can we track where the evaluations are landing? Despite best intentions, is too much of the evaluative effort going into the short term, service delivery activities and not enough into activities that try to improve systems and rebalance power? Ben Ramalingam shared a useful graph that mapped he results context by (1) the nature of the intervention (simple to complex) and (2) the political context (pro-poor to no pro-poor). If all the evaluations and results are accumulating in the simple intervention/pro-poor space then we are not focusing on the portfolio of potential actions in a sufficiently balanced way.

4. Evidence and decision making. Political opportunity trumps evidence, up to a point. From the practitioners in the room we had some good insights on (a) how evidence is used to make the best decisions within the political space that exists (e.g. if political space exists for vaccinations, use evidence to make sure the right vaccinations are delivered to those most in need in ways that promote greatest spillovers) and (b) on how evidence can shape the political space in the medium run (e.g. does the evidence justify the political space that HIV/AIDS commands?).

5. More Evidence. We agreed that the evidence base in development is weak in many areas (don’t forget, we are researchers)--at least for the more RCT type evidence. But we skirted the issue of what constitutes credible evidence, noting that in some cases RCTs will be the “gold standard”, but in many other cases there will be different methods and blends that will be labelled as gold standard. We heard that DFID has commissioned a review of methods which can offer levels of rigour in their context that are similar to the potential rigour RCTs offer in theirs.

6. Need for country led accountability. We did not spend enough time on this. Who are the results for? And who generates them? These are key factors that will shape which policies and actions are evaluated, the definitions about the quality and weight of evidence and, crucially, the incentives to learning from the findings.

It’s clear to me that the “results for change” agenda has great potential. Those of us in the research community must play our part if this potential is to be realised. We must be at the forefront of this agenda, working with aid agencies and Governments to make the results work for “good change”. In doing this, we may well have to change ourselves--to learn from evaluation methods outside of development, to try to evaluate the seemingly unevaluable and to invest more in understanding the political processes within which decisions are made and communicated.

To get a wider range of perspectives from this meeting, I encourage you to visit blog entries from Rosalind Eyben and Simon Maxwell, both of whom who were active participants.