26 June 2012

Impact Horses for Impact Courses: Two Helpful New Reviews


As the Trini Lopez song goes, "If I had a hammer, I'd hammer in the morning, I'd hammer in the evening, all over this land, I'd hammer out danger, I'd hammer out a warning, I'd hammer out the love between my brothers and my sisters, all over this land".

For "hammer" read "my favourite evaluation method".

For "brothers and sisters" read, say,  "randomistas" and "realistas" or "propensitistas" and "participatoristas".

In other words, how do we live up to the ideal of the diagram above (from the Stern et. al. paper reviewed below), whereby we let: (1) the question, (2) the available evaluation tools and (3) the attributes of the programme being evaluated, co-determine the impact design?

One of the ways of moving towards this ideal is knowing more about the "available designs".  Let's face it, we all learn--at most--one or two of these techniques and the effort of learning more is large.

This lack of familiarity with a broad range of techniques makes us wary of ones we do not know well.  Every evaluation problem looks like the nail that our own particular hammer is well suited to hit.

Two new publications set out to help us expand the set of available impact designs and to help identify some common traits across them :  


The goal of the Stern et. al. report:  

"Up to now most investment in IE has gone into a narrow range of mainly experimental and statistical methods and designs that according to the study’s Terms of Reference, DFID has found are only applicable to a small proportion of their current programme portfolio. This study is intended to broaden that range and open up complex and difficult to evaluate programmes to the possibility of IE."

The goal of the White and Phillips report:

"This increased demand for results has resulted in a growth in large n impact evaluations involving tests of statistical significance between outcomes for treatment and comparison groups. However, this type of statistical analysis is not suitable for all evaluations. When there are insufficient units of assignment to conduct tests of statistical significance, then small n approaches are the best available option. However, while there is considerable consensus among large n researchers concerning what constitutes a valid approach, the same is not true for small n approaches. In fact, there is a plethora of possible approaches available to an evaluator, but little consensus concerning what approach, methodology, or methods are most suitable for an impact evaluation, and what constitutes valid causal evidence. This paper seeks to advance this discussion."

Both reports are very long and technical, so I won't go into much detail here.  But they are both excellent and well worth reading (and will, I suspect, prove most useful as reference works).

Some takeaways:

* The field of development evaluation is very technical.  It is not something for dilettantes: if you are serious about it, work with experts

* There is a wide range of approaches out there:  

The Stern paper outlines 5 categories: experimental, statistical, theory based, case based and participatory. 

The White and Phillips paper discusses only "small n" approaches (where there are not many units of observation): 
  • Group 1 approaches, where establishing causality is a key goal:  Realist Evaluations where mechanisms x context=outcomes; General Elimination Methodology which draws up lists of possible causes and then eliminates the least plausible; Process Tracing (as pioneered by Oxfam); Contribution Analysis-building the most plausible and credible contribution story. 
  • Group 2 approaches are less concerned with establishing cause and effect and are more interested in finding out about the drivers of positive change and the role the programme is playing. Examples here include: Most Significant Change (strength is in identifying diverse or unexpected changes); Success Case Method (which is interested in exploring particularly successful or unsuccessful cases); Outcome Mapping (as pioneered by IDRC -- tracking steps between programme activities and programme outcomes); MAPP, a participatory approach (with a quantitative step) to understanding the perceived influence of a development project not he daily life of the population)


* Picking the right approach requires a lot of discipline--when you are writing an evaluation proposal to, say, a 20 day timeline without detailed knowledge of how a programme works or detailed contextual evidence, and without knowledge of the different methods available, you are going to fall back on the tried and trusted (but probably not most appropriate) methods.  Funders of evaluations need to give the evaluators more time at the beginning of a project

* "Mixed methods" is not just about mixing quantitative and qualitative methods, it is about mixing approaches too (say realist and experimental)

* There is still some disagreement on the usefulness of the experimental and statistical approaches to complex interventions--the Stern paper implying that the other 3 approaches are more suited (theory based, case based, participatory) while the White and Phillips paper arguing that it depends on the complexity and it depends on whether you want to assess the overall impact, because the complexity may be too much for any method

* Biases are everywhere.  The White and Phillips paper is particularly strong on this--pointing out the biases in large-n approaches, but also the checks and balances that are in place (which can be verified by others), while arguing that these may not be in place, or so easily checked by others, in the small-n methods. 

Conclusion? Not only do we need a lot of tools in the toolbox, but we need lots of skilled craftswomen and men to use them, with programme implementers and funders giving them enough time and support to get the evaluation done properly.   These two papers set out the magnitude of the task but also help us move to the  ideal solution.  


23 June 2012

Is Africa's Medium Term Future Really Urban? New Evidence

The recent report from UNICEF "Children in an Urban World" has much to commend it.

Urban populations are growing, more children are being brought up in urban areas, and we in the development research community are having to play catch up with this (based on my highly imperfect knowledge of the number of research projects, research funding streams and calls, journal articles etc.).

The UNICEF report confirms much of what we know: on average urban areas are better off (for example the rates of child underweight prevalences are 1.5 times higher in rural areas), and yet the urban areas contain pockets (large ones) of some of the most destitute and poorest people on the planet.

What the UNICEF report does, being a UN report, is use UN urban population data projections.  For example, the inside cover (above) has a graphic that shows  Nigeria with 50% of its population living in urban areas.

But a new paper from Deborah Potts at Kings College (in World Development and available here) points out that the UN projections are based on trends in the 60s and 70s when many African countries were rapidly urbanising (that is, their urban populations were growing fast enough to increase the proportion of the population living in urban areas).  Once we examine more recent data, we learn that these proportions are way off.

Potts' paper spends much of its time in a forensic analysis of old and new data from Nigeria.  She zeros in on an initiative called Africapolis (supported by the French development agency AfD), which cross-references census data, satellite images and secondary data sources (sometimes very very micro) to conclude that 30% of Nigeria's population is urban.  And, more extraordinarily, the percentage will only increase to 31% by 2020.

30% versus 50%.  And Nigeria is by far the most populous country in Sub-Saharan Africa.  This overestimation by UN data trends is constant with new census data emerging from other West African countries, she says.

None of this is to say that urban populations in many African countries are not growing--they are--simply that they may not be growing faster than overall population.

Why has African urbanisation slowed down? Potts' hypothesises that economic growth gains in Africa have not been broadly based in urban areas and that this is slowing  rural to urban migration.

Population in African countries might not be urbanising as quickly as conventional wisdom claims, but my guess is that the focus of the development research community still has some urbanising to do.

13 June 2012

Celebrity Squares: What is the value of the right profile?

The Guardian has a nice piece by Marina Hyde on celebrities and charitable giving based on a recent report from the UK Public Opinion Monitor (UKPOM), co-hosted by IDS.

Ms. Hyde says the UKPOM findings show that "most people claim not to be swayed by celebrity-fronted campaigns, but they do think that other people are swayed by them. Which suggests that celebrity campaigns are popularly believed to be popular – but falsely so."

This is indeed what the data show but they also show that, while less popular than we would think, celebrity endorsements are effective in some domains. For example, 14 out of 100 people report they give more money to a cause than they would have without the celebrity endorsement. That seems like a very high number to me.

It would be interesting to compartmentalise this result. Is this 14% number higher for wealthier people? Older people? Poorer people? Particular ethnic groups? A recent econometric study from the US found that the donations of certain groups responded much more highly to celebrity endorsements than others.

The article by Ms Hyde then goes on to say that the UKPOM data show that Kim Kardashian is the best celebrity to help a child in extreme poverty (although I could not find this in the report) and she "We're all poorer when we seek Kim Kardashian's take on poverty."

OK, first of all I did not know who Kim Kardashian was, and after 2 minutes of internet searching I can understand why she is not exactly a role model. This is certainly an example of Maureen Dowd's quote (she of the New York Times) "Celebrity distorts democracy by giving the rich, beautiful, and famous more authority than they deserve."

But some celebrities deserve respect for their charity work, surely. I'm sure there are many celebrities who know that celebrity is luck-fuelled and needs to be worn lightly but used weightily. They use their celebrity for the charity in an intelligent way--my sense is that there is not much box office in it (although that could also be researched).

Perhaps the main worry about celebrity advocates is that they might endorse the status quo, or not sufficiently challenge the systems that generate the need for charity giving in the first place. Again, I think it depends on the celebrity. The singer Billy Bragg (he of "A New England") certainly does not fall into this category.

So the answer seems to be: select your celebrity as carefully as you select your CEO.

The irony is that the UKPOM itself needs some help in staying funded. We are running it on fumes.

Would I mind if Billy Bragg gave UKPOM an endorsement? No. (Although I would need to do my due diligence!)

11 June 2012

Rio-rdering Priorities at Rio+20: Time to Face up to the Politics of it all


I'm 53 this week, and this prompted a game with my kids (11 and 10) in which they told me what they thought their world would be like when they were 53 (the years 2053-54).   

We talked about what they might end up doing, where they would live and, of course, hovercars.  

I couldn't help wondering about what the air, oceans and rivers would be like.  


The Rio +20 conference is only a week or so away, and more global attention than usual is focused on what the air, oceans and rivers will be like in, say, the year 2053-54.

It's hard to know what to make of this temporary defeat of our chronic attention deficit on sustainable development.  Undoubtedly it's an opportunity for all of us to rededicate attention how we achieve three things simultaneously: 

(1) complete the MDG agenda, 
(2) bring the planetary management agenda to maturation, and 
(3) continue to pursue national interests

If you think of these three as circles, the overlaps are not necessarily large.  For example, green growth does not have to be poverty reducing growth (indeed, some G77 countries think it is rich country code for slow growth); poverty reduction might entail high levels of GHG emissions, and national interests such as energy subsidies (which favour the richer groups who use more energy) serve neither MDG nor planetary management goals.  

But the signs are not great that Rio+20 will be a watershed moment:

* as of last week, 70 of the 329 paragraphs of the revised zero draft were agreed, with 259 containing bracketed text--historically this is a very low proportion of agreed on text only 10 days away.. 
* confidence in intergovernmental collaboration is very low in the wake of the last 3 climate meetings and the eurozone crisis 
* I am told there is a sense among those within the negotiating processes that Environmental ministries have struggled to bring Finance ministries along with them and are not terribly well practised in the art and craft of international relations, thus slowing down agreements..

But there are some causes for optimism

* national policies still really matter because: (1) many environmental issues are local and national (e.g. water and air pollution), and (2) many actions driving global environmental externalities are a result of decisions taken by national governments, e.g fossil fuel subsidies which dwarf ODA and subsidies of other energy sources.  This means that national and local civil society has a good chance to influence national and global policies--this realisation is important in the wake of events like Copenhagen that give us little faith in the ability of world leaders to forge agreements. 

* there is a recognition that metrics matter: (1) several African countries have signed up to an initiative to include natural resources in their national accounts, (2) the Sustainable Development Goals will force everyone to think about what dimensions we really want to track and how we would do that and how does that link to the MDGs, and (3) for the first time business performance could be measured by, say, a beefed up Global Reporting Initiative

* there is a realisation that the UN environmental norm-setting agenda is weak and that there is a need for something like a World Environmental Organisation (WEO) or a substantial strengthening of UNEP, the UN's Environmental Programme.  Set against this, there is the reality that WHO and FAO are far from perfect, although part of this is historical baggage that any WEO would be free from. 

But if the optimism is to be converted into achievement beyond Rio, then the following need to happen:

* Politics needs to take centre stage.  This will appal some of the scientists who already think science is not getting a look in at Rio. But think about it: sustainable development involves tradeoffs between current and future generations, between countries with a lot of growth under their belts and those who are just getting a taste for it, between those who will be winners, losers or neutral from climate change, between different government departments and between national sovereignty and international common good. (As an example, the politics over where a new WEO would sit--France/Germany or Nairobi--and who would run it have threatened to kill off the idea.) All of this sounds like a lot of politics to understand and negotiate and we better bring it out into the open.  My IDS colleagues Melissa Leach and Matthew Lockwood have written interesting pieces on this issue. 

* More lateral thinking to understand multilateral inertia on the environment.  What can we learn from other multilateral efforts?  For example, can we learn from the 1987 Montreal Protocol which has led to a near closure of the ozone hole? A treaty was agreed because the evidence of the threat was relatively uncontested, the nature of the threat was tangible for most people--especially the richer countries (cancer), the chemical and industrial changes were relatively straightforward, and no great changes in lifestyle were required (unless you really have problems with roll on deodorants).  Pretty much none of these conditions hold for global warming, so we know we have a big challenge on our hands, but can we learn from any of the ozone strategies?  Can we learn from the trade failures (see an interesting article from Patrick Low at WTO, an IDS alumnus)?  Or on financial regulation?  Or on nuclear proliferation?  The environment is not the only component of sustainability.    

* To ensure the impact and value for money movements do not divert resources away from research for the kinds of institutional and governance interventions that will be required to incentivise an alignment of the planetary, MDG and pure national interest agendas.  How would you justify , ex-ante, the development of new metrics based on value for money?  How would you do an impact evaluation ex-post?  Not impossible, but as Kahneman says in "Thinking Fast and Slow", when confronted with a really difficult question we tend to substitute an easier one for it. 

Finally, we should not forget about people (see this nice piece from Camilla Toulmin).  It will be a real challenge to remember human welfare when we start talking about energy targets, ocean targets, city targets and atmospheric targets.  

The Rio Earth Summit in 1992 was fuelled by the optimism of the post Cold War era.  The 2012 Earth Summit has optimism tempered by the current global economic slowdown and the nationalism that fuels.  

But if the Rio+20 meetings can:

* fire up national movements to nudge national consensus positions towards more responsible resource use
* put in place a practical but inclusive process for developing a realistic set of SGDs by 2016 
* strengthen the UN to set and monitor norms, and 
* generate some interesting ideas about financing

then maybe, just maybe, the Earth Summit of 2022 will be able to capture this consensus to deliver on definitive actions that make 2053 liveable and sustainable.   

With hovercars (green, naturally).  

05 June 2012

What would you have asked CM Patnaik?


One of the fantastic collaborators I work with, Biraj Swain, is a citizen of Odisha. She was due to be at IDS on May 28th when CM Patnaik (pictured) was originally scheduled to give a talk in Brighton (see previous blog).  I invited her to do a guest blog about her reflections on the CM's performance to date and what she would have liked to talk to him about.  This is what she wrote.

“So Naveen Patnaik, the Chief Minister of Odisha, was going to be at IDS on 28th May giving a seminar and I had a full day of meetings scheduled there. What are the chances of running into one's own home state Chief Minister one whole continent and ocean away in Sussex, with probably a chance for a closer tete a tete?

Alas, Mr Patnaik did not come and perhaps all the potential awkward questions and adverse publicity at home in the build-up got to him as did the unrest in his own government while he was absent.  Missed chance? Perhaps. 

If I had met him I would have congratulated him for universalising the Public Distribution System in the poorest districts of Koraput Bolangir and Kalahandi. It came with electoral returns of course but then aren't welfare states and votes related in a democracy?

He also presents an acceptable secular face for the third front against Congress and the BJP. That the Left leaders bee-lined to him in 2009 was a proud moment for many Oriyas like me who are used to having our leaders on the fringes of national politics. Coalition mantra? So be it.

Having said that, two major mining leases in Odisha (Vedanta and POSCO) have been turned down by a combination of people’s movements and judicial scrutiny of the rigor of the due diligence process.

POSCO was the largest single FDI in India, with the Odisha government not only turning over every single Environment Impact Assessment (EIA) rule in the book but also risking public outrage by locking itself eye-ball to eye-ball against the resisting locals. Given that this EIA was struck down by the National Green Tribunal two months ago leaves a lot on the Chief Minister’s agenda. If I had met him I would have brought the following issues to his attention:

1.    What happened to the much feted Right to Information at the ground zero of base metal in Keonjhar? Transparency quo vadis?
2.    For that matter, what about the Gram Sabha consent to land acquisition or the principle of Free Prior Informed Consent?
3.    The last mile delivery of progressive social protection and welfare programmes has been a challenge all over India, but more so in Odisha (NREGA has had barely 45% uptake).
4.    How can a rice surplus district like Bolangir (per WFP Food Security Atlas) have year-on-year starvation deaths?
5.    Disaster relief and preparedness--the primary agenda on which his government won elections post 1999 super-cyclone--was found wanting in 2011 floods where over 10 districts were submerged for over three months.

But credit where it is due--the CM’s stewardship has brought in a fair amount of participation and governance and perhaps a leader of his stature (with an absolute majority) and acceptance (in Odisha and beyond) has a unique opportunity to fix many of the wrongs. Breaking the bureaucracy stronghold in the state, making the state machinery more accessible and bringing in more public spirited pluralism to his policy agenda will go a long way, as will finite allocation of public natural resources (a.k.a minerals).

Perhaps the CM's absence from IDS was a bigger loss for him than for those gearing up with awkward questions. Because a free and frank conversation, based on the difficult legacy he has legitimately secured, would surely have led him to some new maverick thoughts and solutions!”

Biraj Swain is a citizen of Odisha and a keen student of governance and citizens’ engagement and has a body of work in Odisha too other than South Asia and East Africa. She works with Oxfam India but her views in this blog are personal.

03 June 2012

The Odisha Chief Minister does not visit IDS: A Tale of Politics, Rights and Nutrition


One of the more interesting aspects of my job is having to deal with situations like the visit of Naveen Patnaik, the Chief Minister (CM) of Odisha, the eleventh most populous state in India.

A recent case study that one of our researchers at IDS had undertaken concluded that Patnaik’s government was instrumental in accelerating malnutrition reduction in the state, in part due to the CM’s own leadership.  

So when we were told by DFID that he wanted to visit us at IDS on his UK trip, I thought that would be an interesting opportunity to find out why and how nutrition had risen up the state’s political agenda.

Well that was not the only uprising I was to find out about. Some of our staff and students quickly told me about their concerns at the Odisha government’s actions with respect to the displacement of ethnic minorities from business mining activities and the attention given to incidents involving the persecution of religious minorities.

I checked with two of the leading UK based human rights groups, one said that the CM had done several good things in the past few years, the other said they had concerns.  I then received a long letter from Survival International stating their concerns.  Then came the enquiries from the Indian media.  

The CM’s office were informed that despite all of this they were still welcome to come to IDS.  They would receive tough but respectful questions from our staff and students and I pointed out that there might be protesters outside IDS from groups like Survival.   I was informed a few days later that the schedule had changed and that the trip was off.  I don’t know what the reason was, but the prospect of having to wade through potential protesters cannot have been too appealing. 

I think it is a shame the seminar at IDS did not take place.  The CM is an elected head of state who won an overwhelming majority of constituencies (109 out of 147) in the latest elections in 2009.  He has overseen a rapid decline in poverty and malnutrition rates and has spent a lot of state resources on interventions designed to support those declines.  On the other hand, university campuses are places where free speech is sacrosanct and politicians in democracies have an obligation to defend their government’s actions and they are usually very adept at doing so. 

So, a public opportunity to hear about the politics of nutrition and the politics of development was missed.  I was invited to meet with the CM, some of his cabinet and some of his leading civil service administrators at the Indian High Commission.  I did this, joined by two of my colleagues who work on nutrition governance.  We spent an hour with the CM and his team, talking about the experiences of getting nutrition higher up the agenda.  I asked the CM for his response to the accusations levelled at him. He confidently defended his state’s actions. (He had a busy week--a day later he had to deal with an attempted coup.)

Lessons for me?  Don’t leave due diligence to a third party.  I gave too much weight to DFID’s implicit endorsement of the CM’s visit (he met with Andrew Mitchell the day after he met me). 

Frustration for me?  Not knowing what to believe and how to weigh it. 

The Human Rights Watch chapter on India in their 2011 World Report only mentions Orissa once, and with a positive development:

“a legislator from the ultra-nationalist Bharatiya Janata Party was convicted in June 2010 for his role in violence against Christians in Orissa in 2008 that left at least 40 people dead and thousands displaced when a Hindu mob attacked Christians. In August, 16 others were sentenced to three years in prison for their role in the violence.”

The most recent entry from Amnesty International on Odisha on their website was from July 2011 and was also a positive response to a negative development:

“The High Court of Orissa on Tuesday upheld the Indian government's decision made in August 2010, to reject Vedanta Aluminium's plans for the six-fold expansion of the Lanjigarh refinery, finding that the project violated the country’s environmental laws.”

Freedoms, rights and material improvements—they are all vital to development, and as experiences from Ethiopia, China and Vietnam show, they don’t all necessarily move in the same direction and at the same pace. Development is complex and multifaceted.  

The nutrition story in Odisha sounds like a good case for further in depth IDS study.  So too do the allegations into rights violations and what the government says it has done in response. 

01 June 2012

Ian Scoones on "The Geeks Fight Back"

A very interesting blog in the Huffington Post today by IDS' Ian Scoones about the relationship between scientists and the general public.

His article is mostly in reaction to the Geek Manifesto from the Wellcome Trust's head of communications, Mark Henderson.

Ian argues that the debate this has sparked runs the risk of suggesting that we should leave it to the scientists to determine the role of science in society.  He notes that dissent, debate and deliberation in science, even (especially perhaps) from non-scientists, is important to support.  I very much support this view.

He also endorses the famous Churchill line that "scientists should be on tap but not on top".  I don't agree with this perspective--there is a big space between "on tap" and "on top".  Scientists are citizens too.

The key is, as he says, to set up processes that ensure inclusive and balanced debates about risk, regulation and benefits (see the STEPS Manifesto).

Ian's blog is an important contribution to the debate.