09 May 2012

MVP Impacts? New Paper in The Lancet

A new paper from the Millennium Villages (MV) Project team (pdf) was released today in the Lancet. The paper's conclusion?


"An integrated multisector approach for addressing the MDGs can lead to rapid improvement in child survival in rural sub-Saharan Africa."

Is this impact?

The paper introduces comparison villages in year 3, so MV sites can be contrasted with these comparison sites. This is a move in the right direction, but because the comparison sites are introduced so late on in the project we don't really know how comparable they are and they rely on recall data. 

The assessment of the comparability of these villages (on observable characteristics) is based on 3 sets of data:

(a) village level infrastructure at year 0, but recorded at year 3 for the comparison villages (I think, but the paper is not clear). These are relatively fixed features of villages and for the features that vary over time should be able to be derived from recall

(b) household characteristics at year 3 (collected in year 3 for both sets of villages)

(c) household characteristics at year 0, collected at year 0 for the MV villages, but at year 3 for the comparison villages. This is more problematic--it is is difficult, but not impossible, to construct accurate recall data at the household level over a 3 year period

The mean values of the three sets of variables are pretty comparable across the two sets of villages--in both time periods--so that is somewhat reassuring, but the comparison sites are not control sites (for one thing, we cannot say that the unobservable characteristics have been equally allocated between the two). But it is progress.

The crux of the paper are the results in Table 2. This table makes three sets of comparisons on outcome variables:

(1) for 17 outcome variables, the changes within MV sites between years 0 and 3 are assessed. In 13 out of 17 comparisons, there has been a statistically significant improvement (5%) in the outcome indicator. This could be due to the MV or to some other effect. So the results are neither here nor there. What we are really interested in is the comparison of improvements in MV sites with improvements (or otherwise) in the comparison sites

(2) the second set of comparisons is between the MV sites in year 3 and the comparison sites in year 3.  For these comparisons to isolate impact, the MV and comparison villages need to be comparable at year 0 in observable AND unobservable characteristics. We don't know whether the latter holds and we only have a partial and somewhat fragile perspective on the comparability of the former. Of the 12 comparisons of outcomes between year 3 in MV and year 3 in comparison sites, 3 are substantially improved at 5% significance levels (bednet use, malaria prevalence and access to improved sanitation) 

(3) the third and final comparison is a "kind of" difference-in-difference comparison: what is the difference in the change in MV sites between years 0 and 3 the change in comparison sites between years 0 and 3?  It is "kind of" because the sites were not randomly allocated between MV and comparison at year 0, and the year 0 data for comparisons sites are  collected in year 3. Of the 6 outcome comparisons made here, 2 are substantial improvements that are significant at the 5% level (skilled birth attendance which increased by 24.7 percentage points in MV and by 12.7 percentage points in comparison sites; and mortality rate in children younger than 5 years of age (deaths per 1000 births): in MV sites this number decreased by 24.6 and in the comparison sites it actually increased by 5.9).

So?

  • 5 of the 18 outcome comparisons between MV and comparison sites show significant improvements.  This does not seem like a very high number. Why so few?
  • But some outcomes are more important than others, and a big outcome is the decline in child deaths, and this was in one of the "kind of" difference-in-difference comparisons.
  • If we find the cost data credible, is this a good enough return for a more than quadrupling ($27 to $116 per person per year) of spending over the three years on these multi sector interventions? I have no idea--it would have been really helpful if, for example, the child mortality numbers had been converted into cost per death averted so we could benchmark against other interventions.
  • The more interesting comparisons will be the changes in outcome variables between year 6 (say) and year 3 between the MV and comparison sites, because although the nonrandom allocation of sites problem remains, at  least the observations in years 3 and 6 in MV and comparison sites will not be based on 3 year recalls.
  • The key thing remains sustainability--is there life beyond proof of concept or is the MV doomed to the start-up "valley of death"? To assess this it would be good to have a better sense of what is in and what is out when the cost data were assembled.

Impact? Getting warmer, but more work to do.

Sustainability? This needs some serious research. 

Verdict on MVs? Still out.

6 comments:

Anonymous said...

It would be good, in the MVs, for the authors to collect year 0 data in year 3 as well, so as to see how much this deviates from year 0 data collected in year 0. That would give an idea as to how reasonable (or otherwise) the collection of retrospective data in the comparison villages was

Lawrence Haddad said...

Dear Anon, yes, agree, a very good point... it would also be good if the paper reported on 3 year recall accuracy from other studies...

Lee Crawfurd said...

Martin Ravallion at the WB has a recent paper on this and finds recall data on consumption to be pretty unreliable: http://elibrary.worldbank.org/content/workingpaper/10.1596/1813-9450-5983

Lawrence Haddad said...

Thanks Lee.. I always have a lot of time for Martin's work..

Lawrence Haddad said...

Some serious concerns about the quality of the science in the Lancet article, raised by Gabriel Demombynes and Espen Prydz. Much of the treatment effect claimed by that article arises from an arithmetic error:

http://blogs.worldbank.org/impactevaluations/the-millennium-villages-project-impacts-on-child-mortality

They argue that the annual decline in child mortality decline in MV sites was calculated incorrectly and should be 5.9% not 7.8% as claimed in paper (the authors used the wrong denominator--3 years instead of the 4 which they should have used).

They also argue that if you use the right DHS data sets for comparison (ones that better correspond to the MV treatment spell) then child mortality declines in the MV sites are slower than the national rates of decline and that if you look at the rural rates of decline front he DHS surveys these are even more rapid than the MV sites (which are rural).

These arguments seem pretty convincing to me.

One wonders how this got through the journal peer review process...

dustyjulian said...

You finished on good point Lawrence - does the change in these places last?