"An integrated multisector approach for addressing the MDGs can lead to rapid improvement in child survival in rural sub-Saharan Africa."
Is this impact?
The paper introduces comparison villages in year 3, so MV sites can be contrasted with these comparison sites. This is a move in the right direction, but because the comparison sites are introduced so late on in the project we don't really know how comparable they are and they rely on recall data.
The assessment of the comparability of these villages (on observable characteristics) is based on 3 sets of data:
(a) village level infrastructure at year 0, but recorded at year 3 for the comparison villages (I think, but the paper is not clear). These are relatively fixed features of villages and for the features that vary over time should be able to be derived from recall
(b) household characteristics at year 3 (collected in year 3 for both sets of villages)
(c) household characteristics at year 0, collected at year 0 for the MV villages, but at year 3 for the comparison villages. This is more problematic--it is is difficult, but not impossible, to construct accurate recall data at the household level over a 3 year period
The mean values of the three sets of variables are pretty comparable across the two sets of villages--in both time periods--so that is somewhat reassuring, but the comparison sites are not control sites (for one thing, we cannot say that the unobservable characteristics have been equally allocated between the two). But it is progress.
The crux of the paper are the results in Table 2. This table makes three sets of comparisons on outcome variables:
(1) for 17 outcome variables, the changes within MV sites between years 0 and 3 are assessed. In 13 out of 17 comparisons, there has been a statistically significant improvement (5%) in the outcome indicator. This could be due to the MV or to some other effect. So the results are neither here nor there. What we are really interested in is the comparison of improvements in MV sites with improvements (or otherwise) in the comparison sites
(2) the second set of comparisons is between the MV sites in year 3 and the comparison sites in year 3. For these comparisons to isolate impact, the MV and comparison villages need to be comparable at year 0 in observable AND unobservable characteristics. We don't know whether the latter holds and we only have a partial and somewhat fragile perspective on the comparability of the former. Of the 12 comparisons of outcomes between year 3 in MV and year 3 in comparison sites, 3 are substantially improved at 5% significance levels (bednet use, malaria prevalence and access to improved sanitation)
(3) the third and final comparison is a "kind of" difference-in-difference comparison: what is the difference in the change in MV sites between years 0 and 3 the change in comparison sites between years 0 and 3? It is "kind of" because the sites were not randomly allocated between MV and comparison at year 0, and the year 0 data for comparisons sites are collected in year 3. Of the 6 outcome comparisons made here, 2 are substantial improvements that are significant at the 5% level (skilled birth attendance which increased by 24.7 percentage points in MV and by 12.7 percentage points in comparison sites; and mortality rate in children younger than 5 years of age (deaths per 1000 births): in MV sites this number decreased by 24.6 and in the comparison sites it actually increased by 5.9).
- 5 of the 18 outcome comparisons between MV and comparison sites show significant improvements. This does not seem like a very high number. Why so few?
- But some outcomes are more important than others, and a big outcome is the decline in child deaths, and this was in one of the "kind of" difference-in-difference comparisons.
- If we find the cost data credible, is this a good enough return for a more than quadrupling ($27 to $116 per person per year) of spending over the three years on these multi sector interventions? I have no idea--it would have been really helpful if, for example, the child mortality numbers had been converted into cost per death averted so we could benchmark against other interventions.
- The more interesting comparisons will be the changes in outcome variables between year 6 (say) and year 3 between the MV and comparison sites, because although the nonrandom allocation of sites problem remains, at least the observations in years 3 and 6 in MV and comparison sites will not be based on 3 year recalls.
- The key thing remains sustainability--is there life beyond proof of concept or is the MV doomed to the start-up "valley of death"? To assess this it would be good to have a better sense of what is in and what is out when the cost data were assembled.
Impact? Getting warmer, but more work to do.
Sustainability? This needs some serious research.
Verdict on MVs? Still out.