Today we held the concluding workshop of a project between IDS, the London International Development Centre (LIDC) and the London School of Hygiene and Tropical Medicine (working title: When Worlds Collide).
Four papers—each with some variant of systematic review--were presented on the following questions:
(1) To reduce malarial infections should we be investing more in malaria-preventing development (e.g. building prevention into the design of shelter) or in malaria control (e.g. drugs, insecticides). In an era of pressure on aid flows and when resistance to malarial insecticides and drugs are increasing, the paper concludes that we should be spending more on malaria-preventing development. This paper used systematic review processes and non-systematic review approaches to find its evidence.
(2) Why do some institutional arrangements seem to offer better quality of health (and other) services to low income populations? The paper found that directing resources (whether private or public) to institutions as opposed to individuals improves quality because institutions can signal quality more clearly, are less fly by night, and have stronger incentives for repeat transactions.
(3) Does agriculture-driven food price policy affect undernutrition and diet related chronic disease? Out of hundreds of studies found through online and bibliographic searches, we only found 4 studies that linked agriculture-driven food price policy with nutrition status. The direction of impact were as predicted—higher food prices increased undernutrition and lower food prices led to increased overweight and obesity. The surprising result was the lack of evidence (and we included ex-ante modelling studies and ex post evaluation studies) on the direct link between agricultural driven food price policy and nutrition status.
(4) Do systematic reviews miss multiple effects of interventions? Here the paper was a re-review of an existing systematic review looking at the impact of water, sanitation and hygiene interventions on diarrhoea rates. It found that a high proportion of the studies in the original systematic review contained impacts on other outcomes, sometimes of direct relevance to the interpretation of the diarrhoea outcomes.
The original plan was for the 4 papers—supported by DFID—to come out simultaneously in the Lancet and World Development. In retrospect, this was a ridiculously unrealistic goal. Richard Horton the Editor of the Lancet, and one of the panellists at the workshop, dubbed the project a “glorious failure”. That’s a bit unfair, but not much.
Why failure?
It was a failure in the sense that we could not successfully apply systematic reviews to the really big cross-sectoral questions (the questions just could not be framed specifically enough). We also failed in our quixotic attempt to get the 4 papers published as a set simultaneously in the two journals (although one is with the Lancet, one with World Development, and the other 2 are about to be submitted to 2 different journals).
Why glorious?
Perhaps because those involved in the project learnt a lot and can share a lot. What did we experience?
• We have very different languages (and we were folks generally used to working across disciplinary boundaries). “You say consumption (diet) and I say consumption (expenditure)--let’s call the whole thing off.”
• The development and health journals are not set up to evaluate transboundary research (one journal editor said “there aren’t enough interdisciplinary reviewers”)
• UK universities are facing incentives such as the Research Excellence Framework which tend to place a premium on single disciplines
• The nature of credible evidence to each community is very different—health colleagues tend to be more enamoured with randomised controlled trials (RCTs) and classical systematic reviews, while the development colleagues tend to not worry so much about comprehensiveness and systematic inclusion but more about understanding external validity (context, nuance, formative research)
• Research funders know that they provide important incentives to researchers, and they do not seem as bound by disciplines and sectors as other research actors
• Those in civil society are probably scratching their head about all this as they tend to be eclectic users of evidence in any case
We also felt that there are not many champions for this kind of work. This is a pity and is perhaps an ethical challenge. Many if not most of the policy questions of the next decade will respect no boundaries—how will we get anything but fragments of answers to them? How to create champions? New multidisciplinary MA programmes are surely part of the answer.
Unfortunately the 4 papers are not available for wide sharing just yet, but if you want to find out more contact Lucy Tusting (LSHTM) for the malaria paper, David Leonard (IDS) on institutions, Alan Dangour (LSHTM) on agriculture and Michael Loevinsohn IDS) on water, sanitation and hygiene.