An unsystematic look at the DFID systematic reviews

DFID recently put out a call for 49 systematic reviews on policy related topics as part of a pilot to “help practitioners and policymakers increase the use of rigorous evidence in decision making”. At a medium size of 25k the total resource envelope amounts to an approximate 1.25 million spend.

I spent a couple of hours reviewing the list of 49 questions. They have to be a good source of tea-leaves for researchers interested in mapping perceived policy knowledge gaps.

So what did I read into the tea leaves?

1. The area with the largest number of questions was growth and investment. As with any of these observations, this may reflect many things—the size of the team, the size of the portfolio, the state of knowledge, and the knowledge of the state of knowledge to name a few. In this case it might also signal a fast tracking of questions that the new Growth Centre will not yet be geared up to address. This large number probably also reflects catch up in DFID investment in these research areas. Agriculture and aid delivery had the fewest number of questions at 3 each. Health, perhaps understandably given its tradition of systematic reviews also had 3 questions within Human Development (7 in total).

2. “Impact” is highlighted in 25 of the 49 questions. Various forms of the terms “increase” and “reduce” came up 10 times and “effectiveness” came up 7 times. The focus on impact is not surprising—this is what the development policy field is heavily focused on. Impact was mentioned in every area except, agriculture, whose 3 questions used the term “effective”. I don’t know if this is significant, but it is interesting and I could easily spin it into a story about agriculture not having much of an impact culture, but I won’t.

3. There is some futureproofing embedded in the questions in terms of the interests of possible government configurations. There are questions on vouchers, on the role of the private sector in service delivery and a lot of questions in general about business.

4. Gender issues are explicitly mentioned in only 4 questions.

5. There are a few surprising areas where questions are missed—perhaps they are being covered in other windows: these include

a. Urban issues
b. Science and technology (there are 2-3 questions but not as many as I would have thought)
c. Communicating about and on development—the need to get better at this is a critical behavioural issue for maintaining the consensus on development and aid
d. Does ODA spending in nontraditional areas (security for example) have a poverty impact?

6. The guidelines state that data must focus on developing countries, with a strong rationale needed for including data from other contexts. I can understand the need for focus, but I think this is a bit of a missed opportunity to learn about what works in public policy across rich and poor contexts. These two worlds are far too divided.

In sum, I think DFID will get a lot for this, as will the wider development community (the reviews will be made widely available). I look forward to an evaluation of the difference the pilot makes in terms of feedback from users on whether they used the evidence and whether it influenced their choices. Whether it improved the poverty impacts of those different choices will have to be assessed at a later date.