21 October 2014

Evidence on the impact of research on international development? New DFID literature review.

A new DFID literature review hit the streets in July: 

"What is the evidence on the impact of research on international development?  New DFID literature review". 

It is written by DFID staff, reviewed by 7 external experts.   

This is the concluding chapter in full:

"Investment in research activities can lead to development impacts. However, research is not a panacea; Investments in some research projects will lead to large development gains while investment in some activities will lead to little or no impact.

Having said that, it is possible to consider the general theory of change by which we expect research to lead to impacts and to draw some conclusions on what type of research activity we should invest in to achieve different outcomes.

Key findings

This literature review reveals some general principals regarding investment in research:
  • Investment in public research in low-income countries is unlikely to lead to substantial levels of direct commercialisation of research outputs in the short to medium term. Informal academic engagement with industry may be more economically important but some activities (particularly consultancy contracts) may have negative impacts on research capacity. While the generation of new innovations within low-income countries is unlikely to be a major driver of growth, the absorptive capacity of industry (i.e. the ability to adapt and make use of existing research knowledge) will be a vital driver of growth and interventions which would increase this absorptive capacity (e.g. strengthening of tertiary education) are likely to have large impacts.

  • Research from developed economies reveals that, in contrast to the beliefs of most academics, there is no strong link between the research outputs and teaching quality of tertiary education establishments. As yet there is no evidence to indicate if such a link exists in low-income countries so efforts to improve tertiary education through investment in research should be treated with caution. The human capital developed through investment in research and research capacity building can have multiple positive impacts on development including via the spill-over of former researchers into government and industry and the generation of research experts who can act as policy advisors. Research suggests that investment in doing research in low-income countries on its own will not lead to improved research capacity and that an effective and explicit capacity building strategy must be developed.

  • Investment from both developed and low-income countries in research in public institutions and/or public-private partnerships can generate pro-poor products and technologies. Some of these products and technologies have had dramatic impacts on development. However, there are also multiple products and technologies which have not had the expected impact. When investing in the development of products and technologies, it is vital to carry out research to ensure there is a demand for it and that there are no barriers which will prevent it from having positive impacts. 

    Using evidence to inform policy and practice decisions can help ensure policies and practice achieve their desired impact. Two categories of decisions can be informed by evidence. Firstly, research to understand what works and why can be used to inform decisions on specific interventions. Secondly, evidence which describes the existing context can inform general theories of change. Major barriers to use of evidence are the low capacity of policy makers and practitioners to understand and use research evidence, and the absence of incentives to drive research usage. Interventions which succeed in increasing use of evidence by policy makers and practitioners may lead to important impacts. 

    There are a number of methods which have been used to quantify the economic benefits of investment in research. All methods suggest positive rates of return to research however individual figures are highly sensitive to the assumptions used in the calculations and therefore need to be interpreted cautiously."
A few observations:

1.  For such an important question it is surprising that this was not a systematic review.  The search strategy is listed in Chapter 8, and it seems quite partial.  Key literature like S. Fan et. al. on the impact of investments in agricultural R&D (compared to other investments) on poverty in India, China and Africa is missing.

2. The report would have been much more helpful if it could have said more about the features and attributes of research that give it more of an impact. Some research will have an impact, some won't. No surprise there.  The reasons for impact may relate to relevance, rigour,  originality, timing, sheer luck or some combination of these.  Also, what does the evidence say about the timescales over which we should expect to see impacts?

3. Barriers to use of research.  Good points on the barriers to research uptake although with an over-focus on capacity.  Of course the most relevant, brilliant and accessible research might be ignored if it gives an inconvenient political answer.  So this is more than a capacity issue, it is also a governance issue.  It is not just about demand but about a balance of (sometimes) competing interests (within the public and private sector spheres).

4. The comment that "all methods suggest positive rates of return to research however individual figures are highly sensitive to the assumptions used in the calculations and therefore need to be interpreted cautiously" seems to vaguely cast aspersions on the research community--i.e. we have a vested interest in a certain outcome and we cannot be trusted to assess returns to our work (perhaps why DFID chose to do this review themselves).  Maybe I am misinterpreting, but that is how it reads to me. 

Overall, an interesting and intriguing study.  I wonder what impact it will have? 


Kirsty Newman said...

Many thanks for blogging about this Lawrence and I am glad to hear you found it interesting. I was the lead author on the review and certainly found it interesting to write. We chose to do it because we were asked so often what is the evidence that investing in research is a good thing - and we felt that the only answers we could offer were pretty flimsy. I was particularly interested to find that the review suggested that a couple of 'mantras' on research impact were actually not backed up by the evidence. I have written much more about the review on my own blog starting here: http://kirstyevidence.wordpress.com/2014/10/07/science-to-the-rescue-part-1/

I hope its ok if I offer a few responses to your comments.

1. On why this was not a systematic review - the quick answer is that the topic was just too big! In fact it would need to have been multiple systematic reviews but we decided instead to do essentially a review of existing reviews - a kind of meta-review. Therefore, wherever possible, we limited our search to existing synthesis products. That is the reason that the primary evidence you mention is not included but it is covered by some of the reviews we included. Now that we have done this paper, we may choose to commission some systematic reviews for the areas where none exist - but possibly more importantly was the fact that in some areas there was almost no good quality primary evidence. So that is another priority area.
2. Regarding what features of research make it more likely to have impact, this is covered in brief in the paper but was not a primary focus. In fact there has been a large amount written on this topic in the past - see for example Fred Carden's book or work from RAPID. I think such reserach is interesting BUT... it needs to be interpreted cautiously. There is a danger that people try to imitate the features of research which has impact without asking the more important question of which research ought to have impact. So for example, there is a lot of guidance suggesting that people should get better at writing their research outputs up as policy briefs and publicising via social media. I have a worry that such approaches, in the absense of decision makers who can differentiate good from bad, can lead to poor research with good comms having disproportionate impact.
3. It is true that capacity is not the only factor hindering research uptake - in the paper we talk about both capacity and motivation/incentives (which we use broadly to include a range of political cultural and personal factors). I agree that the latter is very important but I also think that the former tends to get glossed over. There are hundreds of studies looking at the political economy of reserach use - some of the best come again from the RAPID team. However, from my experience working with developing country policy making institutions, I have seen time and time again that reserach is not used even where there is a political will simply because there are not people there to find it, appraise it and summarise it for decision makers. I find this to be a big difference between policy making organisations in the UK (e.g. the UK parliament or UK ministries) which have armies of technical advisors and analysts to support senior decision makers. This layer is simply missing in many developing country policy making organisations. I think the reason capacity is often overlooked is that it is quite politically sensitive to mention. People find it much more palatable to say that reserach isn't used because politicians are swayed by politics that to suggest that there might be a lack of staff with capacity to understand the research.
4. On the final question... I don't think we should distrust all researchers ;-) But I do think we should look at research into rates of return critically and be sure that the methods used are actually giving an acurate reflection of reality and not being driven by other motivations.

Lawrence Haddad said...

Hi Kirsty, thanks for the thoughtful blog--you make many sensible points. On 1, I still think the review of reviews could have been more systematic. On point 2, I think that this is the most important question, so its a pity it was not focused on more (perhaps for a follow up?). On 3, capacity is of course key--for me the big issue is where to know to invest and how to know if it has worked? On 4, glad to know there are some trustworthy researchers out there. Best, Lawrence