15 May 2011

A New Harvest of RCTs?

I am reading The New Harvest by Calestous Juma on how innovation and entrepreneurship need to be rethought in African agriculture. I can recommend the book—lively, provocative and well-evidenced. One of his key points is that agricultural innovation and knowledge generation in Africa needs to be decentralised. He argues that the interactions of climate change and ecosystems are expressed in variable and unpredictable ways and this makes context specificity more vital than ever. It is not that research and technology developed in one area will not be relevant in another, but just that we must continually challenge whether it will be. I agree with this conclusion.

Over the weekend I also read a column by Ben Goldacre on his “fantasy” that UK public policy should be driven by the MIT Poverty Lab approach. I have not yet read the latest book from the Poverty Lab which he refers to, but it strikes me that the randomised controlled trial (RCT) approach is at risk of doing exactly the opposite of what Juma advocates.

RCTs run the risk of locking in a result, not only within a country but across countries and over time. This is partly because: (a) they are costly to run (although I agree with Goldacre when he says you have to compare the cost with the costs of adopting the wrong policy) and so replication will be seen as costly and not terribly sexy to researchers or funders, (b) they are iconic (e.g. Progresa in Mexico, Orange Flesh Sweet Potato in Mozambique) because they are expensive and donors and researchers want to maximise their investments by publicising results (if they are positive), (c) they are not great at exploring the distribution of effects (sample sizes get too small if there are too many treatment variants) and (d) they are not particularly interested in external validity (understanding the likelihood of an interventions shown to be successful in one area being successful in another) because this requires a different set of skills.

It is interesting that RCTs have not been applied to UK public policy (I am assuming Goldacre is correct in this assertion). One could argue that heterogeneity (e.g. relative differences in behaviours and contexts by region and class) in the UK is lower than in emerging economies and hence the above worries about RCT portability are less valid. I wonder if this low RCT uptake in the UK is because of an anticipated stronger push back on ethical concerns (e.g. the challenge of getting informed consent when randomisation of treatment and control happens at the community or cluster level), or because of the political problems of doing such a pilot in the context of UK’s muscular media, or because UK communities would not put up with being seen as laboratory subjects.

Don’t get me wrong, I am not blind to the virtues of RCTs, I just don’t want to be blinded by them.


Charles Godfray said...

One of the most interesting applications of randomised control trials in UK agricultural policy was the Randomised Badger Culling Trial (RBCT) that ran throughout most of the last decade and tried to understand whether bovine TB could be reduced by culling their wild animal reservoir. It provided a clear answer to a specific question, though was very expensive, politically contentious, and illustrated many of the other points explored by Lawrence. A further issue was that participating farmers knew what treatment they were assigned to and this may very well have affected their behaviour and hence the outputs. There's lots about the RBCT on the Defra website and I think there is some academic work in progress reviewing it.

Jason said...

I agree, Africa is such a diverse continent, ecologically, socially, etc, it seems like external validity would be especially hard for RCT's in agricultural development. And in regards to testing policies for food security, picking treatment and control groups would be especially unethical.

Anonymous said...

What interests me is whether RCTs have to be expensive when it comes to agricultural field trials. There is a century of ag research institutions globally testing methodologies that certainly take approaches to randomizing into account, and it is clear to me the development economists and ag scientists have not merged their discourse much at all yet. We cannot assume that the ag research centers are going to find this RCT parlance "innovative"..