A guru has spoken. Angus Deaton at Princeton has a newish (I'm catching up) paper out called “Instruments, Randomization and Learning about Development”. Ok, it’s not the snappiest title, but it is authoritative and by turns sobering, refreshing and taxing.
The whole message of the paper is that a focus on “what works” in the absence of why something works will be nothing more than the generation of isolated bits of knowledge--knowledge that has no transferability or relevance outside of its context.
Deaton argues that randomized controlled trials (RCTs) have accentuated the shift from why to what. But before going on to make this point he notes that RCTs—unless under ideal circumstances—are not even a magic bullet for solving the issue of “what works” in a given context. And even under ideal circumstances he points out that RCTs only generate impact estimates at the mean of the population (i.e. they would not be able to distinguish between interventions that generate very large effects for a few and negative effects for many and interventions that generate modest positive effects for the vast majority).
He argues that the application of experimental and non-experimental methods (such as econometrics) in the absence of theory generates highly localized but decontextualised knowledge—a double whammy. Experimental methods such as randomized controlled trials may even encourage an atheoretical approach.
RCTs will be most useful, as will any analytical method, when they test assumptions and results generated by a theory of change. He says we need to focus on “mechanisms” rather than “projects”—this will increase our chances of learning about what might work outside of a given context.
He runs through all the drawbacks and limitations of RCTs (he is equally harsh with econometrics and the terrible job most of us do with “identification” – i.e. isolating the independent effects of explanatory variables): conceptual, practical, and ethical. My favourite example of the fallibility of randomization is his critique of the use of alphabetization of school names to allocate schools to control and treatment groups (he cites several papers that show that alphabetization is not the same as randomization).
In his conclusions he says “for an RCT to produce ‘useful knowledge’ beyond its local context it must illustrate some general tendency, some effect that is the result of mechanisms that are likely to apply more broadly”. The proponents of RCTs, by definition, focus a huge amount of rigour on establishing internal validity (does the method give a credible estimate of a mean effect in this context?) but much less on external validity (is the effect portable?). In fact they often argue for external validity by failing to allow for the unobservable effects that the internal validity is so keen to control for.
So what are these “mechanisms”? Things like loss aversion, procrastination, risk taking and the way we discount the future. But it’s not clear to me that these have a great deal of portability outside of specific contexts either (many of these studies use US graduate students as their subjects!).
The conclusion, with which it is hard to disagree, is that there is no substitute for specifying the causal nature of the processes of change we are investigating.
As Deaton says “I believe that we are unlikely to banish poverty in the modern world by trials alone unless those trials are guided by and contribute to theoretical understanding." This is actually a pretty mainstream view--we have to have ideas about why and how things work to generate hypotheses to be tested. Perhaps some randomistas and econometricians have forgotten this along the way.
The paper is good, but it takes too long to point out the fallacy of thinking that we can dispense with "why works?" in favour of "what works?".
As development becomes more complex in an increasingly uncertain context, the "why" questions will become more important than ever.