14 September 2014

Fitting the Evaluation Design to the Intervention: New paper on nutrition and mobile phones in Indonesia

It is not often that you see a paper that outlines the process of how an intervention design was created to fit the context and features of the programme and the outcomes of interest.

Often the impact assessment method is fixed and the researchers only take on impact questions that fit the method they prefer, hence there is little effort expended in figuring out what is the right way to evaluate a given programme.

Here is a new paper from Inka Barnett and colleagues at IDS that bucks this trend.  The researchers from IDS were looking for a mobile phone intervention in nutrition to evaluate.  They found a willing partner (World Vision) in Indonesia who said this is what the intervention looks like (mobile phones in nutrition growth monitoring, counselling and service delivery) and this is the flexibility there is to adjust the intervention to permit better evaluation (not much).  The question all parties wanted the answer to was whether mobile phones are better than paper based methods for generating more accurate and timely information on nutrition outcomes and behaviours, leading to more timely and more effective action.  And if so, why?

The study is just starting, but the care with which the impact evaluation design has been created is different.  For example, for various reasons it was not possible to randomise the mobile phone versus paper method of recording, so there can be no treatment and control group.  Undeterred the evaluators scoured the evaluation toolbox and came up with a Mill's Method of Difference design (circa 1843!) which identifies the net effect of mobile phones by looking at the difference of the two methods used simultaneously by the same set of clinics and households.

I recommend the paper--a good example of how to rummage around in the big evaluation tool box for the right tool for the most important job, rather than only picking the jobs that yield to the tool you are most comfortable with using.

1 comment:

Andre said...

Hi there. Thanks for this. Unfortunately, I'm unable to open the link. Sounds fascinating though!