28 July 2012

Is there a "Haddad" effect? Results from a randomised controlled trial

It's not often that researchers from an Institute do a piece of research that evaluates the influence of the Director of the Institute. (The impertinence of it.)
But in a moment of weakness 12 months ago I agreed to play such a role in a project conducted by a team from IDS and 3ie.

The key questions posed in the research: (1) do policy briefs influence readers? (2) does the presence of an op-ed type commentary within the brief lead to more or less influence? and (3) does it matter if the commetary is assigned to a well known name in the field?

So Penelope Beynon and Edoardo Masset from IDS, together with Christelle Chapoy (3ie) and Marie Gaarder (3ie and now NORAD) designed an experiment to try to answer the above questions.  The draft report is here (it will be out soon as an IDS-3ie working paper).

They created 3 treatments and a control: (1) a policy brief based on the agriculture-nutrition systematic review that I was involved in (Masset et. al. 2011), (2) the same policy brief with an op-ed commentary written by me but assigned to "an IDS Fellow", (3) the same policy brief with an op-ed commentary written by me and labelled as being written by me and (4) a control: a recent policy brief on a topic unrelated to ag-nutrition but relevant to the readership of agriculture-nutrition issues (progress towards MDGs).

They constructed a list of about 800 people who agreed to participate in the experiment (from a list of about 75,000 emails) and then randomly assigned the 800 people to the 4 groups (the authors did a power calculation to determine the required sample size). They then conducted questionnaires--would the participant review the message in the brief, share it, seek further information, review their practices, or change practices--immediately afterwards and then one week and 3 months later. They generated 11 outcome variables in all.

The study has limitations, which the authors fully acknowledge.

First there is some serious self selection from the 75,000 people contacted to the 800 who participated (they did a power calculation that showed that about 800 responses were sufficient to detect a difference if one existed). The 800 were certainly not typical of the 70,000 contacted. 
Second, the subject of the generic brief probably mattered: it had to be not a million miles away from ag-nutrition links otherwise it would not be of interest to the people recruited, but it had to be distinct enough to be able to create space for the 3 interventions to have an effect. 
Third, all the outcome variables were self reported, via the internet, with few opportunities to probe answers. 
Fourth, the results will not be valid for other policy briefs on other topics.

So what did they find? There are too many results in the paper to summarise here, so let me focus on the authority (or Haddad) effect.

Intended actions at immediate follow up: compared to the other 3 treatments, the authority effect mattered in 2 of the 11 outcome variables (more likely to share the policy brief on and more likely to tell them about it). These differences remained significant at one week, but diminished to 10% significance.

Actions at three months follow up: the sharing and telling effects remain strong.


So sharing and telling seem to have been affected by the fact that Haddad is on the byline, but the name per se has not changed anyone's practice.

This is somewhat reassuring: the authority name has given people an additional reason to share the brief, but people are still making up their own minds about the issues.

Maybe this is the most we can (and should) hope for from a policy brief, to generate a buzz, to get the dialogue going, to create a space for the evidence to percolate through.

Flawed though the study is, I think it is pathbreaking in that it demonstrates the scope for further study about how to communicate research--this is vital, especially in areas such as nutrition which are so easily neglected.


Dwayne Jacobs said...

"Flawed though the study is, I think it is pathbreaking in that it demonstrates the scope for further study..."
Dear Mr. Haddad/ Ms. Garder/ Mr. Masset/ Ms. Chapoy, I'm sorry but this sounds really lame. At par with findings/recommendations of fact finding missions that take MPs to Bahamas in winter months.

Lawrence Haddad said...

Dwayne, ouch...

What sounds lame... the way the research was done, the way I wrote about it, or the fact that the research was done in the first place?

Dwayne Jacobs said...

Hi Lawrence,
Largely the fact that many would celebrate such a study as pathbreaking. The outcomes were what they were... no one gets into research knowing that it will come up with definite answers. Its okay if a study only comes up with the revelation that further research is needed. But calling it path-breaking is a choice I do not understand the basis of.

thanks, Dwayne

James Muir said...

Hi Lawrence - brave of you to even venture into such a topic! Wasn't there an eponymous Syrian god of storms? That would really confuse the issue!
Best wishes

Lawrence Haddad said...

Dwayne, it's not the result that is pathbreaking, but the fact that the authors made a credible attempt to answer the question in the first place.... have you seen other studies like it in the development communication field? best

Theodore Van said...

One thing that should have been taken into consideration that would also play a factor in influencing readers is the policy's promotion. For instance, when one wants to test readers' reaction to a policy, seo services australia will promote it and gather data on different forums and commenting platforms, thus getting a general idea of how the readers were influenced.

Archie Williams said...

Indeed, any research that is of substantial value should be communicated. A research that remains in its electronic form, optimized for the world to see, yet was not cascaded to the concerned population, then it is of no use and efforts will be for naught.