Process commentary from Lawrence Haddad and Alison Evans, the external reviewers to the MAR and MAR Update

The Multilateral Aid Review Update is now out.  I will do a blog on the content in the next few  days, but I thought I would share with you the process comment from Alison and me (on p 183! of the report).  Here it is:

"As with the original MAR, as external reviewers we were involved in commenting
on all phases of the Update: (a) the Update methodology; (b) moderation of scores and (c) the interim and final Update reports. Here we provide some reflections on the process:

Update methodology

We noted that the purpose of the Update methodology was to assess progress against reform objectives identified in the original MAR assessments. The purpose was not to re-assess original MAR scores. As such we considered the methodology fit for purpose. We did note, however, some challenges in implementing the methodology, particularly in being crystal clear about the thresholds between progress ratings (i.e. between some and reasonable progress and between reasonable and significant progress) and the importance of establishing a consistent rule for both aggregating ratings and for justifying any component score changes that resulted.

All these issues were discussed extensively with DFID and we were satisfied with their response and their willingness to take appropriate action as they refined the methodology.

Moderation of scores and follow-up

Moderation was undertaken in three separate tranches. Each tranche considered a cluster of multilateral organisations (MOs). Extensive internal moderation and consistency checking took place in advance of all three meetings. Both reviewers read all the assessments in full. We also read all the moderation notes. There were no conflicts of interest declared by the reviewers.
The quality of the Update assessments was high overall with considerable efforts made to gather a range of data and to triangulate data wherever possible. We did note, however, the lack of independent evidence (evaluations, independent reviews) for quite a few MOs and the unevenness with which some of the data from country partners was sought and presented. We appreciated the DFID assessments of strength of evidence presented for each MO in each reform component. We discussed the need to distinguish data on country-level impacts from more general data gathered from country partners, something which was followed up on by the MAR Update team.

We were able to ask for clarification on how certain pieces of evidence were used to justify progress ratings, and noted where we felt there were inconsistencies. In some instances the lack of sufficient evidence was itself an important factor in the assessment. In all cases our challenge and our input were taken seriously. In several cases there was so much evidence presented it was difficult to get a sense of its relative importance in driving the progress rating. Our comment here was taken on board and clarifications promised in the final assessments.

After each tranche discussion, checks were carried out to ensure that any ratings changes arising from the external moderation process were done fairly and consistently. Of the 146 component progress ratings moderated, 27 were changed at the final stage of moderation, which we were part of. Seven score changes were made.

While the Update was not as ambitious in scope as the original MAR, it was still a major undertaking. We note once again the high degree of transparency, accountability and sound judgement with which DFID staff approached the whole process. All the documentation was received in good time. Our views as external reviewers were taken seriously and the responses were generally adequate. Where we felt strongly that additional material needed to be provided to justify a rating, more work was done.

It is not yet fully clear on whether an Update similar to this will be undertaken in the future, but if it were, we felt that the following would need to be addressed:

1. Have the MAR and the Update exercises provided good value for money? We feel this is probably the case, but we are close to the process and our views are not based on a systematic review process. We feel the commissioning of a light touch independent review might promote learning for the next MAR and might give some guidance as to the level of effort allocated to it.
   
2. The input of country offices was, we felt, unnecessarily tentative. We understand the need for anonymity, but we felt that allowing country offices to volunteer information on whichever MO they felt like commenting on (if our understanding is correct) is insufficiently rigorous.
   
3. While we were impressed by the thoroughness of the assessments the sheer volume of evidence was sometimes overwhelming and made it difficult to trace the mapping onto the progress rating. We felt that less might be more here and that a focus on the evidence that really counted would have helped the read across to ratings.
   
4. The original MAR clustered MOs around functional type. This was essential because of the need to establish a set of baselines at roughly the same point in time. For various reasons the three tranches of MOs in the progress rating sessions mixed up MO functionality and we feel that this made it harder to ensure fairness and equal treatment (which we feel was, nevertheless, achieved).

We appreciate the opportunity to provide some public reflections on the process and we commend DFID for the professionalism with which they have undertaken a complex and difficult task. We were glad to have played a small part in the process.

Dr Alison Evans and Prof Lawrence Haddad"