Monitoring and Evaluation of Research Communications: Scoping Study

This 30-page report includes details of a study on the monitoring and evaluation (M&E) of research communications for development, in line with what is described here as "a long tradition of seeking to understand the determinants of behaviour change." It draws on research launched at a workshop in February 2006 which included representatives from organisations including Healthlink, Institute of Development Studies (IDS), International Institute for Environment and Development (IIED), International Network for the Availability of Scientific Publications (INASP), Panos, the Overseas Development Institute (ODI), and SciDev. The study is based on a list of relevant literature provided by the steering group and telephone interviews of varying lengths with 16 practitioners working in the field of M&E of research communications; 5 case studies emerged (which are detailed here).

In an opening overview section, key approaches and methods used in research communication - gleaned from the literature - are shared, such as the importance of capturing elements of context in M&E of communication research. For example, one author cited here (O'Neil, 2005) conducted a formal evaluation of 22 International Development Research Centre (IDRC)-supported research projects in developing countries, finding that "the influence that research has on policy is determined in the particulars of time and place, both in the character of research and of the researcher and in the politics and processes of making and conducting policy." She identifies 3 elements that are essential if development research is to have an influence:

1. Intent (researchers must want to communicate their research);
2. Direct engagement with the policy community (this means more than communicating information; rather, it means forming relationships with policy-makers that can endure); and
3. Public participation (members of the research community must become participants in democratic governance).

Beyond this specific illustration, a range of methods and tools are currently in use, according to the researchers; interviewees generally drew on a mix of quantitative and qualitative methods. Although the most commonly used conceptual framework for M&E was found to be the logical framework, no examples of the use of the "Most Significant Change" method were found in the context of research communications, but this is an area which many thought to have potential. (In subsequent sections, of the report, other methods, such as the Theory of Change (ToC) and Mixed Method Indicators (MMI) for undertaking M&E of communicating research are outlined.) A number of interviewees referred to difficulties experienced in designing good interviews, whether structured or semi-structured.

Methods and measures used to assess the impact of research communication are outlined here, highlighting the role of a variety of face-to-face and technology-based ways of gleaning information about what works with regard to evaluating programmes and then disseminating research - and why. One element, for instance, concerns the centrality, within recent approaches to M&E, of the participatory development of indicators, either quantitative or qualitative. Also, based on the findings, the authors suggest that implementers of research communication projects or programmes collect better baseline data, carry out regular monitoring as well as evaluation, undertake more strenuous identification of audiences and pathways for the communication of research, and build space for reflection and learning throughout the project cycle.

The authors note that a number of sources emphasise the importance of establishing a relationship with those that the research is intended to benefit. They focus on the following elements:

1. potential users are more likely to use the research if the research (and source of research) is trusted;
2. research is more likely to be assimilated if it comes through routes that people are familiar with; and
3. the influence of research findings is likely to be cumulative and needs to be built up over time.

In taking learning forward in the area of research communications, the authors propose a number of key areas for discussion, such as:

• the need for further investigation into the models of information flow - how does research from research institutions based in the "west" or in so-called developing countries reach specific audiences in developing countries in the light of ever-changing technologies?
• the need for further investigation into approaches beyond logical frameworks, such as outcome mapping and network analysis in the context of research communications.
• the need for research into such questions as: does good M&E feed into better research design? In what ways? How can researchers learn more from experiences of those who use the research?
• the need to attend to lessons that can be learned for research communications for development from the private and public sector.