Evaluation Framework for Governance Programs: Measuring the Contribution of Communication

In this 16-page report, the World Bank's Communication for Governance and Accountability Program (CommGAP) presents a methodology for evaluating the contribution that communication interventions can make to good, accountable governance in countries around the world. CommGAP engages in 3 complementary programme areas in an effort to amplify citizen voice; promote free, independent, and plural media systems; and help government institutions communicate better with their citizens. These 3 areas, which are described in more detail below, are: research and advocacy; capacity building and training; and support for development projects. This document describes the evaluation framework - that is, the outcome and impact indicators, and the methodology behind the assessment - that CommGAP has developed (with guidance from the Johns Hopkins University Center for Communication Programs).

As described in the introductory section of the report, CommGAP undertakes:

1. Research and advocacy - activities include:
   
   • commissioning a series of case studies that can be used in scientific publications, capacity building and training programmes, and advocacy to promote the use of communication in development.
   • developing practical frameworks and tools that can support development practitioners in their work.
2. Capacity building and training - CommGAP is crafting a set of core training modules in communication (particularly focusing on governance and accountability issues) that can be broadly used at the World Bank and other bilateral and multilateral agencies.
3. Communication support for development projects and programmes - CommGAP assists with governance-related operations in Africa and Asia, including stand-alone governance projects, public sector reform projects, post-conflict programmes, and programmes supporting decentralisation, community-driven development, and social accountability.

Organisers stress that, for purposes of designing and evaluating interventions, it is essential to have a model of how the intervention is expected to work; the reasons for this assertion are found on page 3 of the document (one reason is that developing a model allows one to make explicit (and, thus, open for discussion) the implicit assumptions of how impact should occur). Although this evaluation plan proposes a series of quantitative indicators, the authors indicate that "evaluators would ideally combine the data on these indicators with qualitative assessments of these same factors, gleaned through observation, newspaper reports, or local experts on the subject."

The quantitative model presented here - the logic model - illustrates a method for assessing the objectives of the main project, communication challenges, communication objectives to support main project objectives, the communication intervention itself, the outcomes (what change the communication has produced), and the impact (the contribution of communication to desired change(s) of the overall project - which are "argued, not measured"). CommGAP adapts this diagram to the context of each specific country in which it is working, depending on the objective(s) to be achieved, the strategy designed to achieve it (e.g., strengthen media systems, or support legal and regulatory reforms to improve access to information), and the theory behind specific programme interventions.

Section V of the document addresses the need to translate the concepts (shown in the boxes on the conceptual framework described above) into measurable indicators. Thus, as part of the evaluation, they track changes in these factors (the "intermediate steps"), as well as in the final result or outcome. This tracking, according to the authors of this report, is essential; in addition, if time and resources permit, it is ideal to engage in a process that tests whether the communication intervention actually caused those changes. That is to say, the descriptive approach to evaluation tracks and documents changes over time on each of the results shown in the logic model. Yet, the second (more complex, more expensive, and generally more desirable) model not only documents change, but measures the extent to which change can be attributed to the intervention.

CommGAP explains that some indicators are relatively easy to "operationalise". For example, one could measure "trust in government to provide truthful information" via a survey among a random sample of the general public. In contrast, it is more difficult, the authors argue, to assess "civic competence".

The decision of how best to measure a concept may also depend on the availability of existing data. The logic model requires data from at least 4 sources, which are outlined and described. They include: population-based surveys carried out to obtain data on the knowledge, attitudes, beliefs, and behaviour of the general public; surveys amongst private companies (e.g., to serve as a "barometer" for access to information and transparency in government); expert polls (which are useful, according to the authors of this report, because many of the changes required to obtain a final outcome in democracy and governance projects do not lend themselves to simple quantification); and legislative records.

Prior to several annexes that outline the logic model in more detail based on the specific objectives of the CommGAP initiative, the authors describe the limitations that can hamper the evaluation of communication interventions in the context of democracy and governance projects. The primary one, they argue, is the difficulty in isolating the effects of the communication initiative from other factors (e.g., secular trends) contributing to the desired change. Amongst the other challenges they describe is figuring out how to separate the communication intervention, as implemented by the World Bank or its recipient organisation. from the actions and activities undertaken in the same area by other partner organisations working toward similar goals.