Leaving a Footprint: Stories of Evaluation that Made a Difference

"Evaluation can have a transformative impact on the lives of the users of the programmes and policies by giving them a space for their voice and their expression, contributing to their inclusion in decision-makers' mental models."

Interest in evaluation of public policies and development interventions in general has grown, especially since the launch of the Sustainable Development Goals (SDGs). Evaluation results can provide material for decision making, contribute to the learning of teams and organisations, and increase knowledge of the problems that are the focus of public action. Evaluation practice includes a range of methodological strategies, scopes, and audiences. This book compiles 7 diverse evaluation stories in development contexts that produced positive effects in the locale in which they took place. It may be of use for social programmes' managers and technicians, as well as academics, evaluators, and whoever is interested in the processes through which different social actors change their evaluation experiences in learning social change paths.

Scattered throughout Latin America and the Caribbean, the narratives cover evaluation of interventions that worked with children, rural young people, and indigenous women, as well as evaluation of health programmes and university self-evaluation. Through what are meant to be engaging narratives, these change stories identify the factors that allowed evaluation itself - the very process of it - to enhance local development in ways that go beyond the mere use of evaluation results that change policies or programmes. The stories are accompanied by illustrations that aim to reinforce their transformative nature.

The request for evaluation stories was issued through multiple forums and social networking sites. Seven stories were selected; brief strategic lessons to emerge from each are highlighted, below:

1. If You Don't Ask, You Won't See It! [qualitative evaluation of the Oportunidades Programme, Mexico] - This story illustrates how evaluation identified language barriers that prevented economically poor natives from benefiting from a money transfer programme. Changes to the programme allowed communication in local

languages, which improved the ability of people to understand the programme's requirements (such as children's regular school attendance) and, therefore, enabled them to benefit from the money transfer the programme offered. Without the technical ability and competence of the evaluation team - in this case, the evaluators' anthropological approach - there would have been no guarantee that the evaluation would benefit anyone. An actor who believed in the potential of the evaluation effort and facilitated the implementation of some of the suggested recommendations played the role of a champion.

2. Giving Account and Becoming Fully Aware of the Account [process and impact of youth participation in territorial development in Santa Catalina Valley, Peru] - The interdisciplinary nature of the evaluation team was very important. This team was developed over an extended period of time and was supported by a range of stakeholders.

3. Indigenous Women, Territoriality and Evaluation [the story of a (de)constructed road, Colombia] - Indigenous women were invited for a talk or chat with the intention of "lowering anxiety", without explicitly mentioning that they were being involved in an evaluation process; this was made clear soon after the women arrived. Ultimately, this evaluation validated many of the approaches adopted and indicated the potential that could be developed by having an additional training to support the objectives outlined originally in the project.

4. Local Knowledge and Institutional Actors [evaluation of the Safe and Family Centred Maternity Hospitals Initiative (MSCF), Argentina] - There is not one single way of generating believable and convincing information, given that what is "believable" depends on the situation and on the specific actors. In some cases, the key is to assess the programme results using an approach that helps the process to be perceived as methodologically rigorous. In other situations, a believable evaluation entails understanding the perspectives of the most relevant actors in the intervention (as was the case here).

5. An Evaluation With a Caribbean Accent [Roving Caregivers Program, Saint Lucia, Caribbean] - "The commitment to the use of the evaluation helped the process make a difference in people's lives. In that experience, the 'personal factor' - understood as the presence of an individual or group of identifiable individuals who are personally concerned with the evaluation and its ability to improve people's lives - worked as the main driver for the use and impact of the evaluation."

6. From Indifference to Appropriation [the self-evaluation process in the National University of Lanús (UNLa), Argentina] - The involvement of the university community allowed democratic access to information, interaction, discussion, and debate, as well as strengthening abilities of the participants.

7. The View of Those Who See [participatory evaluation of the cancer care and prevention programme in Valle de la Estrella, Costa Rica] - Credibility was achieved through the active participation of the users in the evaluation process: Regional technical teams were involved and deeply interested in getting to know how the evaluated programme worked in their area. In contrast, the higher authorities limited their participation to approving the evaluation. In this way, recommendations at regional and local levels were applied soon after the evaluation finished, whereas general recommendations - dependent upon the higher authorities - have not yet been applied.

The authors elaborate: "Participation of the actors in the collection and use of the evaluation data is a powerful way of including users and beneficiaries. This allows participants to get involved and understand the data better. An evaluation characterised by a collaborative approach leads participants to take responsibility for the evaluation and then for the change and transformation that follows. In this way, active participation in the evaluation process helps to develop better understanding of evaluation and contributes to commitment and use....Clearly, the more participatory the evaluation is, the more necessary it is to ensure the willingness and motivation of the most relevant actors in the intervention (beneficiaries, local technicians, officers) in order to promote the impact of the evaluation and for it to make a difference." In the Costa Rica experience, having a limited time frame for the training of the team members resulted in certain limitations during the evaluation process. The key factors to be taken into account when developing the skills of evaluation team members are: time, the content to be taught (relevant to the specific evaluation task they have to carry out), support and supervision by one or more experienced evaluators, and application of skills people already have.

Overall: "The evaluation stories...illustrate the value of a positive approach....They emphasize the evidence of what works and/or might work and is worthy of being continued, enlarged or modified. This can be contrasted with evaluations that are focused mainly on the technical problems and deficiencies present in most interventions. This type of approach has been shown to be highly relevant in getting beyond defensive and suspicious attitudes, and instead promoting a constructive focus on possible solutions."

According the book's authors, it was difficult to access "evaluations whose main actors (officers, administrators, evaluators) considered that the evaluation had made a (positive) difference....Generally, evaluators do not go into detail on how their work can have a positive impact on people's lives. They assume that their responsibility and tasks do not extend beyond selecting the appropriate methodology or method capable of influencing decision-making."

Going forward, then, what may be needed is "extending the focus of action of the evaluation to contribute to public good, broadening its interest towards medium and long-term results, to unexpected consequences of development interventions and investigating the causes of some social problems that programmes and policies aim to deal with."

Click here for the 97-page report in PDF format (Spanish).