End of Project Evaluation of the BBC Media Action Radio Waves and Tikambe Projects in Zambia

"BBC MA strives to build the capacities of media partners so they can act as a platform for dialogue and debate."

This report shares the end-of-project evaluation results of two projects implemented by BBC Media Action (BBC MA) in Zambia over the period 2013-2019, with funding from the Embassy of Sweden in Lusaka and the Swedish International Development Cooperation Agency, Sida. It is the first external evaluation conducted on these two projects, which has as its objective "to help assess the success of the projects and provide the Embassy and BBC Media Action [with] input for upcoming discussions concerning the preparation of a new phase of support". Partly summative and partly formative, the evaluation focused on the effectiveness of the two projects, but it also addresses relevance, efficiency, and sustainability.

The evaluation report starts by providing background information on the two projects, which operate in different sectors and are not linked to each other:

1. Tikambe - This project combines the use of media and interpersonal communication to provide information to young people about their sexual and reproductive health and rights (SRHR) in view of reducing their vulnerability to SRH risks. This takes place through: (i) technical capacity support of radio stations to produce engaging quality programmes on the topic, and (ii) interpersonal communication through young volunteers engaging in peer-to-peer education, community awareness, delivery of youth-friendly services in health clinics, and comprehensive sexual education (CSE) in schools. The project has been implemented in two phases in collaboration with Restless Development and currently operates in four provinces.
2. Radio Waves - This project focuses on media development by strengthening the technical and management capacities of radio stations to deliver quality governance programmes that engage audiences and gives them voice and influence. It has been implemented in two phases and covers nine out of ten provinces. The project also focuses on young people and advocacy in the media sector, through a collaboration with Restless Development and the Zambia chapter of the Media Institute for Southern Africa (MISA).

The evaluation is theory-based and relies on qualitative data collection methods. From October 7-21 2019, the evaluation team met with more than 250 persons, of whom half were in focus group discussions, and around 45% were women. These included groups the projects intended to reach, partners, and key stakeholders at the national, provincial, district, and local levels.

The results of the evaluation of each project are outlined in the report, with specific reference to the following factors:

• Relevance: To what extent did the projects respond to the priorities of Sida's policies and strategies and to those they sought to reach, including women and girls?
• Efficiency: To what extent did the management setup facilitate timely delivery of outputs?
• Effectiveness: To what extent did the projects achieve their intended outcomes, including for women and girls? And what unintended positive and/or negative outcomes were achieved, including for women and girls?
• Sustainability: Is it likely that the outcomes of the projects are sustainable?

To cite a few results looking solely at effectiveness and sustainability, the evaluation for the Tikambe project showed that the project achieved its intended results, including for young women, particularly in relation to knowledge gained about SRH and changes in attitude and behaviour in relation to drug and alcohol abuse and uptake of SRH services. School clubs, radio stations, and roadshows in urban areas were shown to be effective. However, key influencers, including parents and civic and religious leaders, were found to not have been sufficiently reached and included in a way that could have driven change at the community level. Related to the sustainability of Tikambe, the report states that the results achieved at the individual and organisational levels are likely to be sustainable if certain conditions are met. This mainly pertains to knowledge gained, communication skills (particularly for volunteers), and uptake of SRH. Having worked within government structures (schools, clinics), the priority given to young people's SRHR is expected to persist, and the improved technical capacities of radio stations have already been applied in other contexts where funding was available.

Results from the Radio Waves project in relation to effectiveness showed that the project achieved results in lifting technical editorial and production skills of partner radio stations to produce quality governance programmes that engage audiences, especially related to obtaining audience feedback. However, stations are not yet on their way to financial sustainability. Also related to sustainability, although staff capacities have been strengthened, the results showed that there are no mechanisms in place that ensure that capacity development can continue without the support of the project.

Following a detailed outline of the evaluation results, the report concludes with a list of overall recommendation for the Embassy and for BBC MA and project partners, as well as detailed recommendations for work with partner stations. For example:

• For the Embassy and related to Tikambe, the report recommends discussing the feasibility of targeting more remote areas within already targeted districts/provinces, where vulnerability of young people to SRHR risks, particularly of young women, is high.
• In relation to Radio Waves, the report recommends supporting the institutional capacity development of MISA in order to build a stronger basis for its sustainability as a key advocacy actor in the media sector.
• For BBC MA and partners, one of the recommendations applicable to both projects is to undertake a learning exercise about the sustainability challenges of individual radio stations (previous and current partners) and to consider how best to integrate holistic sustainability considerations for individual radio stations.
• Recommendations for partner stations include those related to: nurturing sustainable partner stations (through, for example, a review of mentors and trainers); preparing for onward sustainability (by, for example, mapping and identifying which organisations would be best suited to train new radio broadcasters, managers and technicians); ensuring wider outreach to more vulnerable groups (by, for example, considering incentives for volunteer placements within more remote communities); and building resources and building on resources built (by, for example, ensuring that parents and civic and religious leaders are systematically reached and included in delivering activities).

The report appendices include documents related to the evaluation such as the Terms of Reference, the Inception Report (containing the Theories of Change for both projects), information on people consulted for the evaluation, and Tikambe and Radio Waves baseline, midline, and endline overviews.