Assessing Community Change: Development of a ‘Bare Foot’ Impact Assessment Methodology

Published in the Radio Journal, this 11-page paper outlines a “bare-foot” impact assessment methodology that was designed, tested, revised, and implemented with 8 community-owned stations in Mozambique between 2000 and 2005. The purpose of this UNESCO/UNDP Mozambique Media Development Project was to determine whether community radio stations promote democracy and active involvement of communities, and whether they allow people to set their own development agendas. They also wanted to ensure that volunteer community radio producers would be able to carry out assessments by themselves beyond the project's end.

As detailed here, this methodology involves 3 central areas of assessment, each with an associated checklist. The first area, to be conducted twice yearly, involves ensuring that the radio works effectively as an institution and that all groups within the community are involved. An evaluator might pose questions about (and directly to) staff and volunteers, such as "what training has been received?". Work/action plans should be assessed, and programme sustainability gauged. The radio shows themselves should be evaluated by asking questions about content variation, relevance, local production, and the source of content. The extent to which the community is participating in the initiative (e.g., by coming to meetings), and their level of satisfaction with the programme, also need attention.

The second and third areas involve assessing the impact of community radio content (ongoing) and assessing the impact of radio on community development (ongoing and annual). A key strategy for understanding these factors is conducting interviews. These conversations could be informal (while out preparing programmes or doing other radio work) or formal (with people living near individual programmers, or with people during major public events). They might be carried out one-on-one, or they might involve focus group discussions (FGDs) with 6-10 people per group. These FGDs would draw on "distinct profiles", such as young women, young men, women in rural areas, men in rural areas, women in town-like areas, men in town-like areas, etc. In addition, evaluators could register the opinions of listeners that telephone in to the station. They could also register and analyse letters received from listeners, as well as responses to questions printed on the back of returned message slips (used to announce births, deaths, community events, meetings, and so on). A key thinking point is that the "bare foot" evaluator should keep identified problems at the forefront of organisation and planning (in Mozambique these are: food security; health; and security and infrastructure).

Excerpt from the id21 website:

"The impact assessment focussed on three sets of questions:

• Is the radio station working effectively internally and do the volunteers have contracts, rights and clearly defined duties?
• Do the programmes respond to the interests of the public? Are they well researched, using culturally relevant formats such as story telling, songs, proverbs and music? Are they considered good and effective by listeners?
• Does the radio station create desired development and social change (determined by the original baseline research) within the community?

'Barefoot' impact assessments of eight of Mozambique's community radio stations revealed both positive results and potential problems:

• Areas of Dondo, a town in the centre waiting for electricity for years, got it following an intense one month community radio campaign.
• The number of deaths caused by cholera in Dondo during annual flooding in 2004 dropped drastically to zero because during a cholera epidemic the radio broadcast information about, among others, the distribution of chlorine and the importance of putting it in the water.
• The number of people seeking HIV testing increased significantly after radio programmes created an environment where the subject could be discussed openly. Working on and listening to radio programmes also helped young people build up confidence to negotiate practicing safe sex.
• The radios' civic education campaigns resulted in increased participation, heightened debate and community control of election procedures.
• In one case most management functions had been filled by people from the Catholic Church and the assessment discovered that the radio was beginning to be referred to within the community as a Catholic radio, which was potentially divisive.
• One radio station had a high turnover among volunteers, motivating the radio management to discover why they were all leaving and what could be done.

'Barefoot' impact assessments can ensure that community radio stations are on track with their objectives. They can also provide feedback to the communities in which they are working and demonstrate their credibility to local and international funding partners. They need to be simple enough to be sustainable without external assistance and systematic; making sure that impact is assessed at all three levels outlined above."