The Drum Beat 64 - Evaluation of Communication Interventions

EVALUATION

1. Evaluating the impact of communication interventions on development issues is a challenge for most communication for change for change and development programmes and practitioners. The Soul City programme in South Africa has recently undertaken a major evaluation initiative of it's "4th Series". This is the first of two Drum Beats on that evaluation. This Drum Beat focuses on evaluation design and methodology. When the results and analysis from this evaluation process are complete there will be a second Drum Beat highlighting that information. Contact information for the relevant people in Soul City is at the end of this Drum Beat. Please note that this is summary/highlight information. The much more detailed insights and descriptions can be found through the links to the papers on the web site.

SOUL CITY - SERIES 4

2. PROGRAMME COMPONENTS: Prime time television drama; Prime time radio drama; Booklets; Partnership with National Network on Violence Against Women [NNVAW]; Advocacy campaign and Helpline.

3. THEMES: Violence Against Women; Youth sexuality/AIDS; High blood pressure; Personal finance

4. GOALS: Reach over 2/3 adult population; Stimulate constructive dialogue; Shift social norms; Shift knowledge, attitudes, practice; Inform on rights; Connect people to help; Mobilise communities; Advocate for speedy implementation of Domestic Violence Act [DVA]
Further Details: click here.

EVALUATION - KEY OBJECTIVES

5. IMPACT: To measure and capture change... 

...at INDIVIDUAL level - KAP; Perception of social norms; Interpersonal support; level of debate and communication
...at COMMUNITY level: norms/values; mobilisation; action
...at the level of SOCIETY - policy and implementation of policy; policymakers/opinion leaders - national, provincial, local; through knowledge/awareness/norms; general population; key role players
Further details: click here.

EVALUATION - DESIGN SUMMARY AND CHALLENGES

7. There are 7 major components to the evaluation design: National Survey; Media Monitoring and Analysis; Partnership Study on the NNVAW relationship; Institutional/Organisational 'scan'; Cost Effectiveness Study; Qualitative Impact Assessment; and Sentinel Site Studies with Multiple components. See diagramme depicting these components and their relationship to the measurement of the individual, community and social impact factors on page 13 of link below.

8. Some of the design challenges included: Multiple messages/number of themes; variable levels of exposure; Interaction between various media; Sifting through 'background hum'; Soul City as agent of change?; Increasing the power and scope of the design; Mirror the complexities of the intervention; Meaningful integration of qualitative and quantitative methodologies; Theory - operationalisation; Optimise the intervention specific nature of the design at all levels; Research Management - between & within studies.
Further Details: click here.

PARTNERSHIP WITH National Network on Violence Against Women (NNVAW)

10. Objective: Determine whether Soul City-NNVAW partnership had an impact on: creating a supportive environment; improving referrals and strengthening services for women who have experienced domestic violence; increasing awareness of/action on domestic violence on an individual, community and societal level, which could ultimately increase the safety and well-being of women.

11. Design includes: Research interviews with key informants; outcome model; content and discourse analysis; thematic analysis of qualitative data; studies on organisational impact and lessons learnt; impact on access to, and quality of services (helpline and services); impact of advocacy strategy; and quantitative analysis of Telkom data.
Further details: click here.

SENTINEL SITE STUDIES

12. Objective: Assess all of the indicators highlighted above, in-depth, in 2 very different settings.

13. Design: Included: Panel surveys; Collection of existing data; Media monitoring; Qualitative interviews; Community mobilisation interviews; Monitoring of local Network activities; and, Ethnography. There were 4 rounds of data collection with the same respondents throughout. Two sentinel sites were chosen [a] MAMELODI - large, sprawling township in Gauteng high density, mixed population, formal and informal areas, inadequate health & welfare facilities [b] EMAFAKATINI AREA - four small rural villages in Kwazulu Natal, very homogenous population traditional authority, little infrastructure or facilities, Each site ahd a panel of respondents representative of that site.
Further details: click here.

MEDIA MONITORING AND ANLYSIS

14. OBJECTIVES : [a] To assess the impact of the Soul City intervention on media coverage nationally and [b] a specific focus on media advocacy strategy in partnership with the NNVAW on overall Violence Against Women and Domestic Violence.

15. DESIGN: Content analysis of [a] mainstream media - print and electronic - including editorial coverage and [b] Community media in the Sentinel Sites. Searched for Violence Against Women, NNVAW, Women abuse, Domestic Violence, Soul City. Developed database for analysis. Tested reliability.
Further details: click here.

COST EFFECTIVENESS STUDY

N.B. This study is in the developemnt phase 

16. OBJECTIVES: [a] To acknowledge the differing views on measures of effect by representing progressive CE ratios from knowledge change to behavior change to final outcome estimates and [b] To complete this for three key themes, HIV/AIDS, Violence Against Women and Hypertension showing estimates of Soul City’s cost effectiveness as a prevention intervention for each.

17. DESIGN: Key Issues under consideration include: [a] Possible lack of comparator - comparator would logically be those not exposed to Soul City but possible that sample is not large enough [b] Whether there is additional benefit in trying to attribute effect to the different media types - data limitations as there is no weighting on which media type exposed to most and [c] Effect must be scale measured in order to result in a sensible 'cost per' result and co-efficients resulting from factor analysis ordinal may need to define cut-off for "good" knowledge and "good" behavior, for example.
Further details: click here.

NATIONAL SURVEY

18. Objective: Measure the impact of components of the intervention (largely) at an individual level of analysis.

19. Design: 2 national sample household surveys of 2000 respondents; Baseline (pre-test) survey in June 1999, before the Soul City intervention; and, Evaluation (post-test) survey in Feb 2000. The design will measure the same constructs in the two surveys, Monitor changes over time and compare response between Soul City audiences and others. A Multi-stage random stratified (by area and province) sample with 500 enumerator areas at stage 1 and 5 'stands' selected within each EA for stand 2. Included a loglinear analysis to test effect of exposure to Soul City, with access to Soul City, sex, race, age, education and area as input variables with Cross-tabulations in cases where analysis indicates possible influence of Soul City.
Further details: click here.

20. Analysis: Higher order theme analysis of this data will be through Correlation Analysis, Confirmatory Factor Analysis, Scale Development and Regression Analysis.
Further details: click here.

DISCUSS AND DEBATE

Please join the EVALUATION COMMUNICATION CHAT FORUM to discuss and debate any aspects of this evaluation design or issues, issues that it raises for you and your own evaluation experiences and challenges. Soul City staff and many members of the International Advisory panel to this evaluation are part of this on-line evaluation discussion forum.

SOUL CITY EVALUATION CONTACT PEOPLE

Many people contributed to this evaluation. Their names are included in the detailed presentations. The key contact people within Soul City are:

Shereen Usdin shereenu@soulcity.org.za Sue Goldstein suegold@soulcity.org.za

***

8 VACANCIES include EVALUATION within their post description
55 CONSULTANTS list EVALUATION amongst their skills

***

This Drum Beat edited by Warren Feek - wfeek@comminit.com The Chief Editor of The Drum Beat is Deborah Heimann dheimann@comminit.com