If you can't measure it, you can't change it: Lessons learned in developing and utilizing social norms scales to evaluate a norms-shifting intervention

Summary:

Shifting social norms is a promising strategy for SBCC programs across a variety of sectors. However, there are critical gaps in our understanding of how to shift norms largely because the nature of norms make them difficult to measure. The Passages Project is working to expand the boundaries and effectiveness of SBCC approaches by establishing an evidence base on scalable social norms shifting approaches. Within the Passages Project, the Masculinite, Famille et Foi (MFF) intervention, implemented in Kinshasa, the Democratic Republic of the Congo (DRC), is seeking to improve family planning (FP) uptake and reduce intimate partner violence (IPV) through community-based activities promoting positive social norms in faith communities. The MFF intervention is being evaluated through a cluster randomized trial following a cohort of 900 young, partnered men and women from eight experimental and nine control Protestant Congregations. A key contribution of this project is the rigorous process that has been followed to develop and test new social norms scales. Through this process, we identified nine distinct social norms constructs related to FP, IPV, and household gender equity. In this presentation, we will describe the intervention and the evaluation with a focus on sharing insights and findings from the development of the social norms scale measures. We will outline the steps taken to develop normative measures, highlight findings on the independent effect of various social norms on FP and IPV outcomes, and discuss the performance of the measures and their potential for adaptation in other norms-focused projects.

Background/Objectives:

In partnership with a team of global health experts, the Passages Project is working to establish an evidence base on scalable social norm change approaches. Within Passages, the MFF intervention, implemented in Kinshasa, DRC, is seeking to improve family planning (FP) uptake and reduce intimate partner violence (IPV) through community-based activities and engaging religious leaders and faith communities to create normative environments that are more supportive of gender equity and FP use. In order to add to the evidence base on norms-shifting approaches, the MFF research team developed and tested new scale measures of social norms.

Description Of Intervention And/or Methods/Design:

We employed formative, participatory techniques to develop valid normative items about perceptions of social approval (i.e., injunctive norms) and community prevalence (i.e., descriptive norms) relating to FP use, perpetration and experience of IPV and engaging in gender equitable roles and responsibilities. At baseline, we asked these questions to 900 men and women in the MFF study congregations and inquired about the influence of a number of reference groups, including faith leaders, partners, and other congregation members. We then conducted exploratory factor analyses and identified nine distinct measures of social norms. At endline, we conducted confirmatory factor analyses (CFAs) with data from men and women in the same congregations. We assessed the reliability of all measures and the goodness of fit of the CFA models. Structural equation modelling was conducted to assess relationships among norms and between the norms, behavioral outcomes and other important concepts in the MFF theory of change.

Results/Lessons Learned:

Most measures had fairly strong Cronbach's Alpha Coefficients measuring their reliability, ranging from 0.70-0.85 (above the acceptability threshold of 0.70). The initial CFA modeling of the measures produced acceptable RMSEAs for both the male and female models (0.09 for each)and after specifying the items as ordinal items within the model acceptable RMSEAs were retained for both models (0.09 and 0.10 for women and men, respectively) as well as acceptable thresholds for the CFI (0.97 for each) and TLI (0.97 for each) for each structural equation model (SEM). These findings indicate that these scale measures are valid new measures of social norms related to gender equity, IPV and FP use but we will discuss the challenges and lessons learned from our experiences generating these scales and in investigating the complex relationships and pathways of influence between social norms, outcomes and intervention exposures over time.

Discussion/Implications For The Field:

Our new social norms scale measures showed strong internal consistency and reliability and identified differences by gender in norms of relevance and reference groups. Moreover, the SEM models revealed complex relationships between norms and among norms and outcomes. To date, there has been a dearth of rigorous evaluations of norms-shifting interventions largely due to the challenges of measurement. By sharing our scale-development process and our findings, we will impart important insights on measuring and thus evaluating norms-shifting interventions accurately and efficiently. This session is particularly relevant for researchers and practitioners seeking to measure and change social norms.

Abstract submitted by:

Betsy Costenbader - FHI360

Bryan Shaw - Georgetown University

Andres Martinez - FHI360

Courtney McLarnon - Georgetown University

Francesca Quirke - Tearfund

Seth Zissette - Wilson Sheehan Lab for Economic Opportunities at Notre Dame

Prabu Deepan - Tearfund