Using Big Data analytics to establish relationships between conversational properties and self-reported behaviors and to design proxy indicators to predict behavior change in youth exposed to SBCC intervention on Financial and Sexual Reproductive Health

Summary:

Methodological pluralism is the cornerstone of social sciences that inform SBCC. Yet, for a long time survey research has been used as the only reliable and rigorous way to understand individuals/groups targeted by SBCC interventions. New technologies offer approaches complementary or alternative to survey research; they allow collecting organic, voluntarily-provided data to understand social-phenomena dynamics in real time. Since 2015, Shujaaz has been working with Africa's Voices Foundation (AVF) and Bankable Frontier Associates (BFA) to innovate on the analysis and use of the digital data organically generated on social media and SMS platform by 2m+ Shujaaz media fans in Kenya. By using digital data in conjunction with survey data, the collaborators identified a collection of behavioural, attitudinal and conversational characteristics that can predict if an individual (1) has a potential to be financially independent as a micro-entrepreneur, and (2) is likely to use modern contraception in all or most sexual encounters. We called the first collection the Index of Propensity to Financial Success (iPFS) and the second -- the Index of Positive Sentiment towards Modern Contraception (iPSMC). The hybrid nature of the indexes (a combination of survey data and digital-conversation data) allows for a diverse use: For example, as a part of impact-evaluation in pre- and post-surveys, and as an element of the persuasion strategy design (e.g., focusing on the proven set of necessary characteristics), and as a tool for weekly/monthly dashboard for tracking targets for the entire organization.

Background/Objectives:

The main goal of the study was to fill the gap of reliable SBCC evaluation approach that can be deployed frequently (e.g., weekly), and cost-effectively to forecast/predict changes in the norms and behaviours of participants exposed to an SBCC intervention without compromising the M&E rigour. The two objectives were to (1) examine whether changes in social norms and behaviours can be identified, tracked, and documented for particular segments of population using digital conversation data; and (2) determine if data from digital conversations can be combined with survey data to predict shifts in self-reported norms and behaviour.

Description of Intervention and/or Methods/Design:

Each of these studies combines two datasets: In addition to conversational data from Facebook and SMS exchanges with the Shujaaz fictional characters, the iPFS relies on an SMS survey of 2,000+ Shujaaz fans; and the iPSMC uses panel data from a cohort of 750 youth interviewed face-to-face in 2016 and 2017. Among other advantages, the combined data sets enable (1) a fuller analysis of temporal sequence critical for evaluating a SBCC campaign, and (b) a predictive power that help forecast the overall impact and conduct damage-control in real time. In this regard, population surveys are limited because they collect information at a snapshot in time and rely on recall of past media exposure and past behaviors. The two studies presented here are the first attempts to link digital data with traditional individual level, population-based data, which is a novel approach to utilizing the strength of the two different data formats.

Results/Lessons Learned:

The researchers first used the two Indexes to further validate the Shujaaz Theory of Change by noting a sequential evolution in digital conversations from exposure to Shujaaz media, to increased trust, to more frequent conversations about contraception and finance and ultimately to shifts in norms and behavior. Next, repeated assessment of the financial status of a cohort of youth 12 months apart using iPFS as an evaluation tool confirmed the predictive accuracy of the tool. Finally, the regression analysis found positive associations between high iPSMC and positive beliefs about condoms, more frequent conversations about contraceptives with peers and adults and reported contraception trials.

Discussion/Implications for the Field:

This hybrid methodology is well aligned with the recent shifts among research practitioners to experiment with blending methods and modalities and to extend the rigour of survey data analysis to non-numerical types of information including text, speech and image. As new and more powerful analytical methods are becoming available to the development sector, the Shujaaz experience has proven the new approaches to metrics and evaluation can be rigorous, analytically powerful and cost effective, i.e., accessible to a wider range of development agencies regardless of their M&E budget.

Abstract submitted by:

Anastasia Mirzoyants - Well Told Story
Ashrul Amin - BFA Global
Gladys Thyen 
Claudia Abreu-Lopez - University of Cambridge