SETARA Impact Evaluation Report: Global Early Adolescents Study Indonesia Wave 2-3

"The SETARA evaluation shows the important contributions that SRHE [Sexual and Reproductive Health Education] can make in promoting more equal gender attitudes."

This report, published by the Center for Reproductive Health Faculty of Medicine, Public Health, and Nursing at the Universitas Gadjah Mada, presents the findings of the evaluation of SETARA, a 2-year comprehensive sexuality education (CSE) intervention for adolescents (12-14 years) in junior high schools in Indonesia. Looking at the impact of the programme in three different contexts, the study identifies specific factors that led to the success (or lack of success) of the programme, offering insights for planning and carrying out initiatives that cater to the health needs of adolescents.

The SETARA programme (SEmangat duniA RemajA/Teen's Aspirations) is a rights-based, teacher-led reproductive health, and sexuality curriculum for grades 7 and 8 implemented in junior high schools in three different locations in Indonesia: Bandar Lampung (Sumatra), Semarang (Central Java), and Denpasar (Bali). The curriculum consisted of chapters that cover the topics of puberty, gender, decision-making, relationships, violence, mental health, and sexual and reproductive health. In Bandar Lampung and Denpasar, teachers on average delivered 1 session per week (40 minutes). In Semarang, teachers implemented two sessions per week (2 times 40 minutes).

The report outlines the theory of change (ToC) for SETARA, which informed the evaluation. In order to support adolescents' healthy development, SETARA focuses on the development of competencies that help adolescents understand the cognitive, emotional, physical, and social aspects of sexuality in an age- and developmentally appropriate way. Specifically, the programme seeks to: improve knowledge about sexual and reproductive health ("sexual literacy"); develop more empathy and gender-equal attitudes; develop a common understanding of 'consent'; improve negotiation and communication skills; improve critical reflection skills to help adolescents consider different messages, pressures, and expectation (e.g., from peers, media, family, society); and improve coping skills, including managing emotions. In the ToC, these competencies are referred to as healthy sexuality competencies and are seen as short-term outcomes. These competencies are foundational for health and well-being more broadly, but they are also believed to be central to supporting adolescents' healthy and positive sexuality development.

Overall, the competencies enhanced through SETARA are: first, to strengthen belief in the self, and second, to increase self-efficacy and agency to make informed decisions about sexuality and act on these decisions. These competencies help adolescent boys and girls to understand and communicate about their boundaries and respect the boundaries of others. Getting correct information can help them feel more comfortable with the puberty changes they are going through, feel more comfortable with their bodies, and reduce unnecessary anxieties and guilt about their emerging sexuality. It is also expected that a well-implemented SETARA curriculum can contribute to decreased rates of peer violence and bullying, with a positive impact on mental health and sense of well-being among adolescent boys and girls. For those who initiate romantic and intimate (sexual) relationships over time, the idea was that the intervention could have a positive effect on the quality of these relationships or interactions and contribute to reducing sexual and gender-based violence (GBV), improving use of contraception, and preventing unplanned pregnancies and sexually transmitted infections (STIs).

Based on the ToC, the evaluation was designed around the following outcomes: 
 

1. Competencies for healthy sexuality, including: sexual literacy, gender norms and attitudes, and interpersonal skills
2. Personal sexual well-being
3. Relational well-being
4. Mental health

The evaluation was conducted by the Global Early Adolescent Study (GEAS) as part of Rutgers Indonesia's Explore4Action programme. To measure the effects of SETARA, GEAS prospectively followed adolescents who received SETARA (intervention group) and adolescents who did not receive SETARA (control group) at their junior high schools in the three locations from 2018 to 2022. The locations each had very different contexts in terms of religious and cultural values, with some locations being more liberal than others, as well as socio-economic differences that are discussed in detail in the report. The study adopted a non-randomised experimental design to evaluate the impact of the intervention and conducted an implementation research and qualitative evaluation to understand how SETARA was implemented in the different contexts and how the intervention group qualitatively perceived the impacts of SETARA.

The report presents the findings for the three locations separately because of their different contexts, implementation, and results. It also presents the findings on the effects one year after receiving the SETARA programme (wave 2) and two years after the end of SETARA (wave 3) in comparison with the baseline (wave 1).
 
The following are some of the key results as highlighted in the report as per the four main intervention outcomes: 

1. Impact on key competencies for health sexuality
 

• SETARA improved knowledge of pregnancy and HIV prevention among early adolescents in Bandar Lampung and Semarang. In Semarang, the increase in knowledge on where to access contraception is sustained through wave 3.
• SETARA reduced support for stereotypical gender traits (GST) and roles (GSR) in Denpasar only. In Semarang, the effect was observed around equality in household chores, responsibility for pregnancy prevention, and reduction of stigma of women who carry condoms and support for GSR.
• SETARA increases adolescents' discussion on sexual and reproductive health topics. The largest improvement is in Semarang, with increased discussion of four topics (body changes, pregnancy, contraception, and sexual relationships) and then in Bandar Lampung of two topics (contraception and sexual relationships), with Denpasar of only one topic (sexual relationships).

2. Impact on personal sexual well-being - In Bandar Lampung, SETARA improved girls' pride about their transition into womanhood in wave 2 and wave 3 evaluations. The same effect was also found in Semarang at wave 3. Conversely, in Denpasar, body comfort indicators worsen after SETARA administration.

3. Impact on relational well-being
 

• SETARA decreases support on GBV in Denpasar only, specifically in agreement with the statement that it's okay to tease a boy who acts like a girl. No significant difference was noted in GBV experience between the intervention and control groups.
• SETARA reduced the likelihood of becoming a physical bully victim and perpetrator in Bandar Lampung. A significant reduction of verbal victimisation and perpetration was observed in Bandar Lampung and Denpasar. However, this change was not due to the administration of SETARA, for the reduction in the control group was bigger than in the intervention group.

4. Impact on mental health
 

• No evidence of desired impacts of SETARA on general and mental health was found after three years of observation.
• In Bandar Lampung, there was a significant increase in the proportion of adolescents experiencing depressive symptoms in wave 3.
• The level of tobacco use was high among adolescents in Bandar Lampung and Semarang, whilst the highest level of alcohol use was in Denpasar. The level of addictive substance use continues to increase over the course of the study, with no difference between SETARA and control students.

The report reflects on the findings and offers possible explanations for the varying results and how they may have been influenced by the different contexts and the different ways the programme was implemented in the three locations. For example, around gender attitudes (under the outcome for competencies for healthy sexuality), the implementation research revealed that many teachers found that the gender sessions were difficult to implement, and these were among the most skipped sessions. SETARA teachers viewed this topic as sensitive and were afraid of explaining it wrong or experiencing backlash from other (male) teachers. The questioning of established gender roles to promote gender equality as outlined in the curriculum runs against cultural and religious beliefs that define strict boundaries between the roles of men and women. As a result, the sites with an Islamic majority, Bandar Lampung and Semarang, struggled more with applying a gender-transformative approach than the primarily Hindu-practicing site, Denpasar. 

The findings also showed that there was not much of an intervention effect on personal sexual well-being outcomes, except for girls feeling proud of their bodies as they become women and on self-efficacy to prevent pregnancy. According to the report, the lack of effect on outcomes related to body comfort may be related to the heavy influence of social media on (young) adolescents in the three locations. In addition, the limited effect of the programme on students' sense of personal sexual and relational well-being, as well as mental health, could show that factors underpinning peer violence and mental health are embedded in larger social structures and hard to address through CSE alone.

Overall, the effectiveness of SETARA appears to be highly contextualised and related to varying levels of implementation quality and sessions delivered. This finding shows the importance of considering teacher-level factors around values and skills. Therefore, it is necessary to understand the local realities of teachers, such as work pressures, cultural norms around intergenerational communication, unfamiliarity with learner-centred teaching approaches, and taboos and shyness around sexuality topics.

In conclusion, the report highlights the following main implications for CSE and gender-transformative approaches for very young adolescents more generally. 
 

• Start early with CSE: Results from waves 1-3 indicate that programmes like SETARA can build assets for healthy adolescent development for young adolescents. Knowledge and skills gained through SETARA become more salient over time, and adolescents seem to continue to strengthen their skills to communicate about SRH topics even after the intervention has ended. These findings confirm that SRHE helps prepare adolescents for the future and makes the case to invest in building these competencies early.
• Use an ecological approach to address gender inequality and peer violence: Results from  the study show that CSE can improve adolescents' gender attitudes to become more equal. But results are better and more sustained in a socio-cultural context where the broader gender system is more open to discussion and questioning of norms around gender and sexuality. This fact underscores the call for multi-level interventions to address gender inequality by working with teachers, schools, parents, and community members to ensure that their messages about gender and sexuality are aligned with key themes of the intervention and that adolescents are enabled to act in line with their improved gender-equal attitudes.
• Keep in mind implementation quality and dose when scaling up: The evaluation data show the importance of quality of implementation and dose delivered, next to other contextual factors. For example, the implementation research on SETARA found that good introduction, coordination, and collaboration with local government offices was a key factor for the success and scale-up of SETARA in Semarang and Denpasar. Furthermore, capacity of local implementation organisations needs to be considered in supporting schools and governments to implement CSE, and further collaboration might be needed to address structural barriers towards healthy and safe schools and to make linkages with other programmes and services.