An Evaluation of the Educational Impact of the Sisimpur Community Outreach Project

This report examines Sisimpur, an educational television programme designed for Bangladeshi children between the ages of 3 and 7 years old. Designed by Sesame Workshop, the programme is supported by the Government of Bangladesh and the United States Agency for International Development (USAID). Nayantara Communications, a Bangladesh company, is responsible for the production of the show and its accompanying outreach materials. An independent social research agency in Bangladesh, MRC-MODE, was commissioned by Sesame Workshop to undertake this evaluation.

As detailed here, the educators, researchers, and producers on this Sesame Street co-production work with a curriculum that not only exposes children to academic areas like literacy, math, and science, but also incorporates messages about health, hygiene, respect, understanding, social relations, and cultural knowledge. Since many children in Bangladesh have little or no consistent access to television, Sisimpur's messages are also featured in community outreach materials designed for use in a variety of settings, including early childcare centres, pre-schools, kindergartens, and homes. Local Bangladeshi educators, writers, and illustrators work with Sesame Workshop to produce these materials in the form of outreach kits. The kits, which contain books, games, flash cards, and growth charts, are distributed to caregivers from low-income households through a series of 3 workshops conducted by specialised trainers from 12 non-governmental organisations (NGOs).

To evaluate the impact of the Sisimpur outreach programme - specifically, Kit 1 (distributed in November 2005) - researchers examined differences among control and intervention groups in health, hygiene, and nutrition outcomes using bivariate analyses and multivariate logistic regressions. The sample comprised a total of 1,801 participants: 600 caregivers and 600 children in the intervention group, and 301 caregivers and 300 children in the control group. Data from were collected from 4 sites where Sisimpur outreach training occurred: an urban location (Dhaka) and three rural locations. In addition, one-on-one interviews were conducted with 20 caregivers in the intervention group and 5 trainers who conducted training workshops. Household observations were also completed in 150 homes (100 in the intervention group and 50 in the control group).

Overall, the survey findings indicated that programme beneficiaries mostly belong to households with economically poorly educated household heads engaged in low-income work such as day labour, agriculture, and rickshaw pulling. The caregivers themselves also have low levels of education: nearly half of them are illiterate. Urban groups tend to have more education, although these differences only appear in the lower stratums of educational attainment. Further, 62% of school-aged children in programme households have never attended school. (78% of urban children had ever attended schools, compared to 58% of the rural children). Forty-six percent of current school-goers reported participating in formal education, while the remainder is enrolled in non-formal schools. In addition, the survey findings confirm that most households that participated in the outreach programme earn considerably less than the national average; families with lower incomes may lack access to materials needed for good hygiene and nutrition habits. Overall, 69% of programme caregivers reported that they do not have access to any basic services. Among those with access to one or more services, 30% have access to electricity, 11% have access to gas, and 3% have access to land-line phones. Almost all the programme beneficiaries who lack access to basic services come from rural areas: 90% of rural programme respondents reported they do not have access to any basic services. Overall, only 40% of caregivers have reported that they can watch TV on a daily basis. Sixty-nine percent of children said they watch TV (79% of urban children and 66% of rural children). Urban children usually watch TV at their own homes, while rural children usually watch at a neighbour's home. These findings are compatible with the intentions of the outreach programme design, which was sought specifically to improve hygiene and nutrition habits of children in vulnerable communities.

An investigation of programme caregivers' teeth-cleaning habits indicates that participation in the Sisimpur outreach programme is associated with increased availability of effective teeth-cleaning materials like toothbrushes and toothpaste in the programme households. These positive indicators are apparent in both urban and rural programme locations. Access to these materials is also linked to income levels.

Overall, 93% of programme caregivers reported that they clean their teeth more than once a day, as compared to only 49% of the control respondents. The data also show that programme participation is linked to positive teeth-cleaning habits of both urban and rural caregivers: more than 90% of programme beneficiaries reported cleaning their teeth more than once a day in both rural and urban locations whereas only about half (49%) of the individuals in the control group reported doing so. A chi square test indicates that there is a statistical association between participating in a workshop and having a higher frequency of cleaning teeth in both urban and rural areas. Further, while 79% of programme caregivers reported cleaning their teeth before going to bed, this same figure is only 24% for non-programme caregivers. Almost two-thirds of the programme children in the intervention group reported cleaning their teeth an average of 2 times every day, compared to one-fifth of the children in the control group. These differences are statistically significant. Children in the intervention and control groups were equally likely to brush their teeth in the morning, but those in the intervention group were more likely than those in the control group to brush their teeth at night. Eighty percent of programme caregivers reported that their children were in the habit of cleaning their teeth twice a day, compared to only 25% of the non-programme caregivers. Both urban and rural programme children displayed this "improved" behaviour more frequently than their counterparts in the control sample.

Programme caregivers were asked to provide opinions about whether they believed they had changed their teeth-cleaning habits because of the outreach programme. Ninety-eight percent of the programme beneficiaries replied in the affirmative. Seventy-six percent of the programme caregivers felt that good teeth-cleaning practices were needed to keep "teeth clean", while 43% opined that such practices kept their teeth in "good condition". Compared to the control group, a higher proportion of test respondents said that proper teeth-cleaning habits were required for "good health", "to prevent bad breath", and to "prevent dental caries".

Qualitative insights were gleaned from a series of interviews with caregivers. On the whole, evaluators found that most of the caregivers recalled basic health and hygiene information they learned during the training sessions. Almost all the caregivers interviewed reported implementing their newly acquired knowledge about health and hygiene in their daily lives. They mentioned practicing the behaviours themselves, as well as teaching and guiding their children about hygiene.

One of the programme's main goals was to improve beneficiaries' diets through messages focused on: carbohydrates, which was promoted as a food group that "gives energy", proteins (promoted as a food group that "builds the body"), and vitamins (promoted as a food group that "prevents diseases".) Consequently, researchers assessed the effectiveness of the nutritional component of the programme by determining the number of food groups test respondents consumed for each meal. Overall, the survey showed that a higher portion of programme caregivers than non-programme caregivers as consuming a 3-food-group based diet for all three main meals. The findings also showed that a higher portion of programme children (15%) had diets that covered the 3 food groups, compared to control respondents (9%). The subsequent investigations into the regularity of children's protein and vitamin intake also indicated positive results: Food items like eggs, milk, fish, and fruits were consumed daily by a considerable portion of programme households.

In-depth interviews conducted to supplement the information gathered in the quantitative survey revealed that most caregivers were able to recall some of the basic nutrition information they had been taught during the training sessions. Almost all the caregivers interviewed seemed to be trying their best to practically implement their newly acquired knowledge about nutrition. However, a sizeable portion of interviewees said they had failed to provide a sufficiently balanced diet for their children and their family because of their inadequate incomes.

In short, the evaluation found that the Sisimpur outreach initiative had a positive impact on participants' health, hygiene, and nutritional knowledge and practices. These findings remained robust despite controlling for key sociodemographic factors (like income and education) that are known to strongly influence such outcomes. Caregivers seem to have applied the educational messages not only to their own daily habits, but also to their children's lives.

The programme, however, did not appear to be linked to the likelihood of consistently using soap to wash hands before a meal or preparing food, or child behaviours such as frequency of bathing, use of soap when bathing, hair washing, or protein and vitamin consumption.

After investigating caregivers' and children's hygiene and nutrition behaviour, the next goal of the research was determine their use of the outreach kit.  First, researchers determined which outreach kit materials the programme beneficiaries still possessed at the time of the survey (flash cards, matching cards, two storybooks, board game, and growth chart). The materials in the kit - particularly the matching card game - were well received and well liked by both adults and children. Ninety-nine percent of programme caregivers reported having the outreach kit. Sixty percent of the programme caregivers reported using all the materials in the outreach kit, 36% reported using some of them, and 4% said they had never used them. In qualitative interviews, most caregivers felt that the outreach kit materials were handy in teaching their children about health, hygiene, and nutrition. In many cases, the materials persuaded the children to adopt behaviour they resisted previously. However, some of the participants complained about the books given; because they were illiterate, many of these caregivers felt that it was awkward to tell stories using pictures.

In qualitative interviews, 9 out of 10 programme caregivers found the workshops to be "very informative", while the remainder found them "informative". Furthermore, "All caregivers reported feeling deeply satisfied with their trainers. They described the trainers as attentive, helpful, and instructive. They felt that the trainers used effective teaching methods such as repeatedly reviewing the information for the participants' benefit, regularly answering participants' questions, and engaging participants by asking them questions....Some of the participants even said that their trainers had visited participants' homes to remind them to come to the next training session." Interviews also indicated that caregivers often retained the information taught at the workshops.

In conclusion, "Analyses indicate that the outreach program has been executed well and has improved caregiver's knowledge and awareness of health and hygiene issues. It is also found that household income and education level exert significant influences on the program's success. The program has a particularly high impact on behavior changes that require little financial investment.... Changing behavior is known to be a very difficult task: in light of this fact, the initiative has been especially successful."