Remedying Education: Evidence from Two Randomized Experiments in India

This 54-page evaluation presents the results of two overlapping 2-year randomised evaluations conducted in Mumbai and Vadodara, India, designed to evaluate ways to improve the quality of education in urban slums. The paper examines the results of two remedial education programmes developed by Pratham, a network of Indian non-governmental organisations (NGOs) based in Mumbai that works closely with the government. The authors note that little research has shown that interventions that purport to improve the quality of the learning experience actually increase attendance while also improving the test scores of those already in school. It is this void that evaluators of the Pratham initiative sought to address. That said, the authors stress that attendance rates in urban India are already high, so - though attendance is important - the programmes emphasise changing the paedagogy used in the classrooms so as to reshape the way that students are taught.

Specifically, as part of the first programme, Pratham hires young local women ("balsakhi", or child's friend) who have the equivalent of a high school degree from the local slum communities in which the schools are located. These women then provide remedial education to children in standard 3 and 4 who have not acquired the basic competencies of standard 1. Students in the remedial education programme are pulled out of class for 2 hours a day to work on literacy and numeracy skills. The strategy involves providing what the authors describe as individualised, nonthreatening attention to children who are lagging behind in the classroom. Pratham's thinking is that children may feel more comfortable with women from their own communities than teachers, who are often from different backgrounds. Furthermore, as the balsakhi's class size is relatively small, she may tailor the curriculum to the children's specific needs. Part of the goal of the balsakhi programme is to improve academic performance by making easier for parents to play a role in their children's education, by serving as an intermediary between parents and the school environment.

In the second programme, children in grade 4 participate in computer-assisted learning (CAL) featuring games to reinforce mathematic competencies. The students assigned to receive the CAL programme get 2 hours of shared computer time a week to work on math skills only. One day of the week they attend the class during the regular school day and their second hour of the programme occurs either directly before or after school on another day of the week. To develop the CAL programme, Pratham hired a team of instructors from the local community and provided them with 5 days of computer training. The purpose of the programme was to offer students access to a suite of educational software that closely followed the official math curriculum. In the first year of the programme, Pratham relied on internally developed and off-the-shelf software, and in the second year, they partnered with Media-Pro to develop programmes to cover missing competencies. The use of entertaining information and communication technologies (ICTs) was part of a strategy here for making school more attractive for students, thereby hopefully increasing attendance and, more importantly, changing how students are taught. The programme costs 722Rps per student per year (or about US$15); the software used in this programme is being used in other ongoing Pratham computer-based initiatives.

As part of the evaluation process, schools in Mumbai and Vadodara were randomly assigned to one of two groups: one group received the balsakhi programme in one grade and the other in the other grade. A simple cognitive test designed by Pratham (given as a pre-, mid- and post-test) was used as the metric for the programme's effectiveness. Grade 3 students in schools where the programme was allocated to grade 4 form the comparison group for grade 3 students in schools where the programme was allocated in grade 3.

Main Results:

• Neither programme had significant impact on attendance.
• The balsakhi initiative improved overall test scores by 0.14 standard deviations in the first year, and 0.25 standard deviations in the second year. The CAL programme lifted test scores by 0.35 standard deviations in the first year and 0.47 in the second. Treatment effects were similar across standards (grade level), gender, and city.
• Programme efforts were more effective for children initially at the bottom portion of the class (i.e. those actually removed from class for the remedial education), so the programme effectively decreases inequality in classroom test scores. The improvement in math scores for the bottom third of the children was 0.51 in the second year.
• Individuals who participated in the treatments directly benefited equally. In the CAL programme this means that gains were equally distributed across all students since all students participated, while in the balsakhi programme the benefits were concentrated in low-performing students since these were the most likely to have been sent to work with the balsakhi. According to the authors, these gains are quite dramatic, with students who actually worked with the balsakhi improving by 0.6 standard deviations.
• There were no benefits to stronger students of being in a class that received the balsakhi programme. These students experience a smaller, more homogenous classroom when their lower-performing peers were removed to work with the balsakhi, but this did not result in academic gains.
• The CAL programme is so much more expensive than making use of the locally available balsakhis that the balsakhi programme is 5 times more cost effective for math scores and 7.5 times more cost effective for math and language scores. (Cost effectiveness is based on improvement in test scores.)

"These results show that it is possible to dramatically increase the quality of education in urban India, a very important result since less than a third of Indian school children can read when they leave school. However, this is not likely to be achieved by simply increasing resources without changing the way teaching is conducted."