Evaluating the Experience of GAPS - A Methodology for Improving Quality of Mass Immunization Campaigns in Developing Countries

This paper examines a field-based methodology for predicting, in advance of an immunisation campaign, the sites which are most likely to have a pocket of unvaccinated persons and then using this information to improve planning, supervision, and evaluation of the campaign. To date, Geographic Assessment of Planning and Services (GAPS) has been applied in Nepal and Ethiopia. This paper evaluates these two applications of GAPS and makes recommendations regarding its future use.

GAPS was developed to respond to the fact that achieving a high percentage of vaccination coverage is necessary but not necessarily sufficient for the elimination or eradication of disease. The existence of pockets of under-vaccinated persons has allowed outbreaks of disease in countries that have achieved high levels of vaccination coverage. For this reason, information about coverage levels in the general population is not sufficient to help understand the risks of disease transmission.

GAPS is a rapid assessment method carried out in 3 phases:

• Improve planning before the campaign begins:
  1. Identify a geographic area - usually a district or province that will participate in a mass vaccination campaign.
  2. Identify sites within that area that warrant special attention through semi-structured expert interviews that follow a guide available in the manual [available from the authors; please see below].
  3. Review campaign plans for the sites selected for special attention.
  4. Address any gaps identified in the campaign plan for sites designated as high-priority. For example, if there is a gap in social mobilisation activities in a site, an non-governmental organisation (NGO) might conduct a community meeting or other health-promotion activities, such as a skit, rally, or parade. "By these actions, before the campaign begins, we may prevent problems in the campaign."
• Improve implementation and monitoring of the campaign:
  1. Decide - from information gathered during Phase I - which sites will be given priority attention for supervision during the campaign, given available resources for supervision (time, vehicles, and number of persons).
  2. Identify if there are any problems in implementing the campaign that may result in a site having a pocket of unvaccinated persons. Using a supervision checklist (usually one that is in use nationally), observe immunisations provided at any fixed posts and by house-to-house vaccination teams. Observe the quality of vaccine, equipment, and supplies and assess knowledge and practices of vaccinators using a convenience sample. Visit homes in a site using a convenience sample to check the vaccination status of persons in these homes and inquire about reasons for any missed vaccination.
  3. Present findings from the supervision to organisers of the campaign (e.g., district health officer, district immunisation programme manager, immunisation officers of World Health Organization [WHO] or United Nations Children's Fund [UNICEF]) and then participate with them in follow-up actions to prevent or address problems.
• Carry out the post-campaign evaluation and follow up:
  1. Decide what sites will be visited during the post-campaign evaluation. This decision is based partly on observations of the campaign by GAPS participants and campaign officials, partly on discussions with campaign organisers prior to the campaign, and partly on resources (manpower, time, money) available.
  2. Design a hypothesis test of the post-campaign evaluation. The hypothesis is that the campaign coverage in a site to be evaluated is unacceptable because the site has a pocket of unvaccinated persons (defined as 20% or more of the intended population unvaccinated in that site). Use lot quality assurance sampling (LQAS) techniques to classify whether or not the campaign coverage in a site is acceptable.
  3. Carry out the hypothesis testing in sites selected. In each household (randomly selected), interview caretakers to determine the vaccination status of each child aged less than 5 years and to identify self-reported reasons for why a child was missed during vaccination rounds.
  4. Participate in quality-improvement activities, then present evaluation information to organisers of the campaign - discussing with them about what can be done to correct any deficiencies identified, and then taking the needed action(s).

The authors examine 2 specific applications of GAPS: in one woreda (district) in the Southern Nations Region of Ethiopia and in one terai district in Nepal that shares a border with India. Both the applications of GAPS were carried out at the district level for improving the quality of mass polio-vaccination campaigns, although GAPS can be used for other types of campaigns. Researchers interviewed 3 campaign organisers following the GAPS application in Ethiopia, and 4 in Nepal - all of whom expressed the general belief that GAPS was useful and worth the effort. For example, a non-governmental organisation (NGO), district-level, respondent in Ethiopia said that GAPS "provided knowledge and skills...to find the effectiveness of some programmes. Also in that country, a United Nations (UN), national-level, respondent called GAPS "cost-effective time-wise and practical...it is an effective rapid assessment." And a government-, national-level respondent in Nepal noted that GAPS "gives the community awareness of the importance of vaccination, an appreciation of our interest in what happens there".

"In summary, although this evaluation was limited in the number of applications evaluated, GAPS appears to have promise as a practical method to help improve the quality of mass immunization campaigns in two main ways: (a) testing hypotheses about pockets of unvaccinated persons and (b) providing a population-based source of data to help triangulate administrative estimates of coverage. Further applications in different settings will be needed to confirm these findings and/or under what circumstances GAPS might be used for complementing the existing methods and procedures to improve mass immunization campaigns. GAPS may also be considered for improving other types of health campaigns, such as distribution of insecticide-treated bednets, vitamin A capsules, and deworming medications."