Data Quality on Expanded Programme on Immunization in Pertussis Outbreak Affected District: The Case of Dara Malo woreda, Southern Ethiopia

Background: The aim of this study was to investigate the quality of immunization data and monitoring system in the Dara Malo district (woreda) of Southern and (SNNPR) of Method: a cross-sectional study was conducted from August 4 to September 27, 2019 in Dara Malo District. The district was purposively selected during the management of a pertussis outbreak based on the hypothesis ‘there is no difference in reported and recounted immunization status of children 7 to 23 months in Dara Malo District of Ethiopia’. A quality index (QI) and proportions of completeness, timeliness and accuracy ratio of the first and third doses of pentavalent vaccines and the first dose of measles-containing vaccines (MCV) were made. Results: in this study 336 month-facility data were extracted. In addition, 595 children aged 7 - 23 months, with a response rate of 94.3%, were assessed and compared for immunization status using register and immunization cards or caretakers’ oral report through the household survey. The ratio of vaccination data compared using tallies against the reports showed evidences of over reporting with 50.8%, 45.1% and 46.5% for first pentavalent, third pentavalent and first dose of measles vaccinations, respectively. Completeness of third dose of pentavalent vaccinations were 95.3%, 956% and 100.0% at health posts, health centers and at the district health office, respectively. The QI scores ranged between 61.0% and 80.5% for all five categories, namely, 73.0% for recording, 71.4% for archiving and reporting, 70.4% for demographic information, 69.7% for core outputs and 70.4% for data uses and were assessed as suboptimal at all levels. Conclusion: The immunization data completeness was found to be optimal. However, in the study area, the accuracy, consistency, timeliness and quality of monitoring system were found to be suboptimal. Therefore, poor data quality has led to wrong decision making during the reported pertussis recording,

data uses and were assessed as suboptimal at all levels.
Conclusion: The immunization data completeness was found to be optimal. However, in the study area, the accuracy, consistency, timeliness and quality of monitoring system were found to be suboptimal. Therefore, poor data quality has led to wrong decision making during the reported pertussis outbreak management. Enhancing the capacity of healthcare providers on planning, recording, archiving and reporting, analyzing, and use of immunization data for evidence-based decision making is recommended.
Immunization is recognized as one of the most successful and cost-effective public health interventions to ensure the healthy lives of children. Every year, vaccines save 2-3 million lives, while millions more are protected from disease and disability. Immunization is one of the best buys in global health strategies and has contributed to achieving Millennium Development Goal 4 (MDG 4) three years ahead of target, reducing the child mortality rate by 23% in Ethiopia [1]. Immunization improvements also contribute to achieving the Sustainable Development Goals (SDGs) [2]. Ethiopia recognizes the crucial role immunization plays in child health and affirms its responsibility to ensure that every child is protected from vaccine-preventable diseases.
In 2017, about 85% of infants worldwide received 3 doses of pentavalent vaccine (Diphtheria, Pertussis, Tetanus, Hepatitis B, and Hemophilus Influenzae type b). However, notwithstanding the progress that has been made, lower-middle-income countries continue to encounter a substantial burden of vaccine-preventable diseases, and their vaccination coverage is lagging behind that of upper-middle-income countries [3], leaving 19.9 million children vulnerable to vaccine preventable diseases [4]. Among those, around 60% are in 10 developing countries including Ethiopia [5].
Studies have found the high burden of vaccine preventable diseases among vaccinated children due to low vaccination coverage rate, inappropriate timing of the vaccination, and inadequate potency of the vaccine [6]. The average three doses of pentavalent vaccine coverage in low-income countries was 15 percentage points lower than that of high-income countries in 2011 [7]. Based on 2017 WHO and UNICEF estimates of national immunization coverage, the three doses of pentavalent vaccination in Africa was 72% [5] The Expanded Program of Immunization (EPI) in Ethiopia, which was launched in 1980, has shown steady progress in increasing coverage for all antigens [8]. Additional vaccines beyond the traditional ones, Hib and Hep-B in 2007; PCV in 2011 and MCV2 in 2019 were included in the routine immunization program (Additional file1) [8]. The study area, Dara Malo district, in 2017 and 2018 had reported 103.0% and 94.8% for three doses of pentavalent vaccination, respectively [9]. Despite the reported high immunization coverage, from August 2018 through to January 2019, there was an outbreak of whooping cough, with 1840 cases and six subsequent deaths. The outbreak investigation team hypothesized that poor data accuracy and poor quality of monitoring systems were the most likely reasons for the occurrence of the outbreak. The team observed discrepancies between the reported and household survey coverages [10]. Therefore, conducting DQS would have helped to strengthen the community-based surveillance system and identify activities that can improve the routine EPI services in the district [11]. This study aims to investigate quality of immunization data and monitoring system in the Dara Malo district of Gamo Administrative Zone, SNNP region of Ethiopia.

Study area and population
Dara Malo is one of 160 districts administered under the SNNP region in Ethiopia. The district has 24 of the lowest administrative structures which consist of one urban and 23 rural villages [9]. The facility-based survey collected from 23 health posts; four health centers and a district health office. The household data collected from sampled households. The sample size was determined using single population proportion formula with the assumptions of 53% proportion of third dose pentavalent vaccination coverage, 5% marriage of error with 95% confidence interval and 10% nonresponse rate [12]. The calculated sample size was 631.

Data collection and analysis
The WHO (2005) recommended self-assessment tools were customized for this study [11]. The tools were adapted to capture accuracy, completeness, timeliness of routine immunization data from different sources; namely, registers, tally sheets, household survey, and reports. In addition, the data quality monitoring tools help to capture the availability of essential supplies like vaccines, diluents, syringes, vaccination cards, tally sheets and registers.
The data were collected by twelve trained health professionals with Bachelor of Sciences (BSc.) level of educational achievements whom had experiences of document reviews. In addition, two trained health workers with master's degree in public health oversaw the quality of collected data.
A two-day training was facilitated for data collectors and supervisors on the objective of the study, data collection methods and ethical issues. Collected data were checked for completeness and consistency on a daily basis and data were double entered for consistency.
Facility-based survey data on first and third doses of pentavalent and first dose of measles containing vaccines were reviewed from July 2018 through June 2019. In addition, trained assessors evaluated the data quality monitoring system using WHO's (2005 ) DQS tools [11]. Finally, household surveys were conducted to collect primary data on immunization status for children aged between 7 to 23 months.
The data were entered into a spreadsheet, cleaned, summarized and analyzed using Microsoft Excel® and Statistical Packaged for Social Sciences Research (SPSS IBM V 20) [14]. Accuracy ratio, proportions of completeness and timeliness were calculated. The definitions and assumptions are presented below.

Accuracy ratio
Based on WHO's definition, data accuracy is the ratio between the number of vaccinations verified or re-counted from a source at one level (numerator), compared to the number of vaccinations reported by that level to more central levels (denominator) [11]. In this study, first and third doses of pentavalent and the first dose of measles vaccination recounted from the register, tally sheet, and caretakers' report during household survey were taken as a numerator and report of health posts, health centers and woreda health office then submitted to the next higher level were considered as denominators. To characterize the report consistency, national recommended categories were used i.e. consistent if the ratio reported was ≥ 90.0 and < 110.0%; overreported if it was < 90.0%; and underreported if the ratio was ≥ 110.0% [15]. In addition, the overreporting health entities were further classified with reasons for inconsistencies, namely, missing tally sheets, discrepant recounted tally sheets against reports and both missed and/or discrepant reports.

Completeness
According to WHO's definition data completeness is a percentage with the number of reports received in the numerator and the number of reports expected during a period of time as a denominator [11].
In this study, the number of the health posts, health centers and woreda health office reports received at the next higher-level health system tier for 12 months (July 2018 -June 2019) were computed against the number of expected reports [15].

Timeliness
Defined by the WHO as a percentage with the number of reports that were received on time (using a deadline set by the EPI office) as the numerator and the number of reports expected during a period of time as denominator [11]. In this study timeliness was considered for health post, health centre and woreda health posts if there was an evidence of written a date at least in once in each report in line with the national health information management system recommendation received before 22nd -23rd and 26th day of each month at the next reporting level for 12 months (July 2018 -June 2019) were considered to evaluate timeliness at each level [15].
Quality Index WHO's (2005) DQS tools, has 45 elements of variables for health facilities and 51 elements of variables for district health office. This tools are adapted for this study and assessed by trained data collectors on the quality of the immunization monitoring system [11]. The variables were categorized into five; namely, recording; archiving and reporting; demographics; core outputs or analysis and evidence of use of data for action. Trained data collectors provided scores for each question. A "no" response scored "0"; and "yes" response was scored from "1 to 3" based on its importance; and a "NA" response was not considered in the denominator. The QI was calculated after summing up all scores of evaluators and divided by sum of possible maximum scores in five categories [11].

Ethical Clearance
Ethical clearance was obtained from both JSI Research & Training Institute, Inc. and the SNNPR Health Bureau, Ethiopia, Institute Review Boards (IRBs). Permission to conduct the study was obtained from facility managers. During the household survey, 665 and 278 children who missed their pentavalent and measles vaccination schedule were immunized.

Accuracy ratio
Three hundred-and thirty-six facility months' data were extracted. In addition, 595 children aged 7-23 months, with a response rate of 94.3%, were assessed and compared for immunization status between register and immunization card or through caretakers' oral report on household surveys. At the district level, the proportion of the re-counted vaccination data on EPI registers for the first dose of pentavalent was 95.2%, 104.2% for the third dose and 98.6% for measles (Fig. 2). The ratio of vaccination data compared using tallies against the reports had evidences of overreporting which were 50.8%, 45.1% and 46.5% for first pentavalent, third pentavalent and first dose of measles vaccination, respectively. Eighteen (64.3%) facilities were classified as discrepant recounted and the rest 10 (37.3%) were both missing and discrepant recounted facilities (Fig. 2).
Completeness and timeliness of the reporting Table 1. below illustrates the completeness and timeliness of reports on the third dose of pentavalent and first dose of measles at primary health care entities. The completeness of both the third dose of pentavalent and measles ranged from 95.3-100.0%. The lowest completeness report was for health posts at 95.3% and the highest completeness percentage was 100.0% for district health offices. The timeliness of immunization reports ranges from lowest 0.0% for the district health office, followed by 55.8% for the health posts and highest 66.4% for health centers.   Fig. 3 illustrate the quality index (QI) scores using five major broad categories; namely, recording, archiving and reporting, demographics, core output and data use. The lowest QI score was 61.0% for recording at the health post; and the highest QI was 79.2% for demographic information at health centers. The quality of monitoring system for the district health office ranges from 69.7% on core outputs or analysis to 73.3% for recording category.

Discussion
This study evaluated the immunization data accuracy and quality of monitoring system using oneyear data extracted from 23 health posts, four health centers and a district health office. The result has provided objective evidence showing the existence of wide differences between reported and recounted immunization status of children in Daro Malo district. EPI data used to assess the performance of Dara Malo district were found poor in accuracy, consistency, completeness and timeliness. This might have led the health system to misunderstand the quality of services as the district health system lacks capacity to effectively use quality of immunization monitoring system for evidence-based timely decision making and response for vaccine preventable diseases outbreak [16][17][18] Hence, the most likely reason for delayed identification, verification and control measures against the reported pertussis outbreak might be related to the poor immunization data quality and its use for health improvement in Daro Malo districts [19].
The accuracy ratio calculated between routine immunization reports and evidence of hand-written information on the EPI register was consistent with the Ethiopian national recommended good quality category [15]. However, there were wide discrepancies between these routine immunization reports, tallies and household survey data. The lowest over reporting was revealed as 60.8% for the first dose of pentavalent vaccination and the highest over reporting was 44.9% for the third dose of pentavalent which were documented in health centers. Similarly, over reporting was observed at health posts, and at the district health office level. The high level of over reporting might have occurred due to lack of supplies. This was consistent with the observed availability of tally sheets in two thirds of primary healthcare entities (64.3%;18/28). This finding was much higher than the reported poor data concordance between child registers and facility tally sheets [20][21][22][23].
The timeliness of immunization reports ranged from zero percent at district health office to 64.6% for health centers. This finding was in line with the observed schedule for arranged outreach immunization services which overlaps with the reporting deadlines schedule in all 12 months. Since there were no static immunization services in the district, the health office prolongs the reporting period to include most recent activities until the health posts and health centers complete outreach services and submit the latest reports.
The completeness of immunization reports was found from 95.0-100.0%. The results showed that almost all reports during the study period were available in the next higher reporting level of primary health care entities. This finding was consistent with high rate of completeness reported in Iran [24].
The study shows that in the health posts, health centers and in the district health office, there are problems with data recording, analyzing and use of data for evidence-based decision making at the point of data production. The results of QI in all its five major categories, namely, demographic information, recording, achieving and reporting, core-outputs (analysis) and use for data for evidencebased decision were suboptimal at all primary health care entities. This could be explained by a number of influencing factors, to mention some: lack of capacity and skill to improve performance of essential health services, poor technical support, lack of verification, motivation and rewarding mechanisms at system level. This finding was consistent with quality index of 55.0% report from Tunisa by Chahed et al., (2013) [25]. Similarly, Yawson et al., (2017) confers that in Ghana, more than two thirds of districts were challenged to achieve effective coverages [26]; Machingaidze et al., (2013) [21] and Bosch-Capblanch et al., (2009) identified that shortage of skilled manpower which affects the strength of immunization system in Africa and 41 low-income countries, respectively [27].
But, this The result of the facility and community-based survey revealed that there are areas that need improvement at the primary health care entity level in terms of accuracy, timeliness and immunization monitoring system.

Conclusions
The immunization data completeness was found optimal. However, in the study area, the accuracy, consistency, timeliness and quality of monitoring system were found sub-optimal. Therefore, the poor data quality has led to wrong decision making during the reported pertussis outbreak management.
Enhancing the capacity of healthcare providers on planning, recoding, archiving and reporting, analyzing, use of immunization data for evidence-based decision making is recommended. Availability of data and materials The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare that they have no competing interests.   Quality indices for five categories of quality monitoring system at health post, health center and district health office, September 2019.