Process Evaluation on Community Health Information System in South-East Zone Tigray 2016

Back ground: As per the world health organization (WHO), Health Information System (HIS) is one of the six building blocks of the health system. Nations establish their HIS as per their context. Generally HIS regarding data management (the ability to collect, store, analyze and distribute data) is limited in Sub Saharan African countries (1).However, literatures found were not able to show the magnitude of the problems of the program implementation status. As the result of the of initiatives aligned with which to to decrease child mortality, maternal health, and combat and and diseases. So community health information System


Need of the Program
As per the world health organization (WHO), Health Information System (HIS) is one of the six building blocks of the health system. Nations establish their HIS as per their context. Generally HIS regarding data management (the ability to collect, store, analyze and distribute data) is limited in Sub Saharan African countries (1).However, literatures found were not able to show the magnitude of the problems of the program implementation status.
As the result of the announcement of WHO "health for all", global health initiatives started to improve community health aligned with millennium development goals which emphasized the need to strengthen the primary health care to decrease child mortality, improve maternal health, and combat HIV/AIDS, malaria, and other communicable and non-communicable diseases. So community health information System became public health issue in Ethiopia (2).
Cognizant of the importance of health information Federal Ministry of Health (FMOH) started reform of health information system and monitoring and evaluation (M&E) components to solve the data collection, reporting gaps and to strengthen the information Utilization (2).
As the result of the above scenario, FMOH started to strengthen the HMIS and its principles standardization, speci cation, integration so as to improve the data collection, summation, analysis and dissemination for decision and action. Here Community Health Information System was scaled-up along with HMIS (2).
Community Health Information System (CHIS) was designed to standardize data collection and integrate data systems to provide relevant information for decision-making at the health posts and to feed the HMIS on a regular basis. Family folder is a pouch, which is the main part of CHIS, is a data collection tool designed by the FMoH for Health Extension Workers (HEWs) to document both individual and household level data to be utilized as a source of information at the grass root level (3) Unlike the health centers in health post there were different types of unorganized registers which were used to collect information on services provided within a single health domain, sometimes requiring a large group of registers to cover all health domains related to population. In addition, there was no any mechanism to identify which group of people needs, which type of health care services .But family folder informs patient care through the data contained in individual and family records, identi es patients in need of care through a set of tickler les, and enables reporting through supplementary tally sheets and family folder cards. The overall design of the FF innovation was to simplify the work ow of the community health worker and focus health care delivery at the community level (4).

Statement of the Problem
Federal ministry of health took the lead in giving a due attention on implementation of CHIS and it also engaged various partners at country level; mainly John Snow, Tulane University, Italian Cooperation and World Health Organization (WHO) -in supporting the scale-up of HMIS in the regions (5).According to the health sector monitoring and evaluation strategic plan (Health Sector Transformation Plan) 2016-2020 (6), there were identi ed challenges of the CHIS implementation: Though Integrated child case management (ICCM) register is the only register permitted at health post level, some health posts use other registers like outpatient department (OPD), Expanded Program on Immunization (EPI), etc, incomplete and inappropriate recording and lack of updating FF data at health post. There was also inconsistent use of FF cards during service provision at health post and lack of FF tools revision to capture data of newly implemented services including EPI, nutrition. In addition, there was also shortage of pre-requisites required for CHIS implementation including cards, pouch, shelf and tickler le box, due to lack of knowledge and skill. Most health posts were not performing Lot Quality Assurance Sampling (LQAS) as per the standard; poor utilization of tickler le system for client monitoring (6). Accordingly assumptions have been focused on how well the CHIS is Implemented. However, there are no scienti c evaluations which can quantify the magnitude of the gaps of the system listed above. Moreover, there is limited information on the dimensions of the evaluation of CHIS (Completeness, Consistency, compliance and availability) in the study area. Therefore this study was conducted to ll the gaps by exploring the magnitude of the problem and status of the system regarding evaluation dimensions in the study area.

Rationale for evaluation
At the time of the evaluability assessment in Enderta woreda, the stake holders identi ed problems such as parallel reports & multiple reporting formats (not related to CHIS) that gives a burden to the HEWs and leads to ineffective implementation of CHIS.HMIS/CHIS. The stakeholders include: experts, planning process owner in Tigray regional health bureau, woreda health o ce heads, HMIS focal persons and HEWs. They discussed and accepted the evaluability assessment and on how they would use the ndings and results of this study. So during the assessment there were shortage of CHIS tools and in appropriate reporting and recording procedures that were also identi ed during integrated supportive supervision in the Tigray rural health posts. In the region about 98% of rural health posts have implemented the community health information system starting from December 2013 (5). Hence, it was very important to make a process evaluation on it.
This study described the scale-up (implementation) status of CHIS in four districts and clearly indicated the achievements, challenges, and experiences which were captured from the documents during the assessments. The assessment was considered as helpful to implement CHIS in the other zones. Thus, it would help decision makers at different level to minimize the problems, to improve the status of implementation of CHIS (program improvement) and ultimately to improve service delivery. In addition it would also be important as an input to expand the lessons learned in to the urban CHIS. Generally the rationale of this evaluation has an implication in improving the data collection & information use at grass root level. Moreover, it would be used as an input for other evaluators/researchers and for vital event registration especially for births and deaths.

Description of the Program under the Evaluation
Globally, Health information systems serve multiple users and a wide array of purposes that can be summarized as the generation of information to enable decision-makers at all levels of the health system to identify problems and needs, make evidence-based decisions on health policy and allocate scarce resources optimally. Data from different sources are used for multiple purposes at different levels of the health care system (7).
The countries in the developing world face a wide variety of health-related challenges, and the health systems that address those challenges are struggling with limited resources and capability. Health leaders must therefore focus on maximizing the value of scarce resources and nding ways to make health systems operate as e ciently as possible. Having reliable data on the performance of different parts of the health system is the only way to devise, execute, and measure health interventions.
Successful strengthening of health systems will require relevant, timely, and accurate information on the performance of the health system itself. The goal of a health information system (HIS) is to provide that information (8).
Ethiopia has established to have a nationally standardized comprehensive health management information system (HMIS) which is one of the cross cutting attributes in the health systems strengthening framework. HMIS is a broad system for collection, compilation, analysis and utilization of routine health service data at all level. Similarly CHIS is a system for collection, compilation, analysis and utilization of routine health service data at community (grass root) level. In the perspective of this setting, the government committed to promote a household centered health care through the community based health extension program (HEP). As part of strengthening the community health information system, the Federal Ministry of Health used the family folder (FF) as a tool of data collection and for grasping essential information at the family level. This collected information is used for decision making in terms of health promotion and disease prevention at the grass root level (9). Health cards are usually used when a client is sick and came to health post seeking medical service and be kept inside the family pouch. In addition we can record information regarding family planning services. It can be divided in to two types called Male Card which is colored by blue and Female Card which is colored by Yellow. Every member of the family who is ≥5 years of age is issued a health card; for those <5 years, their records are kept in their mother's Health Card till they reach the age of 5 years. Integrated Card is used to record services such as ANC, Delivery, PNC, New Born Care, and Immunization and so on. And it is issued to a mother who is pregnant, laboring or lactating mother and her new born (11).
A tickler le is one component of the FF shelf which is essential for CHIS implementation. It is used for tracing the defaulters from the appointment. Family folder shelf used to contain the family folder pouches and segregated into different kushets and house numbers. It holds also tickler le.
Tally sheets used for tallying the diseases and service delivered. There are four types of tally sheets service, disease, family planning dispensed count and tracer drug tally sheets.
Reporting formats used to collect the monthly and quarterly data are out patient (OPD) and Service delivery reporting formats.
Master family index is also one component of CHIS which is important for recording the house hold heads alphabetically and use to retrieve the family folder from the shelf.
Field book used to record the identi cation details and service data of clients receiving service at household or outreach and for whom the HEW did not carry the Family Folder.
Completing all components of the CHIS is a simple HMIS recordkeeping and reporting procedure that feeds community level health information (11).

Goal of Community Health Information system
The goal of the program is improving availability, quality and utilization of information so as to ensure quality of service delivery at community level.

Program Speci c Objective
The speci c objectives of CHIS are:- reporting formats are also required as part of resources. Skilled information experts, HEWs and supervisors including the capacity building are also vital components of the human resources. As part of the infrastructure and furniture, buildings

Methods
The study focused on process evaluation, on the implementation of CHIS in South-East Zone in Tigray. South East Zone was purposely selected due to the short distance to Mekelle that indirectly helped to reduce cost. Process evaluation, normative evaluation approach was used; Quantitative data was used to assess the implementation of CHIS in south east zone Tigray. A descriptive evaluation designs also used to describe activities which the program carried out. The sampling area was health posts found in south-east zone Tigray which were randomly selected from the nearest kushet to the health post and the Sample size was 634 family folders & respective houses. The dimensions used for the evaluation were: availability, compliance, completeness and consistency. These dimensions were helped to assess the CHIS implementation status in the south-east zone in Tigray.

Sampling Size Determination
During the study the total households registered for family folder program in the zone were 109,523. The sample size estimation was calculated by using n = Z 2 p (1-q) / w 2 . The study used 95% con dence interval and a signi cance of 0.5 to calculate the sample size. 50% P was taken because there was no speci c research evidences related to the research area.
Therefore the calculated sample size was 384 family folder cards and households. Hence, considering a design effect of 1.5 and non-response rate of 10%, the total sample size was 634. And these 634 FF cards and households were proportionally allocated to each heath posts in each woreda households that have family folder and were selected randomly from the nearest Kushet in each health post.

Sampling Technique
South east zone was selected purposely from the six zones in Tigray. There are four woredas in south east zone: According to the 2007 census projection, the region has a total population of 5,055,999 in 2015. Agriculture is the main means of subsistence for 85% of the Region's population. Irrigation and terrace farming are used on the steep slopes producing teff, wheat, barley, beans, lentils, onions and potatoes. The annual population growth rate for Tigray is 2.5% and the total fertility rate is 4.5 children/woman of reproductive age (mini-EDHS 2014).
Public health care services in Tigray are delivered through 1 specialized hospital, 15 general hospitals, 20 primary hospitals, 204 health centers and 712 health posts. The primary M&E tool used in the Region is the Health Management Information System (HMIS), which draws its data from routine services and administrative records.
Tigray has an estimated number of 930,135 rural households. There are more than 660 rural health posts working with 210 HEW supervisors and more than 1430 HEWs. Rural health requires a specialized approach, therefore, CHIS was designed to make the health information and decision-making effective and successful at the grassroots level.
The TRHB HMIS unit established a monthly reporting system at all levels, starting from the last quarter of 2012 on a monthly basis, each health post reports to the respective Health center, and each Health center reports to their respective Woreda health o ce. Every Woreda and hospital in the region reports aggregated data to the Regional Health Bureau on a monthly basis. The TRHB then sends a complete report monthly to the FMOH. The timeliness of the monthly HMIS report as a region is 81.7%, which is an improvement from 2013, but still below the standard of 90% (5). (See Figure 4)

Study period
The study was conducted from January 2016 to June 2016.

Study design
The study was conducted using cross sectional study design at facility and household level.

Source population
The source population was all family folders in south east health posts, Tigray from 2013 to 2016.

Study population
The Study population was all family folders in the south-east zone from the selected health posts from 2013 to 2016

Study Unit
The study units were the selected family folder in the south-east zone from the selected health posts.
1.9. Inclusion and exclusion Criteria 1.9.1. Inclusion Criteria All health posts that have implemented family folder in south east zone were included in this evaluation.

Exclusion Criteria
Health posts that do not have their own health posts (building) were excluded from the evaluation study.

Evaluation Type
A process evaluation type was used to assess the implementation status of community health information system in the study area. Descriptive evaluation design was used to check and describe whether the family folder activities were carried out based on the guidelines or not. Furthermore, the evaluation design checked the strengths and weaknesses of the implementation of family folder program.

Evaluation approach
The approach for evaluating the implementation of the community health information system program was normative approach. Normative approach was an appropriate approach of evaluation to assess if there is an improvement undergoing in a program which is under implementation. And it has used the compliance data quality dimension to check whether they used the guideline or not and, availability dimension was used to check the availability of all necessary CHIS tools.

Evaluation focus
The study has mainly focused on process evaluation which measures the implementation of the program & described how the program was delivered. It looked to uncovered management problems by discovering how a program was implemented and whether it was implemented as planned or met speci cations.

Data collection Tool and Data collection procedure
A self-administered questionnaire adopted from similar studies was used to assess knowledge of HEWs about the utilization of CHIS tools and check list was used to assess the availability and consistency CHIS tools. The quantitative data was

Data Management and Quality Control
The data was collected from 17 health posts and 634 households using self-administered questionnaire which customized from reviewed literatures. Data collectors were given training on how to collect the data from family folder and how to visit households. The questionnaire was designed in EPI-data version 3.1 to enter the collected data for analysis. The quality of data was checked by cross checking the collected and entered data during both the data collection and data entry by the principal investigator. The collected data was entered in to statistical software (epi-data) to check its completeness and consistency.

Data analysis
The data cleaned in EPI-data was exported to SPSS 20 for further analysis purpose. A descriptive analysis was done to show the frequencies and percentages of the observations. Tables and cross tabulation was used to describe the characteristics of the health posts with respect to the evaluation dimensions. Finally the results of the analysis was compiled and summarized for reporting.

Results
A total of 17 health posts were visited in this study to evaluate the implementation of the program, from these 17 health posts 634 family folders with their respective households were assessed in the four woredas of south east zone.

Compliance
According to the result of the study, 94% (16) of the health posts had provided family folder to their respective households found their catchment area. But only three of the four (75%) of the health posts in woredas Saharti-Samre provided family folder to their households in their catchment area.
The family folder/pouch were organized and shelved according to HH number &kushet which was aligned with the standard and its compliance was 100% in 17 of the health posts.
About 65% (11) of the health posts had provided health cards to their clients (providing health cards to a client when he/she came to health posts seeking treatment) according to the guideline. Speci cally, the provision of health cards to clients was 40% in Hintalo Wajarat, 25% in Saharti Samre and 100% in Dogua Tembien and Enderta woreda. All HPs in all woreda were retrieving FF from the shelf using MFI by his/her name, which was done according to the guideline and its compliance was 100%.
Regarding to the registration of household heads in the MFI, all household heads were recorded in the MFI alphabetically in all HPs and its average compliance was 100%.
During the study period all health posts have had tally sheets and were using it appropriately. They had used house number to record the services and diseases in both the service and disease tally sheets respectively according to the guideline and its average compliance was 100%.
According to the CHIS guideline, house number should be given 5 and 7-digits for each household member. In all HPs the house hold number was given 5-digits, and 82% of the HPs had given 7-digits to their household members. Particularly, health posts in Enderta and Saharti Samre woredas were using 7-digits to their household members with a compliance of 66.7% (2) and 75% (3)  Consequently, an average of 94% of the HPs updated FF to all new HHs, new births & deaths.
In general, the average compliance of all the health posts in south east zone was 92.54% during the study period.

Discussion
In this study, the four dimensions, availability, compliance, completeness and consistency were used to assess the implementation status of CHIS/FF in south east zone. They were graded as 30%, 20%, 30% and 20% respectively.
According to the standard of HSTP II under the information revolution transformation the availability of CHIS guidelines is considered as excellent if it scores above 95% but in this study the availability of CHIS guidelines was 76%, which was marked as good. Speci cally, it was, in Hintalo wajarat in 3/5 HPs, in Saharti Samre 3/4 HPs, in DoguaTembien 5/5 HPs, and in Enderta 2/3 HPs are available in each woreda. Even though, it was not stated in percent, in a study conducted in Amhara region by JSI and FMOH recognized the availability of guidelines was important for implementing the FF on a large scale (21).
This might be due to the guidelines were not distributed from woreda to each health posts and the HEWs were new and not trained on CHIS manuals. The availability (88.7%) in this study was consistent as compared to a case study conducted in Bungoma County Kenya which was 88.7%. The reason why the results of this study are similar with the study conducted in Kenya might be due to that they were conducted at a similar time and, the geographic and economic status of the countries is similar. The availability of shelves (100%) in this study was also consistent with a pilot test study entitled as "From Multiple Register to Family folder" (100%) and conducted in Ethiopia. And this achievement may be due to the concern of the government and stakeholders in providing the required amount of shelves to each health post.
Though the availability of health cards and integrated cards were good (88%) but it is still low as compared to the national CHIS guideline which suggested that all health posts must have all health cards. This indicates that there was a gap in registering all the needed information of clients, and under reporting& LQAS inconsistency. This might be due to the failure in distributing of all the necessary health cards from the districts to the health posts.
The availability of four types of CHIS tally sheets (service tally sheet, diseases tally sheet, family planning service delivery tally sheets and family planning dispensed count tally sheet) was 100%. Regarding the utilization of eld book, 30% of the HEWs were using it as a replacement of FF, while they were expected to use it independently. This shows that the HEWs were not well aware about the utilization of eld book.
In this study 94% of the respondents explained that they used tickler le system to order patient records according to the future date that requires a follow-up visit. After each household visit the HEW records a date for the next follow-up visit for the speci c household member. The health card is removed from the FF and placed in a box corresponding to the month of the next appointment. At the beginning of each month, the HEW reviews the cards in the box and prepares an agenda that includes providing appointment reminders or conducting household visits (Regional Health Bureau 2014). In complement to the above guideline, this shows that almost all the HEWs were using tickler les appropriately.
In this study, all health posts used tally sheet and 5-digits for house numbers in service and disease tally sheets. by the HEWs to generate the report and a minimum of one page and maximum fteen pages were required. But the main objective of the CHIS is to minimize parallel recording and reporting. So, this miss-use of the CHIS tools may be due to that the CHIS is not revised recently in a way that it could incorporate the interest of other stakeholders recording and reporting system.
As the study by Oslo University in Tanzania and Mozambique explained there were no standardized recoding and reporting formatting tools which resulted for preparing local recording and reporting tools and leading to duplication (14). The data collection tools do not meet either the report requirements or health facilities day-to-day functioning requirements. This implies when it is compared with CHIS in Ethiopia, it is lacking behind because CHIS has standardized recording and reporting formats but not meet all the requirements as in the case on Tanzania &Mozambique. And what is lled in the form is sometimes what the health worker thinks was done during the day, this contradicted with the assumption that tally sheets should be tallied daily(14).
Completeness: As this study revealed in some health posts Aynimbrkekin, Michael Abyi (D/Tembie) and Endamaernet (Enderta) the completeness of family folder registration was more than 100%. This achievement may be due to that the estimated households given by bureau of nance were less than the actual number of households in the tabia. And it may be also due to that the average fertility rate in the tabias was less than the actual fertility rate in the speci c tabias. And in some health posts the difference was very high for example, Adiawso HP in SahartiSamre (expected=1795 and FF registered =1507, difference=288), DebreKebie/enderta (expected=1493 and FF registered=833, difference=660), 44% house hold heads were not registered. In contrast to a study conducted in Kenya which revealed that completeness of registration was 74.6%, the completeness in this study was very high (95.8%). This might be due to that the average household members of the population in the study area were less than the conversion factor. In general the implementation status was excellent 95.8% according to the standard stated by stakeholders.
Consistency: In this study a large number (64.7%) of health posts were using other recording and reporting tools in addition to the standard CHIS tool. So this may contribute to the inconsistency of data recorded and reported in all the tools.
According to a study conducted in the SNNPR, the English family folder had translated in to Amharic to ensure correct recording of data and then MFI was prepared and Tabia pro le. The Master Family Index, which is a village-wise list of the household heads in alphabetic order helps identify the household number of the family, and thereby retrieve the family Folder from the shelf. In this study, the HEWs use a tally sheet for recording the services they provide daily and about 97.9% of data between MFI and FF were consistent (21).
As the CHIS scale up project study mentioned that importance of tally sheet in data consistency, recording of the HH number against the services provided by the HEW, is proving very valuable in assuring data quality. Likewise, this study has also revealed that all the HEWs of health posts had tallied the health service using house number. This implies that the quality of data in the health posts is assured as a result (21).
As an evaluation conducted in Kenya the proportion of households with latrines was 87.7% in the intervention sites and 84.4% in the comparison sites, but in this study, the proportion of health posts with latrine was 78.2%, which is low as compared with intervention site but good form the comparison sites (23).
The case study gave more emphasis on the data collection and reporting tools used by HEWs at community level. HEWs were used exercise books to capture data in heath posts & during outreach. As stated in a case study from multiple register to family folder conducted in-Ethiopia: It was noted that HEWs have used notebooks to record the health service given during home visiting. They transfer data from their notebooks to the main register at the health post. But in case of this study eld book was used to capture the activities done during outreach. In the visited health posts, there were many registers ranging from 7 to 12 to document different health services. For instance, in one of the visited health posts, there were 10 registers of which only three were readymade to document speci c health services whereas others were bare exercise books that need to be tailored as a register. Similarly the ndings of this study showed that more than 64.7% of health posts used more than 10 parallel recording and 76.4 % of health posts used reporting formats; in addition they were also preparing additional 1 to 15 pages by spending 1 to 3 days for doing these parallel reports (17).
As the study conducted in Oromya region, accuracy or consistency has expresses as the consistency between the records of births and deaths in family folder and the number of births and deaths in the community or households. But this study has used HH head name, house number, number of HH members' e.t.c. and consistency of the records in MFI and FF, the records in the FF and information in houses, the parallel recording and reporting tools used to assess the overall consistency of the program. Hence, the average consistency of the program implementation was 68.16%and it was jagged as very good implementation status (20).

Conclusion
According the results revealed from this study, it can be concluded that the overall implementation status of community health information system/FF in south east zone was acceptably improved. Based on the grades labeled for each of the four evaluation dimensions, the implementation status of CHIS was very good (87.48%) according to the judgment criteria's.
In general the overall availability of necessary resources for the implementation of CHIS was very good implementation status according the judgment criteria's. it can be said that most of the CHIS materials were available in all health posts of all woredas. However the availability of some of the resources was low, for e.g. Tigrigna manual and inks and brushes were very less available. So the implementation of the CHIS system was affected directly by the unavailability of these resources.
Regarding the compliance of utilization of the available resources according to the standards, most of the health posts were utilizing their resources according to the standards. Most of the health posts in the study area had provided FF to the population in their catchment area. But there was less achievement in the utilization of health cards i.e. in providing cards to each member registered in the FF when they came seeking treatment. The utilization of MFI and organization of FF in the shelves using HH number and kushet in all health posts were very appropriate. So, we can conclude that, this achievement can be taken as good utilization of the resources according to the standard guidelines and has contributed to the very good implementation of the CHIS system.
The completeness of, in general, the implementation of the system was also very adequate seeking slight improvements. Most of the health posts in the study area have provided FF to the HHs in their catchment area. In some of the health posts the registered number of households in family folders was greater than the expected number of households to be registered in the health posts. While in some of the health posts was vise-verse.
Regarding the consistency of the data elements in the family folder and the actual information in the households, we can conclude that it was good which needs notable improvements, especially in updating of family folder information and assigning numbers to newly registered household members. The study has revealed also that data quality assurance was done in most of the health posts, which indicates that the HEWs in most of the HPs were well aware of data quality.
In general, according to the standard of HSTP II 2016-2020, the implementation status of CHIS in south east zone, Tigray region was very good.   Map of South East Zone, Tigray