Aim
The aim of the trial is to address the research question: what are the effects of an innovative paper-based HIS (PHISICC) on data use and quality, quality of health and HW perceptions compared with the current HIS, in rural PHC settings?
Study design
The study is a CRCT in each of the three countries. In each setting, 70 health facilities are randomised to intervention or control (35 per arm). The intervention arm HF use the new PHISICC tools (substituting the usual HIS tools) and the control arm HF use the regular HIS tools. The trial is implemented in the real life contexts of HF carrying out their usual duties.
The CRCT are implemented in the real life contexts of HF carrying out their usual duties. The trials started between the end of 2019 and beginning of 2020, depending on the country, when the intervention was installed and the baseline surveys carried out; and will last till mid-2021.
Study areas
Ministries of Health (MOH) officials in several countries were contacted before submitting the proposal to the funding agency in order to explore the willingness to engage in a project focusing on paper-based tools. Officials in several countries rejected the offer on the grounds of upcoming digitalisation plans of the HIS in the country. We partnered with MOH that found the research relevant to their context in three countries.
In each country, the eligibility criteria of study areas were that they had to belong to the operational area of research partners; contain a large enough number of health facilities and their catchment population; include vulnerable population (e.g. with low vaccination coverage, high childhood mortality); and be comparatively neglected in terms of infrastructure and services. We excluded areas with concurrent research or other types of activities that could conflict with the CRCT (such as the co-existence of another health-related study, massive developments in infrastructure or activities involving migration of the population, such as temporary work sites or changes in working sites) and areas with threats to safety or security that could jeopardise research activities.
The study areas are located in Adzopé, Agboville, Tiassalé and Sikensi districts (Côte d’Ivoire); in Funhalouro, Govuro, Homoine, Inhambane, Inharrime, Inhassoro, Mabote, Maxixe and Panda (Inhambane province, Mozambique); and in Yala Local Government Authority (Cross-River State, Nigeria).
Eligibility of health facilities
The intervention is implemented at the HF level. The eligibility criteria of the HF were that they had to be located in the study areas, belong to the governmental health sector and their main activity should be the delivery of PHC services. HF were excluded if they had specialised clinical services, inpatients, physicians providing care or with plans for staff turn-over involving intervention and control HF.
A ‘master list’ of eligible health facilities was prepared based on information provided by the MOH across all study areas. We aimed at selecting 70 of the eligible HF in each country, using simple random techniques in R [].
Allocation and blinding
Allocation of the 70 HF per country into the intervention and control arms took place in a formal event, gathering research partners and MOH officials to offer transparency and promote study ownership by local and national authorities. Equally sized, folded pieces of paper with the names and codes of included HF written on them were introduced in an opaque receptacle where they were manually and blindly mixed. A second receptacle contained two equally sized pieces of paper, one with the word ‘intervention’ and another one with the word ‘control’. A selected person in the meeting, not belonging to the research team, extracted one piece of paper at a time to reach half the number of included HF. Then, a paper was extracted from the second receptacle to assign those HF to the intervention or control arms. The rest of the papers were extracted as well to verify completeness and no duplication of names, and those HF assigned to the other arm.
Once HF were selected, all villages or settlements for each health facility catchment area were listed and three in each catchment area were selected. In practice, we selected all villages because the numbers were below (in Côte d’Ivoire) or just above (in Nigeria) the needs. For each village, we used Google satellite maps to identify and geo-locate every visible roof. Where there were many houses per village (roughly, more than fifty or so), a researcher would mark four points in the map slightly beyond the northernmost, southernmost, easternmost and westernmost roofs seen and 30 random points were selected within that square. From the mapped points, 10 per village (with 10 more acting as reserve) were randomly selected and marked on another map used in the field for data collectors to approach households. Where technical problems impeded this approach in a given village, a field supervisor would rotate a bottle on the floor towards the centre of the village and would select at random 10 households in the direction pointed by the bottle, from the outer limit of the village till the centre [].
Blinding is only feasible for the research team members carrying out the CRCT data collection and the analyses of the CRCT findings. The intervention (i.e. paper tools) are by design very different from the existing system and it is not possible to blind participants or principal researchers.
We already had the agreement of the MOH and selected HF compliant with the inclusion criteria were provided with the intervention shortly after completing the baseline data collection. Therefore, recruitment as such took place at the same time of the allocation of HF into intervention and control arms.
The intervention
The PHISICC paper-based intervention is a full set of paper-based tools to support decision-making by frontline HW. These are the only tools to be used by HW in the intervention arm. The PHISICC tools encompass the whole system (i.e. recording and reporting) and all clinical and public health care areas and are characterised by: a common visual language (e.g. spaces for digits and text), standardised formats across health care areas; support to critical data items (e.g. respiratory rate in infants); graphic artefacts to distinguish severity degrees of signs or symptoms; documentation of diagnoses and treatment decisions; and aides memoires, among others.
The PHISICC tools have been developed over 18 to 20 months prior to the CRCT, using a Human Centred Design approach []. A strength of the Human Centred Design approach is its ability to unlock the user's perspective so that designers can build solutions that are fully reality-based and work well. Co-creation groups were formed in each country with researchers, staff from partner institutions and healthcare workers, led by a team of professional designers. Based on co-creation, participatory processes, and Human Centred Design principles, many iterations took place between co-creation groups and end-users of the tools, the frontline HW, till reaching a design that considered and addressed the main issues raised by HW (i.e. usability, clarity, size of tools). The PHISICC tools have been produced in French for Côte d’Ivoire, in Portuguese for Mozambique and English for Nigeria, which are the official languages used in the health systems in the three countries; using the official logo of the MOHs. Health care areas covered include: family planning, antenatal care, including tetanus toxoid vaccination, delivery, post-natal care, vaccination, sick child, adults outpatient consultation, tuberculosis diagnosis and treatment, and HIV. Referral forms were also designed.
The PHISICC tools have three sub-components: registers, tallies and reports. Registers are formed by seven DIN-A3 and one DIN-A4 (for referrals) book covering all health care areas except for tuberculosis treatment, for which DIN-A3 cards where used. Register books have 100, 200 or 400 pages depending on the country and health care area. They are used to record individual clients’ data for each health care event, either of clinical or public health nature. Some register books have clinical notes at the very beginning, as ‘aide memoires’, and an example of a filled-in form, to assist HW when doubting how to proceed.
Tallies are DIN-A3 single sheets which contain a list of the indicators to be transferred to higher levels of the health system, with a series of small ovals, grouped in fives, to mark with tally sticks with a pen. In contrast to the current systems that have no tallies or only for vaccination, tallies were created for all health care areas. In the middle-right side of the tally, a column accommodates cells aligned with the ovals to insert the count for each indicator; and in the far right of the sheet there is a replica of the count column, separated with a perforated line, which is detached and sent, as part of the monthly report to the higher level in the health system.
During three or four days, HW were trained on HIS before the start of the trial. In the intervention arm they were trained on the PHISICC tools; and the control arm received a refresher training about the regular tools, during the same number of days.
Additionally, given that the regular tools already contained information on past vaccination history of children still to complete their vaccination schedule, we created a mechanism to retrieve data of children’s vaccination status to transcribe into the new vaccination register book in the intervention arm (‘system transition’).
Tools were endorsed by MOH, printed in local printing companies and distributed to HW at the end of the training sessions. A digital spreadsheet was created to monitor consumption and order additional tools to cover health facility needs during the life of the trial.
Outcomes
There are five primary outcomes (Table 1). Vaccination adherence is defined as the total number of vaccine doses given during the trial period in the correct time interval to children over the total number of vaccine doses that should have been given during the same period. Antenatal care visits uptake will also be considered depending on the expected number of pregnancies in the study areas. Both are used as proxies for health outcomes in terms of protection against disease [] and prevention of pregnancy complications []. Data concordance is defined as the level of agreement of HIS indicators between (i) records, (ii) tallies and (iii) reports [3]. In terms of data use for decision making, we will estimate the diagnostics scope in the sick child (i.e. number of different diagnoses per child; and treatment appropriateness (i.e. number of prescribed treatments that are supported by a documented diagnosis). Health workers satisfaction will be assessed using a standardised questionnaire [,,]. While the intervention targets HF, some of the outcomes are measured at the level of HF, and some from patients clustered within HF catchment areas.
Table 1
Outcomes and parameters used to estimate the sample sizes.
|
Outcome name
|
Subjects
|
Definition
|
Baseline estimate
|
Expected change
|
Comments
|
1
|
Vaccination adherence
|
Children under-1 in the households
|
Number of vaccines given in the previous calendar year over the number of vaccine due in the same period
|
75 given per 100 due
|
Increase of 10 per 100
|
Vaccines are clustered within children, and children within HFs
|
2
|
Data concordance
|
Recording tools in health facilities
|
Number of health care events (e.g. vaccinations, antenatal care consultations) recounted in the previous calendar year versus the number of health care events reported in the same time period
|
7 recounted for each 10 reported [14]
|
Increase of 2 recounted
|
A single estimate can be obtained in each HF or by time periods (no clustering)
|
3
|
Diagnostic scope
|
Records of sick child consultations
|
Number of diagnosis in each sick child consultation during the previous calendar year
|
1 or 2 per child
|
30–35% with more than 1 diagnosis
|
Individual consultations are clustered within HF
|
4
|
Treatment appropriateness
|
Records of sick child consultations
|
Number of treatments correctly prescribed in each sick child consultation during the previous calendar year
|
Half appropriate over all consultations
|
Increase to three quarters appropriateness
|
Individual consultations are clustered within HF
|
5
|
Health workforce satisfaction
|
Health workers
|
Degree (score) of satisfaction across all health facilities in each arm, with the intervention
|
5 out of 10
|
9 out of 10
|
Maybe two or three health workers can be approached in each health facility
|
Secondary outcomes are classified under the following domains: data quality, data user, mortality, HW experience, clients experience and resource consumption
- Data quality
-
Completeness of recording and reporting in specific forms; i.e. prevalence of unduly missing data items; partograph used;
-
accuracy of recorded figures in comparison to real events (e.g. physical counting of commodities, such as number of 500mg Paracetamol tablets as recorded versus number of 500mg Paracetamol tablets as counted;
-
timeliness of reporting, as documented by time stamps in forms;
loss of data or data which does not reach the next upper administrative level.
-
Data use
-
in terms of knowledge (e.g. vaccines due based on date of birth; weight for length assessments);
-
cases of different conditions properly treated in (e.g. diarrhoea cases given oral rehydration therapy according to national guidelines; pneumonia cases given appropriate antibiotic according to national guidelines;
-
public health decisions: availability of lost to follow up lists or plans for vaccination, tuberculosis and or HIV/AIDS treatment control;
occurrence of stock outs of essential drugs
-
Overall under-5s mortality and under-5s mortality excluding peri-natal mortality [].
-
Health workers ‘human experience’ and satisfaction
-
District health information officers’ ‘human experience’
-
Clients ‘human experience’ and satisfaction
-
Resources consumption (e.g. time use, costs)
intervention costs: tools, training, start-up;
time used for recording and reporting (e.g. time-motion study) [
];
cost-effectiveness per unit of additional improvement in outcomes of interest.
In addition, we will consider ‘explanatory outcomes’ that will help to understand how the measured effects have taken place and why. We will look at the details of the interplay between the intervention, the system, the users and the context. Process indicators will be based on the documented activities that have taken place, from the conception of the intervention, up to its implementation, monitoring and evaluation. Process indicators may include: intervention set up and implementation, monitoring of the use of the intervention, special activities targeted at vulnerable populations, district reactions related to the intervention, handling of data coming from the new system, sustainability based on costs information and perceptions, alignment with national health policies and donor priorities. We will also explore health care services characteristics looking at generic indicators from health facilities, such human resources profiles and relations with the communities, population characteristics and system and context characteristics captured in early stages of the project, where data are available.
Sample size calculations
The required sample sizes for each primary outcome were determined using simulation. This allowed us to account for levels of clustering (Table 1). We used the regression models detailed in the data section to analyse the simulated trials and estimate the power. The simulation code was written in R.
For each country, we required the probability of a type I error (rejecting the null hypothesis when it is actually true) α to be less than 0.05 and a power of 80%.
The sampling frames are the study areas in each country, which include the health facilities and households in their catchment areas.
For vaccination adherence, using a sample size of 35 HF per arm, we would have 80% power in each country to detect as significant a difference between a proportion of due vaccines given from 75% in the control to 85% in the intervention arms, assuming one child per household, 30 households per HF and a between-HF variation equivalent to a k of 0.1, where k is equal to the standard deviation divided by the mean. The value of k is unknown, but was chosen in line with general observations by Hayes and Bennet [].
For data quality outcomes, with 35 HF per arm we would be able to detect as significant a difference from a ratio of 0.7 (reported : recorded) vaccinations in the control arm to 0.8 with the intervention with 80% power, assuming 100 recorded vaccinations per HF and a standard deviation of 0.1 in the ratios between HF.
In terms of diagnostic scope, we would be able to detect an increase in the proportion of child-visits with more than one diagnosis from 30–35% with 80% power with 35 HF per arm, 60 records per HF and assuming a k of 0.1 [18].
We would be able to detect as significant an increase from 50% of treatments having a corresponding appropriate diagnosis to 60% with 80% power assuming 35 HF per arm, 1 treatment per child, 25 children per HF and variation between HF corresponding to k = 0.1 [22].
For the outcome related to health workers’ satisfaction, we would be able to detect as significant an increase from 50% of health workers satisfied to 90%, with 80% power assuming 35 HF, three health workers per HF and a variation between HF equivalent to k = 0.1.
In summary, in each country we require 35 HF per arm, three HW per HF, 100 vaccination records per HF, 60 sick child records per health facility and 30 children per health facility catchment area.
Data collection and management
Data collection took place at baseline and will take place again at the end of the study. Data is collected from health facilities, from the households in the catchment areas of the included health facilities and also from district offices.
For data quality and data use outcomes, HF registers, tallies and reports will be scrutinised. For population based outcomes, we carry out household surveys at baseline and at end-line. We use standard approaches for these types of surveys []. Households are visited, the research project is briefly introduced and consent requested.. Ideally, mothers of alive children or women in child-bearing age were interviewed in order to obtain information on living children (i.e. vaccination history) and death events, respectively, using home-based records if available and accessible. Patients’ satisfaction will be assessed using the PSQ-18 satisfaction questionnaire [,,]. Essentially, the tool enables practitioners to investigate the extent to which their health care service meets the perceived needs of their client group and pinpoint areas for improvement [19]. The interview will be conducted with consenting patients as close to their care encounter as possible []. Data tools are translated into the official languages of the study countries and pilot tested for consistent meaning and relevance to the setting. Data collectors are also able to communicate in local languages. The Satisfaction of Employees in Health Care (SEHC) survey is a validated tool to assess staff satisfaction. It was first developed and validated in a low-income country (Ethiopia) [] and later successfully validated in a high-income country (USA) [].
We use a mix of paper and electronic data (ODK []) collection tools. Data collectors are trained to minimise error. Tools are piloted before implementing. ODK data is regularly stored and sent to secure servers, as soon as data collectors reach their office base. Data from paper tools is double entered and compared and sent to secure servers. Each data collection tool has its corresponding electronic database that is cleaned and submitted to the analyses. All data is anonymised at the point of data collection or as soon as possible in the data management process. Data is labelled with an arm code (e.g. ‘A’ or ‘B’) without any further information allowing to disclose which data items belong to the intervention or to the control arms, ensuring blinding during data analyses.
Quality will be assured through several mechanisms: piloting of data collection tools; thorough training of field workers; checking missing data related; double, independent data entry from papers into digital databases; early descriptive analyses to detect potential outliers; fieldworkers tracking and supervision.
Data analysis
The analysis will be carried out for each country separately, and based on intention-to-treat.
At baseline, data on population and health facility characteristics (i.e. basic demographic characteristics of population and health workers, professional profile of health workers, health facility size and services) will be produced and presented. If large imbalances are detected at baseline, this information can be used to adjust the effect estimate comparisons [,].
The analyses vary for the different primary outcomes due to the unit of measurement and levels of clustering, the type of variable, and whether measurements were taken at baseline and endpoint or endpoint only. We use regression models to allow us to estimate the effect of the outcome while flexibly accounting for these issues and allowing adjustment for potential confounders.
Logistic regression will be used for the binary variables: vaccine adherence is measured by determining whether each vaccine due was received, and treatment appropriateness by whether each treatment was correctly prescribed. Data concordance and diagnostic scope are count variables and may be analysed with Poisson regression, depending on their distribution. The regression model for HW satisfaction will depend on how it is distributed.
The outcomes have different levels of clustering (children or consultations, HW, HF). We will account for these levels of clustering by including random effects in the regression models.
Four of the primary outcomes are measured at baseline and end-line. The effect of the intervention will be estimated using an interaction term between arm and survey in the regression models: ie is the change in the outcome between baseline and follow-up in the intervention arm different to the change between baseline and follow-up in the control arm. The effect of HW satisfaction, measured only at end-line, will be estimated as the difference between the intervention and control arm.
All estimates for the effect of the intervention will be presented with 95% confidence intervals. The analyses will be carried out using R [].
Measures to minimise bias
Statistical analyses will be carried out blindly, without knowledge of what health facilities or population in the catchment area belong to the intervention or control groups. Only when the analysis code is considered as definitive and fixed, will results be shared with the wider investigators team and the arms for health facilities and population will be disclosed.
Outcome measurement bias may take place where data from the HIS, which is the focus of the intervention, is used to measure outcomes. However, we will minimise this by assessing population based outcomes at household level.
Contamination (i.e. the intervention affects individuals or units assigned to the control arm) may happen via the exchanges between health workers from health facilities in both arms; for example: in monthly district data quality meetings, managerial meetings; or through inputs from supervisors who influence control health facilities with intervention tips encountered in health facilities of the intervention arms. One mechanism to address this issue is using a district-based cluster randomisation scheme. However, we consider that (i) contamination, despite increasing the awareness of health works in control health facilities, will hardly influence the decision making mechanisms that the HIS intervention focuses on; and (ii) randomisation at the level of district poses additional challenges that are not worth the marginal benefit of reducing a doubtful contamination [].
The spill over effect (i.e. benefits of the intervention extend beyond their direct recipients) [] may take place in higher levels of the health systems; e.g. districts data managers and programme managers may experience the benefits of better structured and more timely data produced in health facilities in the intervention arms. The trial will have no capacity to quantitatively account for spill overs at higher levels of the system, due to the limited number of higher level administrative areas that will be involved in the trial. However, through process indicators, we will consider potential benefits and harms of the intervention at higher levels of the system.
A challenge is the Hawthorne effect (i.e. observer effect). Both health workers in the intervention and in the control sites will have an awareness of being observed as data collection activities will be at the same level of intensity in the two arms. Therefore, there should be no differential effect.
Analyses will be based on the intention-to-treat. It is important to closely monitor if the intervention HFs consistently use the new HIS tools and approaches. The data collection team and the trial monitoring team will check if old forms are still being used in the intervention health facilities. However, we do not expect health facilities to migrate between intervention and control arms, or vice versa, due to feasibility issues. On the other hand, some household members in a given catchment area may decide to seek for health care in a health facility belonging to another trial arm. In these cases, households will be analysed as belonging to the original trial arm.