Evaluating performance of health care facilities at meeting HIV-indicator reporting requirements in Kenya: An application of K-means clustering algorithm

doi:10.21203/rs.3.rs-54718/v1

Download PDF

Research article

Evaluating performance of health care facilities at meeting HIV-indicator reporting requirements in Kenya: An application of K-means clustering algorithm

https://doi.org/10.21203/rs.3.rs-54718/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 06 Jan, 2021

Read the published version in BMC Medical Informatics and Decision Making →

You are reading this older preprint version

Read the latest preprint version →

Introduction: The ability to report complete, accurate and timely data by HIV care providers and other entities is a key aspect in monitoring trends in HIV prevention, treatment and care, hence contributing to its eradication. In many low-middle-income-countries (LMICs), aggregate HIV data reporting is done through the District Health Information Software 2 (DHIS2). Nevertheless, despite a long-standing requirement to report HIV indicator data to DHIS2 in LMICs, few rigorous evaluations exist to evaluate adequacy of health facility reporting at meeting completeness and timeliness requirements over time. In this study, we report on a comprehensive assessment of the reporting status for HIV indicators, from the time of DHIS2 implementation, using Kenya as a case example.

Methods: A retrospective observational study was conducted to assess reporting performance of health facilities providing any of the HIV services in all 47 counties in Kenya between 2011 to 2018. Using data extracted from DHIS2, K-means clustering algorithm was used to identify homogeneous groups of health facilities based on their performance in meeting timeliness and completeness reporting requirements for each of the six programmatic areas. Average silhouette coefficient was used in measuring the quality of the selected clusters.

Results: Based on percentage average facility reporting completeness and timeliness, four homogeneous groups of facilities were identified namely: best performers, average performers, poor performers and outlier performers. Apart from blood safety reports, a distinct pattern was observed in five of the remaining reports, with the proportion of best performing facilities increasing and the proportion of poor performing facilities decreasing over time. However, between 2016 and 2018, the proportion of best performers declined in some of the programmatic areas. Over the study period, no distinct pattern or trend in proportion changes was observed among facilities in the average and outlier groups.

Conclusion: The identified clusters revealed general improvements in reporting performance in the various reporting areas over time, but with noticeable decrease in some areas between 2016 and 2018. This signifies the need for continuous performance monitoring with possible integration of machine learning and visualization approaches into national HIV reporting systems.

Medical Informatics

K-means clustering

completeness

timeliness

performance

dhis2

The Human Immunodeficiency Virus (HIV) epidemic remains a challenge globally with highest infected numbers found in countries in East and Southern Africa, which accounted for an estimated 20.7 million infected individuals in 2019 [1]. Efforts to eradicate the HIV epidemic have seen affected countries in LMICs receive substantial support from donors and multilateral global organizations in order to scale-up HIV services such as antiretroviral therapy (ART), prevention of mother-to-child transmission (PMTCT) of HIV, and HIV testing and counselling (HTC) [2]. This has brought about the need to strengthen strategic information on HIV. Health Management Information Systems (HMIS), through better data quality, improves decision-making such as informing policy, measuring program effectiveness, advocacy and resource allocation [3]. Ministries of Health (MoH) and donor organizations require facilities providing HIV services to report several aggregated HIV indicators as part of Monitoring and Evaluation (M&E) program [4, 5].

The scale-up of HIV services has contributed to strengthening of HMIS in many low-and-middle-income-countries (LMICS), resulting in improved availability of routinely generated HIV aggregate indicator data from health facilities to the national level [6]. HIV indicator data typically comes from aggregation of monthly reports generated by various facilities that are collated in summary forms and submitted to an aggregate-level HMIS or reporting system [6]. One such national-level data aggregation system is the District Health Information Software Version 2 (DHIS2), which has been adopted by many LMICs [7].

Aggregate data stored in systems such as DHIS2 are only as good as their quality [8]. Therefore, the ability to report complete, accurate and timely data by HIV care providers and other entities is a key aspect in monitoring trends in HIV care. Evaluations on quality of indicator reporting have previously been conducted within DHIS2 based on various data quality dimensions [9–13]. Some of these evaluations are on selected regions, and largely cover a limited time period [11]. In addition, despite a long-standing requirement to report HIV indicator data to DHIS2 in LMICs, few rigorous evaluations exist to evaluate adequacy of health facility reporting at meeting completeness and timeliness requirements over time.

Rigorous reporting by facilities into DHIS2 over time is imperative to identify changes in trends and implement timely interventions [14]. In this study, we conduct a comprehensive assessment of the reporting performance for HIV indicators at the national-level by facilities using completeness and timeliness indicators, with Kenya as a case example.

Study setting

This study was conducted in Kenya, a sub-Saharan country made up of 47 counties. Administratively, the health care service delivery system has six levels, namely: community, dispensary, health center, district hospital, provincial hospital, and national referral hospital [15]. Kenya adopted the DHIS2 in 2011 at the national level for aggregation of health data across different levels of the health system [7],[16].

Study design

A retrospective observational study was conducted in order to identify reporting performance over time by health facilities in meeting completeness and timeliness reporting requirements.

Data source

Data for facilities reporting completeness and timeliness between the years 2011 to 2018 were extracted from the DHIS2 in Kenya. These data were extracted for all facilities in all the 47 counties in Kenya. Systematic procedures were used in cleaning the data using a generic five-step approach as outlined in Gesicho et al.[17]. Data used were only for facilities that offered one or more of the outlined HIV services that required reporting, namely: (i) HIV testing and counselling (HTC), (ii) prevention of mother to child transmission (PMTCT), (iii) care and treatment (CRT), (iv) voluntary medical male circumcision (VMMC), (v) post-exposure prophylaxis (PEP) and (vi) blood safety (BS). These data were derived based on the MOH 731 Comprehensive HIV/AIDS facility-reporting form, which is the major monthly HIV summary report required by the MOH in Kenya and used by health facilities for reporting of HIV indicators into DHIS2. It is worth noting that health facilities are not required to report on indicators for all the six programmatic areas, but only those for which they provide services. As such, there are variations in number of facilities (n) in the various programmatic reporting areas.

Measures

Facility reporting completeness and timeliness

Percentage completeness in facility reporting is calculated automatically within Kenya’s DHIS2 and is defined as the number of actual monthly reports received divided by the expected number of reports in a given year. Percentage timeliness in facility reporting is also calculated automatically within Kenya’s DHIS2 and is defined as the number of actual monthly reports received on time (by the 15th of every month) divided by the expected number of reports in a given year. Facility reporting completeness and timeliness were selected as indicators for assessing reporting performance as they were readily available within DHIS2 for a 8-year period covered by the study.

Outcome Measures

The primary outcome of interest consisted of identifying the performance in reporting by health facilities over time (2011–2018), with facilities put into various performance clusters and performance evaluated in the various programmatic areas.

Data Analysis

K-means algorithm was preferred due to its suitability in pattern recognition, its simplicity, efficiency, ease of implementation and empirical success [18]. K-means algorithm is a non-hierarchical procedure where k represents the number of clusters, which need to be specified prior to any clustering [19]. Given that K-means algorithm uses unsupervised learning, the idea was to group the health facilities into k homogeneous groups based on their performance in completeness and timeliness, in each of the six programmatic areas for each of the study years. Based on the data set and purpose of this study, we used the average silhouette coefficient, which is an intrinsic method of measuring the quality of a cluster [20]. The average value of the silhouette coefficient ranges between − 1 (least preferable value indicating poor structure) and + 1 (most preferable value indicating good structure). According to Kaufman and Rousseeuw, average silhouette measure that is greater than + 0.5 indicates reasonable partitioning of data, whereas greater than + 0.7 indicates a strong partitioning [21]. On the other hand average silhouette measures lower than + 0.5 indicate a weak or artificial partitioning, whereas below + 0.2 indicates no clusters can be exhibited from the data [21].

In order to determine the number of clusters (k) to be generated, the Euclidean distance measure was applied and k was specified within a set of values [18], [22]. The range of k values was then iteratively re-run with two values of k ( k = 3 and k = 4) and inspecting the average corresponding silhouette values [23].

The proportion of facilities in the various cluster groups was then determined by calculating the percentage number of facilities in a particular cluster group out of the total facilities in that particular year. To illustrate the average performance of facilities within the various cluster groups, we developed a scatter chart visualization using Tableau [24]. In addition, HTC programmatic area was used as an illustrative example for the visualization, given that it is one of the most reported programmatic areas. All analyses were performed using SPSS [25].

Results from the silhouette coefficient average measures for each reporting area are presented in Table 1. The results ascertain that the average silhouette values for both k = 3 and k = 4 produce reasonable to strong partitioning except for 2011 under CRT where the values for k = 3 where below 0.5, hence k = 4 was used in this case. Therefore, based on method criteria and interpretability of the data set, either k = 3 and k = 4 were used where reasonable to strong partitions were identified in the average silhouette measures. As such, k = 4 was used when more variation could be provided in the data from four clusters, and k = 3 was used when three clusters provided more variation than four clusters. For VMMC and PEP programmatic areas, the number of health facilities was not enough to conduct cluster analysis in the year 2011.

Table 1

Average of the Silhouette of a k-means clustering when k = 3 and k = 4.
Average Silhouette Measures
HTC			PMTCT			CRT
Year	K = 3	K = 4	Year	K = 3	K = 4	Year	K = 3	K = 4
2011	0.800	0.775	2011	0.674	0.706	2011	0.368	0.582
2012	0.526	0.563	2012	0.585	0.588	2012	0.556	0.599
2013	0.659	0.648	2013	0.654	0.632	2013	0.637	0.618
2014	0.669	0.669	2014	0.676	0.666	2014	0.692	0.663
2015	0.737	0.709	2015	0.649	0.711	2015	0.710	0.705
2016	0.749	0.754	2016	0.791	0.774	2016	0.708	0.710
2017	0.685	0.673	2017	0.699	0.677	2017	0.696	0.700
2018	0.593	0.714	2018	0.689	0.707	2018	0.654	0.701
VMMC			PEP			BS
Year	K = 3	K = 4	Year	K = 3	K = 4	Year	K = 3	K = 4
2011	a	a	2011	0.704	0.679	2011	a	a
2012	1.00	b	2012	0.593	0.605	2012	0.734	0.730
2013	0.64	0.669	2013	0.639	0.629	2013	0.732	0.687
2014	0.634	0.661	2014	0.675	0.667	2014	0.712	0.650
2015	0.733	0.681	2015	0.682	0.673	2015	0.617	0.641
2016	0.708	0.699	2016	0.696	0.665	2016	0.719	0.680
2017	0.765	0.733	2017	0.621	0.611	2017	0.577	0.637
2018	0.657	0.636	2018	0.650	0.673	2018	0.610	0.607
a: There are not enough valid cases to conduct the specified cluster analysis.
b: In the data, there is insufficient variation to honor the four clusters specified. The number of clusters is reduced to 3

The four clusters were characterized based on health facility performance as follows:

Best performers

This cluster consisted of health facilities that had the highest percentage in reporting completeness and timeliness in a particular reporting year.

Average performers

This cluster consisted of health facilities that had lower percentage in reporting completeness and timeliness compared to best performers in a particular year.

Poor performers

This cluster consisted of health facilities with lowest percentage in reporting completeness and timeliness in a particular year.

Outlier performers

This cluster consisted of health facilities with high percentage in completeness compared to average performers, but with low percentage in timeliness in that particular year.

It is worth noting that there were no clusters with low completeness and high timeliness as reports cannot be on time if they were not submitted in the first place. Detailed results by cluster for each reporting programmatic area are outlined below.

In Table 2 and Fig. 1, we present the segmentation of facilities based on performance clusters according to the HTC programmatic area. Based on performance trends presented in Fig. 1 for HTC, the proportion of best performing facilities accounted for 72.55% in 2016, which was a progressive increase from 31.50% in 2012. Nonetheless, in 2017 and 2018 the proportion of best performing facilities accounted for 58.30% and 51.08% respectively, which was a progressive decrease from 72.55% in 2016. On the other hand, the proportion of poor performing facilities accounted for 3.40% in 2016, which was a progressive decrease from 74.93% in 2011. However, the proportion of poor performing facilities accounted for 13.49% in 2018, which was a progressive increase from 3.40% in 2016.

The proportion of average and outlier performing facilities varied in the different years with no steady trend. Nonetheless, in the latter years, the proportion of average performing facilities accounted for 20.02% in 2018, which was a progressive increase from 6.00% in 2016. On the other hand, proportion of outlier performers accounted for 15.40% in 2018, which was a decrease from 18.02% in 2017.

Table 2

HIV Testing and Counselling (HTC)-health facility segmentation based on performance clusters
Year	2011				2012
Clusters	Best n = 0	Average n = 177	Poor n = 556	Outlier n = 9	Best n = 1206	Average n = 1301	Poor n = 794	Outlier n = 528
MOH 731-1 HTC Completeness	0.00	24.49	13.07	91.67	90.08	55.30	25.68	86.75
MOH 731-1 HTC Timeliness	0.00	16.63	2.91	21.30	80.47	45.65	16.11	46.17
Year	2013				2014
Clusters	Best n = 3219	Average n = 806	Poor n = 437	Outlier n = 427	Best n = 3837	Average n = 568	Poor n = 297	Outlier n = 615
MOH 731-1 HTC Completeness	96.73	68.77	32.96	89.86	98.18	73.07	33.75	95.94
MOH 731-1 HTC Timeliness	89.55	57.33	21.63	43.00	92.96	62.42	23.02	54.06
Year	2015				2016
Clusters	Best n = 3916	Average n = 1172	Poor n = 296	Outlier n = 282	Best n = 4376	Average n = 362	Poor n = 205	Outlier n = 1089
MOH 731-1 HTC Completeness	99.40	88.30	34.57	93.09	99.34	69.15	31.47	91.29
MOH 731-1 HTC Timeliness	96.33	71.71	27.45	33.45	95.89	51.07	20.29	74.04
Year	2017				2018
Clusters	Best n = 3698	Average n = 1164	Poor n = 338	Outlier n = 1143	Best n = 3403	Average n = 1334	Poor n = 899	Outlier n = 1026
MOH 731-1 HTC Completeness	97.98	64.47	32.69	94.20	88.48	52.68	26.87	77.35
MOH 731-1 HTC Timeliness	93.92	57.04	23.59	64.33	86.93	48.84	22.98	64.65

In Table 3 and Fig. 2, we present the segmentation of facilities based on performance clusters according to the PMTCT programmatic area. Based on performance trends presented in Fig. 2 for PMTCT, the proportion of best performing facilities accounted for 74.01% in 2015, which was a progressive increase from 18.80% in 2011. Nonetheless, in 2018 the proportion of best performing facilities accounted for 47.15%, which was a progressive decrease from 74.01% in 2015. On the other hand, the proportion of poor performing facilities accounted for 3.66% in 2015, which was a progressive decrease from 77.07% in 2011. However, in 2018 the proportion of poor performing facilities accounted for 14.61%, which was a progressive increase from 3.66% in 2015.

The proportion of average and outlier performing facilities varied in the different years with no steady trend. Nonetheless, for the latter years, proportion of average performing facilities accounted for 20.34% in 2018, which was an increase from 17.19% in 2017. On the other hand, proportion of outlier performers accounted for 17.90% in 2018, which was an increase from 3.65% in 2016.

Table 3

Prevention of mother to child transmission (PMTCT) - health facility segmentation based on performance clusters
Year	2011				2012
Clusters	Best n = 132	Average n = 20	Poor n = 541	Outlier n = 9	Best n = 1052	Average n = 1230	Poor n = 782	Outlier n = 508
MOH 731-2 PMTCT Completeness	21.67	38.32	12.91	91.67	90.03	55.51	26.09	85.65
MOH 731-2 PMTCT Timeliness	18.64	4.58	2.81	18.52	80.87	45.55	16.20	47.33
Year	2013				2014
Clusters	Best n = 2277	Average n = 1188	Poor n = 527	Outlier n = 444	Best n = 2737	Average n = 1210	Poor n = 277	Outlier n = 586
MOH 731-2 PMTCT Completeness	97.73	84.02	37.19	85.98	98.61	89.43	37.03	96.26
MOH 731-2 PMTCT Timeliness	92.11	63.53	26.11	29.70	92.31	59.29	24.02	14.54
Year	2015				2016
Clusters	Best n = 3785	Average n = 517	Poor n = 187	Outlier n = 625	Best n = 2732	Average n = 1156	Poor n = 237	Outlier n = 194
MOH 731-2 PMTCT Completeness	98.84	75.61	30.34	98.13	99.43	90.03	37.95	89.32
MOH 731-2 PMTCT Timeliness	91.22	61.72	21.97	38.76	95.42	72.36	25.98	38.46
Year	2017				2018
Clusters	Best n = 3456	Average n = 944	Poor n = 348	Outlier n = 744	Best n = 2685	Average n = 1259	Poor n = 832	Outlier n = 1018
MOH 731-2 PMTCT Completeness	97.58	64.96	38.51	93.73	88.48	53.03	27.69	79.02
MOH 731-2 PMTCT Timeliness	91.54	58.59	26.55	54.52	86.72	48.22	22.65	63.20

In Table 4 and Fig. 3, we present the segmentation of facilities based on performance clusters according to the CRT programmatic area. Based on performance trends presented in Fig. 3 for CRT, the proportion of best performing facilities accounted for 75.49% in 2016, which was a progressive increase from 5.65% in 2011. Nonetheless, in 2018 the proportion of best performing facilities accounted for 53.24%, which was a progressive decrease from 75.49% in 2016. On the other hand, the proportion of poor performing facilities accounted for 2.99% in 2016, which was a progressive decrease from 71.75% in 2011. However, in 2018 the proportion of poor performing facilities accounted for 17.47%, which was a progressive increase from 2.99% in 2016.

The proportion of average and outlier performing facilities varied in the different years with no steady trend. Nonetheless, for the latter years the proportion of average performing facilities accounted for 24.81% in 2018, which was an increase from 7.06% in 2016. On the other hand, proportion of outlier performers accounted for 4.48% in 2018, which was a progressive decrease from 14.46% in 2016.

Table 4

Care and treatment (CRT) - health facility segmentation based on performance clusters
Year	2011				2012
Cluster	Best n = 20	Average n = 76	Poor n = 254	Outlier n = 4	Best n = 634	Average n = 662	Poor n = 430	Outlier n = 98
MOH 731-3 Care and Treatment Completeness	42.50	21.61	12.49	93.75	90.00	57.90	24.69	84.61
MOH 731-3 Care and Treatment Timeliness	2.09	17.79	2.70	22.93	76.54	46.74	15.29	22.81
Year	2013				2014
Cluster	Best n = 1063	Average n = 587	Poor n = 217	Outlier n = 219	Best n = 1407	Average n = 554	Poor n = 204	Outlier n = 236
MOH 731-3 Care and Treatment Completeness	97.67	81.29	31.24	90.05	98.67	87.86	34.51	94.53
MOH 731-3 Care and Treatment Timeliness	90.81	59.82	19.79	24.03	92.01	62.55	27.03	24.65
Year	2015				2016
Cluster	Best n = 1647	Average n = 607	Poor n = 132	Outlier n = 227	Best n = 2171	Average n = 203	Poor n = 86	Outlier n = 416
MOH 731-3 Care and Treatment Completeness	99.00	93.63	35.73	93.96	99.09	76.65	27.22	97.21
MOH 731-3 Care and Treatment Timeliness	94.71	66.06	23.13	25.27	91.13	59.43	16.15	38.87
Year	2017				2018
Cluster	Best n = 1837	Average n = 750	Poor n = 264	Outlier n = 241	Best n = 1676	Average n = 781	Poor n = 550	Outlier n = 141
MOH 731-3 Care and Treatment Completeness	98.82	92.41	43.10	95.74	86.94	55.13	26.68	71.65
MOH 731-3 Care and Treatment Timeliness	94.22	65.41	32.70	27.61	81.75	50.71	23.26	21.37

In Table 5 and Fig. 4, we present the segmentation of facilities based on performance clusters according to the VMMC programmatic area. Based on performance trends presented in Fig. 4 for VMMC, the proportion of best performing facilities accounted for 54.35% in 2016, which was a progressive increase from 8.70% in 2013. Nonetheless, in 2018 the proportion of best performing facilities accounted for 17.31%, which was a progressive decrease from 54.35% in 2016. On the other hand, the proportion of poor performing facilities accounted for 13.04% in 2016, which was a progressive decrease from 39.13%% in 2013. However, in 2017 and 2018 the proportion of poor performing facilities accounted for 21.88% and 21.15%, which was a progressive increase from 13.04% in 2016.

The proportion of average and outlier performing facilities varied in the different years with no steady trend. Nonetheless, for the latter years, the proportion of average performing facilities accounted for 25.00% in 2018, which was an increase from 15.63% in 2017. On the other hand, proportion of outlier performers accounted for 36.54% in 2018, which was a progressive increase from 10.87% in 2016.

Table 5

Voluntary medical male circumcision (VMMC)-health facility segmentation based on performance clusters
Year	2012				2013
Cluster	Best n = 0	Average n = 2	Poor n = 2	Outlier n = 2	Best n = 2	Average n = 7	Poor n = 4	Outlier n = 5
MOH 731-4 VMMC Completeness	0.00	17.00	8.00	8.00	54.50	35.57	13.89	51.80
MOH 731-4 VMMC Timeliness	0.00	17.00	8.00	0.00	50.00	19.00	7.33	23.40
Year	2014				2015
Cluster	Best n = 7	Average n = 14	Poor n = 16	Outlier n = 5	Best n = 15	Average n = 7	Poor n = 7	Outlier n = 15
MOH 731-4 VMMC Completeness	85.86	51.14	20.38	81.80	95.07	50.00	15.57	86.67
MOH 731-4 VMMC Timeliness	81.14	39.36	13.00	36.60	88.38	42.86	14.43	62.20
Year	2016				2017
Cluster	Best n = 25	Average n = 10	Poor n = 7	Outlier n = 4	Best n = 28	Average n = 10	Poor n = 14	Outlier n = 12
MOH 731-4 VMMC Completeness	97.12	67.60	17.86	70.75	92.61	52.40	17.79	86.83
MOH 731-4 VMMC Timeliness	90.00	62.60	13.14	16.75	86.88	37.40	10.57	58.31
Year	2018
Cluster	Best n = 9	Average n = 13	Poor n = 11	Outlier n = 19
MOH 731-4 VMMC Completeness	85.73	43.94	19.09	61.s58
MOH 731-4 VMMC Timeliness	81.11	36.15	16.36	55.26

In Table 6 and Fig. 5, we present the segmentation of facilities based on performance clusters according to the PEP programmatic area. Based on performance trends presented in Fig. 5 for PEP, the proportion of best performing facilities accounted for 66.76% in 2015, which was a progressive increase from 2.99% in 2011. Nonetheless, in 2018 the proportion of best performing facilities accounted for 51.24%, which was a decrease from 66.01% in 2017. On the other hand, the proportion of poor performing facilities accounted for 3.91% in 2016, which was a progressive decrease from 17.76% in 2013. However, in 2018 the proportion of poor performing facilities accounted for 18.59%, which was a progressive increase from 3.91% in 2016.

The proportion of average and outlier performing facilities varied in the different years with no steady trend. Nonetheless, for the latter years the proportion of average performing facilities accounted for 28.76% in 2018, which was an increase from 17.09% in 2017. On the other hand, proportion of outlier performers accounted for 1.41% in 2018, which was a progressive decrease from 24.78% in 2016.

Table 6

Post-exposure prophylaxis (PEP)-health facility segmentation based on performance clusters
Year	2011				2012
Cluster	Best n = 0	Average n = 2	Poor n = 63	Outlier n = 2	Best n = 173	Average n = 256	Poor n = 328	Outlier n = 34
MOH 731-5 Post-Exposure Prophylaxis Completeness	0.00	54.20	13.48	95.85	84.98	54.24	23.56	89.71
MOH 731-5 Post-Exposure Prophylaxis Timeliness	0.00	4.15	6.18	8.35	73.74	44.72	15.65	34.07
Year	2013				2014
Cluster	Best n = 583	Average n = 281	Poor n = 205	Outlier n = 85	Best n = 677	Average n = 221	Poor n = 124	Outlier n = 281
MOH 731-5 Post-Exposure Prophylaxis Completeness	94.44	61.18	29.01	87.45	97.04	56.66	23.39	83.53
MOH 731-5 Post-Exposure Prophylaxis Timeliness	88.01	51.75	20.00	41.74	93.14	40.20	17.24	63.91
Year	2015				2016
Cluster	Best n = 954	Average n = 305	Poor n = 103	Outlier n = 67	Best n = 953	Average n = 161	Poor n = 61	Outlier n = 387
MOH 731-5 Post-Exposure Prophylaxis Completeness	97.14	76.33	27.25	78.37	98.15	59.58	27.85	83.22
MOH 731-5 Post-Exposure Prophylaxis Timeliness	93.05	62.86	22.34	29.24	95.37	46.37	22.83	70.99
Year	2017				2018
Cluster	Best n = 1031	Average n = 267	Poor n = 137	Outlier n = 127	Best n = 725	Average n = 407	Poor n = 263	Outlier n = 20
MOH 731-5 Post-Exposure Prophylaxis Completeness	95.73	66.51	38.29	90.02	85.04	54.06	24.07	80.50
MOH 731-5 Post-Exposure Prophylaxis Timeliness	91.35	59.21	28.23	54.50	82.38	49.99	20.32	36.46

In Table 7 and Fig. 6, we present the segmentation of facilities based on performance clusters according to the BS programmatic area. Based on performance trends presented in Fig. 6 for BS, the proportion of best performing facilities accounted for 26.67% in 2015 and 2016, which was a decrease from 33.33% in 2014. Nonetheless, in 2018 the proportion of best performing facilities accounted for 15.38%, which was a decrease from 32.00% in 2017. On the other hand, the proportion of poor performing facilities accounted for 20.00% in 2015 and 2016, which was a progressive decrease from 43.48% in 2011. However, in 2017 the proportion of poor performing facilities accounted for 24.00%, which was an increase from 2016. For the latter years, the proportion of average performing facilities accounted for 28.00% in 2017 and 38.46% in 2018. On the other hand, proportion of outlier performers accounted for 16.00% in 2017 and 23.08% 2018. Nonetheless, there have been a general progressive decrease in facilities submitting BS indicators from 2013 to 2018.

Table 7

Blood safety (BS) - health facility segmentation based on performance clusters
Year	2012				2013
Cluster	Best n = 3	Average n = 8	Poor n = 10	Outlier n = 2	Best n = 8	Average n = 8	Poor n = 11	Outlier n = 12
MOH 731-6 Blood Safety Completeness	69.67	43.75	18.30	100.00	94.88	37.50	15.82	75.75
MOH 731-6 Blood Safety Timeliness	67.00	35.25	14.23	54.00	91.50	30.13	8.18	57.00
Year	2014				2015
Cluster	Best n = 11	Average n = 10	Poor n = 9	Outlier n = 3	Best n = 8	Average n = 14	Poor n = 6	Outlier n = 2
MOH 731-6 Blood Safety Completeness	95.55	67.60	47.33	97.33	87.38	62.43	22.17	58.00
MOH 731-6 Blood Safety Timeliness	87.95	62.50	40.33	22.33	81.25	45.86	15.17	8.50
Year	2016				2017
Cluster	Best n = 8	Average n = 9	Poor n = 6	Outlier n = 7	Best n = 8	Average n = 7	Poor n = 6	Outlier n = 4
MOH 731-6 Blood Safety Completeness	94.88	69.56	47.33	27.14	83.25	56.00	26.33	79.25
MOH 731-6 Blood Safety Timeliness	92.79	62.00	40.33	17.86	78.13	54.86	22.33	41.50
Year	2018
Cluster	Best n = 2	Average n = 5	Poor n = 3	Outlier n = 3
MOH 731-6 Blood Safety Completeness	85.00	54.00	26.67	66.67
MOH 731-6 Blood Safety Timeliness	75.00	34.00	26.67	53.33

Scatter chart visualization of HTC performance clusters

In this section, we present an interactive visual representation of performance clusters using scatter charts. As an illustrative example using performance reporting of the HTC programmatic area, Fig. 7 demonstrates the visualization of the average performance of facilities by county for the period 2011 to 2018. Each of the four performance groups are represented using a similar color approach in Fig. 1 to Fig. 6. Each point contains the following attributes: name of county, number of facilities represented in that county, and the average completeness and timeliness for the facilities, which are displayed upon hovering the mouse on a point. For example, a green point may represent the average completeness and timeliness for the number of facilities in Nairobi county, which were in the best performing cluster in a particular year. This scenario is replicated for other counties and performance clusters. It is worth noting that facilities represented in each point are of varying characteristics such as type (hospital, health center), and ownership (private, public), hence are clustered based on performance. As such, the points in the scatter chart visualization provide a clear illustration of the four performance cluster groups and their behavior over time. For instance, the initial year of reporting shows only few clusters. Nonetheless, as reporting increases with time, more clusters develop.

Moreover, the outlier performance cluster has shown some improvement in performance as demonstrated with the left movement in the chart over time. The best performing cluster (green) also demonstrates a similar observation with the most improvement in 2016. The illustration in Fig. 1 further shows the proportion of best performing facilities being higher in 2016. Further still, the average completeness and timeliness among the average performance cluster (orange), seemed to have improved in 2015 compared with previous and subsequent years, based on the upward shift in the chart.

The results of our study demonstrate how k-means clustering and interactive cluster-based visualization can be used in identifying patterns and categories within national-level HIV reporting systems, uncovering previously unrecognized patterns. The four categories identified (best performers, average performers, poor performers and outlier performers) reveal the variation in reporting performance among facilities with respect to year and programmatic area. Moreover, apart from the BS programmatic area, a distinct pattern observed in five of the other programmatic areas was that as the proportion of best performing facilities increased, the proportion of poor performing facilities decreased. In addition, the proportion of facilities in the best performing cluster was higher over time, compared to the proportion of facilities in the other performance clusters. These observations denote improvements in reporting over time within Kenya.

Factors that could explain these improvements in part include data quality improvement procedures done through progressive trainings of those collecting primary data and of health records information officers, provision of technical reporting support to facilities [7]. Other factors such as automation of indicator reporting by electronic medical records (EMRs) to the DHIS2, have the potential to improve routine reporting based on evidence from feasibility studies conducted [26]. With future prospects on automating indicator data reporting, cohort studies can be conducted to establish their impact based on facility reporting completeness and timeliness performance in DHIS2. Further, concerted efforts in improving routine performance of HMIS, touching on technical, behavioral and organizational domains can improve reporting in Kenya [27].

However, despite the observed improvements in performance, there was a decline in proportion of best performing facilities in different years (between 2016 and 2018), depending on the programmatic area. It is worth noting that Kenya experienced one of the longest health worker strike in the public-sector from 5 December 2016 to November 2017, lasting a total of 250 days [28]. The first phase (5 December to 14 March 2017), involved a doctors strike lasting 100 days [28]. Whereas the second phase (5 June to 1 November 2017) involved a nurses strike lasting 150 days [28]. As such, although there may have been other factors that contributed to the decline in proportion of best performing facilities, we suspect that these strikes might have also affected the reporting process. In addition, the decline in 2018 may be attributed to the introduction of new MOH731 summary reporting tools revised in 2018. As such, some facilities were still using the old tool while others had already began using the new tool, signifying the need to improve approaches during transition of reported data.

In overall, we observed that average percentage timeliness tended to be lower compared to average percentage completeness in all the four performance groups. This observation is reflected in other similar studies [11], [29]. Nonetheless, as much as this observation was common among the four performance groups, the outlier performance group specifically brings to light larger disparities between average completeness and timeliness. For instance, as presented in Table 2 for the year 2011, we see that average completeness is 91.67% and timeliness 21.30%. Similar observations can be made for subsequent tables in the various programmatic areas.

Given that timeliness plays an important role in decision-making, there is a cause for concern when there is good effort in submitting of reports, with limitations on timeliness especially in the outlier performance group. As such, there is need for qualitative enquiries to investigate the large disparities in average percentage completeness and timeliness. This is because various factors could act as barriers or facilitators to health facilities ability to attaining and maintaining good completeness and timeliness reporting performance. These factors could be targeted by ministries of health in developing strategies to improve reporting performance of health facilities.

A limitation observed in the scatter chart was that the data points become densely packed in cases where they are many in a small area, hence making it difficult to identify the various points within a cluster. An example is best performers (Fig. 7), more so in 2016. Nonetheless, interactive components (mouse hovering and filtering) incorporated within the scatter chart facilitate access to detailed information. As such, this allows for closer examination of various elements within the data set such as performance in individual counties and number of facilities within a county for a particular performance cluster. This also enables identifying areas that warrant further investigation in their performance, which contributes to informed decision-making. The interactive approach was also used based on the need to visualize various facets of data simultaneously, which can be a challenge [30].

Incorporation of these analyses as well as visualizations to run in real time within aggregate-level HMIS, have the potential to allow monitoring and timely responsiveness to performance changes. Moreover, off shelf software such as tableau [24], which provide basic modules for free usage can be leveraged as a cost effective alternative for representing and sharing analysis for routinely collected data that has been extracted from large data systems

The scope of the study can be relevant for many countries dealing with HIV reporting in aggregate-level HMIS. However, the limitation in this study is that data have been collected and analyzed for one country only. Nonetheless, the indicators used (completeness and timeliness) could also be relevant in other contexts. Further, the findings only reflect trends and associations, and do not explain causality. Investigations, including use of qualitative approaches, are needed to definitively determine causes of the observed trends and variations. While we only looked at clustering based on performance, we recognize that performance can be associated with several other factors including facility ownership (private vs public), facility type and level, (for example hospital, dispensary), presence or absence of electronic reporting systems, geographical location and infrastructure availability, among others.

One of the future aims will be to determine factors influencing movement of facilities between clusters with special attention to factors associated with decrease in performance.

K-means clustering and interactive cluster-based visualization was applied to identify patterns of performance in terms of completeness and timeliness of facility reporting in six HIV programmatic areas. This resulted to four clusters: best performers, average performers, poor performers, and outlier performers, depending on average percentage of completeness and timeliness. The identified clusters revealed general improvements in reporting performance in the various reporting areas over time, but with most noticeable decrease in some programmatic areas between 2016 and 2018. This signifies the need for continuous performance monitoring with possible integration of machine learning and visualization approaches into national HIV reporting systems.

As future work, we will also work with the relevant decision-makers in the study country to incorporate the demonstrated machine learning and visualization approaches for use in automatic and continuous assessment of reporting performance within Kenya.

ART

Antiretroviral therapy

Blood Safety

CRT

Care and Treatment

DHIS2

District Health Information System Version 2

EMRs

Electronic Medical Record System

HTC

HIV Testing and Counselling

HIV

Human Immunodeficiency Virus

HMIS

Health Information Management Systems

LMICs

Low and Middle Income Countries

M&E

Monitoring and Evaluation

MoH

Ministry of Health

PEP

Post-Exposure Prophylaxis

PMTCT

Prevention of Mother to Child Transmission

VMMC

Voluntary Medical Male Circumcision

Ethics approval

Ethical approval for this study was obtained from the Institutional Review and Ethics Committee (IREC) in Moi University-No. 0003362.

Consent for publication

Not applicable

Availability of data and materials

The data sets generated and analysed during the current study originated from the national District Health Information Software 2 online database at https://hiskenya.org/.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Ministry of Health in Kenya.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was supported in part by the NORHED program (Norad: Project QZA-0484). The content is solely the responsibility of the authors and does not represent the official views of the Norwegian Agency for Development Cooperation.

Authors' contributions

M.G, A.B, and M.W designed the study. A.B and M.W supervised the study. M.G and A.B analyzed the data. All authors discussed the results, reviewed, and approved the final manuscript. M.G. wrote the final manuscript.

Acknowledgments

Not applicable

Global HIV. & AIDS statistics — 2020 fact sheet | UNAIDS. https://www.unaids.org/en/ resources/fact-sheet. Accessed 14 July 2020.
UNAIDS. Towards universal access. In: UNAIDS Annual Report. 2009. http://www.unaids.org/en/KnowledgeCentre/Resources/Publications/default.asp. Accessed 14 July 2020.
Mbondo M, Scherer J, Aluoch GO, Sundsmo A, Mwaura N. Organizational HIV monitoring and evaluation capacity rapid needs assessment: The case of Kenya. Pan Afr Med J. 2013;14:1–7.
Porter LE, Bouey PD, Curtis S, Hochgesang M, Idele P, Jefferson B, et al. Beyond Indicators. JAIDS J Acquir Immune Defic Syndr. 2012;60:120–6.
Ekouevi DK, Karcher S, Coffie PA. Strengthening health systems through HIV monitoring and evaluation in Sub-Saharan Africa. Current Opinion in HIV and AIDS Jul; 2011. p. 245–50.
Saito S, Howard AA, Chege D, Ellman TM, Ahoua L, Elul B, et al. Monitoring quality at scale: implementing quality assurance in a diverse, multicountry HIV program. AIDS. 2015;29:129-36.
Manya A, Braa J, Øverland L, Titlestad O, Mumo J, Nzioka C. National Roll out of District Health Information Software (DHIS 2) in Kenya, 2011-Central Server and Cloud based Infrastructure. IST-Africa 2012. 2012;1–9.
Manya A, Nielsen P. Reporting practices and data quality in health information systems in developing countries: an exploratory case study in Kenya. J Health Inform Dev Ctries. 2016;10:114–26.
Bhattacharya AA, Umar N, Audu A, Allen E, Schellenberg JRM, Marchant T. Quality of routine facility data for monitoring priority maternal and newborn indicators in DHIS2: A case study from Gombe State, Nigeria. Bazzano AN, editor. PLoS One. 2019 J;14:e0211265.
Githinji S, Oyando R, Malinga J, Ejersa W, Soti D, Rono J, et al. Completeness of malaria indicator data reporting via the District Health Information Software 2 in Kenya, 2011–2015. BMC Malar J. 2017;16:1–11.
Adokiya MN, Awoonor-Williams JK, Beiersmann C, Müller O. Evaluation of the reporting completeness and timeliness of the integrated disease surveillance and response system in northern Ghana. Ghana Med J. 2016;50:3–8.
Kiberu VM, Matovu JK, Makumbi F, Kyozira C, Mukooyo E, Wanyenze RK. Strengthening district-based health reporting through the district health management information software system: The Ugandan experience. BMC Med Inform Decis Mak. 2014;14:40.
Nisingizwe MP, Iyer HS, Gashayija M, Hirschhorn LR, Amoroso C, Wilson R, et al. Toward utilization of data for program management and evaluation: Quality assessment of five years of health management information system data in Rwanda. Glob Health Action. 2014;7.
WHO. Data Quality Review (DQR) Toolkit. WHO. In:World Health Organization; 2019. who.int/healthinfo/tools_data_analysis/en/. Accessed in 5 March 2020.
Muga R, Kizito P, Mbayah MM, Gakuruh T. Overview of the Health System in Kenya.In: Kenya Service Provision Assessment Survey 2004;1999.p. 13–24.
Karuri J, Waiganjo P, Orwa D, Manya A. DHIS2: The Tool to Improve Health Data Demand and Use in Kenya. J Health Inform Dev Ctries. 2014;8:38–60.
Gesicho M, Babic A, Were M. Data Cleaning Process for HIV Indicator Data Extracted from DHIS2 National Reporting System: Case Example of Kenya. Preprint at https://www.researchsquare.com/article/rs-21675/v1 (2019).
Jain AK. Data clustering: 50 years beyond K-means. Pattern Recognit Lett. 2010;31:651–66.
Everitt BS, Landau S, Leese M, Stahl D. Cluster Analysis. Cluster Analysis: 5th ed. wiley; 2011.
Rousseeuw PJ. Silhouettes. A graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math. 1987;20:53–65.
Kaufman L, Rousseeuw PJ. Finding Groups in Data, An Introduction to Cluster Analysis. John Wiley & Sons, Inc.; 1990.
Pham DT, Dimov SS, Nguyen CD. Selection of K in K-means clustering. Proc Inst Mech Eng Part C J Mech Eng Sci. 2005;219:103–19.
Thinsungnoen T, Kaoungku N, Durongdumronchai P, Kerdprasop K, Kerdprasop N. The Clustering Validity with Silhouette and Sum of Squared Errors. In International Conference on Industrial Application Engineering; 2015. p. 44–51.
Murray D, Chabot C. Tableau Your Data!: Fast and Easy Visual Analysis with Tableau Software. John Wiley & Sons, Inc; 2013. 528 p.
IBM Corp. IBM SPSS Statistics for Windows Version 25. Armonk: IBM Corp; 2017.
Kariuki JM, Manders E-J, Richards J, Oluoch T, Kimanga D, Wanyee S, et al. Automating indicator data reporting from health facility EMR to a national aggregate data system in Kenya: An Interoperability field-test using OpenMRS and DHIS2. Online J Public Health Inform. 2016;8:e188.
Aqil A, Lippeveld T, Hozumi D. PRISM framework: A paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy Plan. 2009;24:217–28.
Irimu G, Ogero M, Mbevi G, Kariuki C, Gathara D, Akech S, et al. Tackling health professionals’ strikes: An essential part of health system strengthening in Kenya. BMJ Global Health. BMJ Publishing Group; 2018. p. 1136.
Joseph Wu T-S, Kagoli M, Kaasbøll JJ, Bjune GA. Integrated Disease Surveillance and Response (IDSR) in Malawi: Implementation gaps and challenges for timely alert. Uthman O, editor. PLoS One. 2018;13:e0200858.
Ola O, Sedig K. Beyond simple charts: Design of visualizations for big health data. Online J Public Health Inform. 2016;8:e195.

Download PDF

Journal Publication

published 06 Jan, 2021

Read the published version in BMC Medical Informatics and Decision Making →

Editorial decision: Major revision
11 Sep, 2020
Reviewer #2 agreed at journal
10 Sep, 2020
Review #2 received at journal
10 Sep, 2020
Review #1 received at journal
08 Sep, 2020
Reviewer #1 agreed at journal
22 Aug, 2020
Reviewers invited by journal
19 Aug, 2020
Editor assigned by journal
07 Aug, 2020
Submission checks completed at journal
06 Aug, 2020
Editor invited by journal
06 Aug, 2020
First submitted to journal
05 Aug, 2020

You are reading this older preprint version

Read the latest preprint version →

Evaluating performance of health care facilities at meeting HIV-indicator reporting requirements in Kenya: An application of K-means clustering algorithm

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Methods

Study setting

Study design

Data source

Measures

Facility reporting completeness and timeliness

Outcome Measures

Data Analysis

Results

Scatter chart visualization of HTC performance clusters

Discussions

Conclusions

Abbreviations

Declarations

Ethics approval

Consent for publication

Availability of data and materials

Disclaimer

Competing interests

Funding

Authors' contributions

Acknowledgments

References

Status:

Journal Publication

Version 1