Evaluation of eHealth Implementations in Uganda: Investigating Practices, Challenges and Insights

The application of information and communication technology is becoming more popular in healthcare management evidencing improvement of effectiveness, access, quality, and efficiency of the healthcare systems. With increased investment and implementation of eHealth across the world, there is a need to evidence its value. That is, its evaluation is required in order to get the most benefits out of them. To this end, this research study investigates the practices, challenges, and insights regarding the evaluation of eHealth implementations in Uganda. A qualitative approach was employed to conduct the study investigation with key eHealth implementers in Uganda considered as respondents to establish an understanding of their perspectives with respect to ehealth evaluation practices and challenges faced, as well as to derive insights from these perspectives in relation to the World Health Organization (WHO) understanding of digital health evaluation. Results show that Uganda has implemented various eHealth initiatives; however less to none evaluation is undertaken, as it is not a key activity with most of the eHealth implementers. The focus is put on monitoring the ehealth initiatives’ functionality and adoption rather than their outcome and impact.


eHealth Implementation Challenges
Difficulties and challenges in eHealth implementation are an international phenomenon to all countries irrespective of their development status [28]. Some aspects that threaten ICT systems implementation in the health sector involve economic resources, income disparities, exorbitant costs of usage fees, excessive costs for even rudimentary health information systems, lack of human trained resources, lack of governmental policies that address a well-defined health system that incorporates eHealth, cultural aspects and some resistance to the use of computers for health care processes [29], absence of rigorous evaluation research of such technologies on health outcomes [30], challenges in systems integration [31,32] and other organizational barriers to health information technology uptake [33]. Scott & Mars [16] noted that for most developing countries, eHealth remains a proof-of-concept activity, with only modest value demonstrated within small pilot projects.

An Overview of eHealth Implementations Evaluation
The concept of evaluation can be defined as a systematic and objective assessment of an intervention that aims to determine the fulfilment of objectives, efficiency, effectiveness, impact, and sustainability [34]. Yusof et al [35] point out that the questions of why (reason for conducting an evaluation), who (which stakeholders' view is being evaluated), when (which phase of system development life-cycle is being evaluated), what (aspects of the system are being evaluated), and how (choice of the research approach) need to be answered upon undertaking an evaluation. WHO [10] further defines evaluation, as measures taken and analysis performed in order to assess the interaction of users or a health system with the digital health intervention strategy, or changes attributable to the digital health intervention. Throughout the implementation of eHealth initiatives, their evaluation is required in order to get the most benefits out of them [36,37]. Related to 6 evaluation is monitoring. Though in most cases monitoring and evaluation are conducted concurrently, the two concepts are different in the context of measuring performance and impact of eHealth implementations. WHO [10] emphasizes that monitoring is the routine collection, review, and analysis of data intended to measure implementation progress for an eHealth initiative, and results into adjustments in intervention activities necessary to maintain or improve the quality and consistency of the eHealth deployment. In contrast, evaluation measures changes in outcome and impact that are attributed to the ehealth initiative. WHO [12] observes that monitoring and evaluation of eHealth implementations plays an essential role in demonstrating the progress that a country is making towards the development of its national eHealth environment. Lau & Kuziemsky [9] note that an eHealth system covers not only the technical ICT artefacts but also the socioorganizational and environmental factors and processes that influence its behaviours, so they argue that the scope of eHealth evaluation can cover the entire life cycle, which spans the planning, design, implementation, use, and maintenance of the eHealth system over time; and that depending on the life cycle stage being evaluated, there can be different questions raised.
Evaluation of eHealth implementations is a challenging undertaking [9,38,39]. There are a few published evaluations on eHealth implementations [6,30,[40][41][42][43] especially in the developing countries [39]. The difficulty is because such evaluation does not focus on technology only but often needs to consider how the technology components interact with other processes in the eHealth implementation [44], which in turn broadens the scope of the evaluation [45,46]. Secondly, the evaluation takes place in a complex healthcare setting that involves multiple stakeholder categories (such as patients, clinicians, administrators, IT specialists, funders) on top of legislation, social, political and economic environments [47]. This poses challenges to the evaluation since different stakeholders 7 present different expectations and perspectives of a successful eHealth implementation, which may lead to conflicting evaluation criteria, and require multiple study designs and evaluation methods [38,48,49]. eHealth evaluations are also resource-intensive and are always hampered by insufficiency of resources like time, funding, human resources, and subject participants [38].
Notwithstanding the challenges, eHealth evaluation efforts are worth undertaking [50].
Implementers and countries that have evaluated their eHealth implementations have benefited from the knowledge about results of the implementations in the respective programmes [9] and this knowledge base helps to inform decisions on policies, practices, and research [51]. In Europe, the topic of impact assessment as well as evaluations for eHealth had gained considerable momentum by 2011 to an extent that half of the countries had designated a specific body/institution that was responsible for eHealth evaluation activities . Various Canadian eHealth evaluation studies evidenced positive benefits from the implementation of electronic medical records and drug information systems [52][53][54], and such helped to answer questions concerning whether there was sufficient value for money on Canadian electronic health records investments which were  [56]. Evaluation done for the United Kingdom's implementation and adoption of the nationwide electronic health records system indicated limited visible benefits for clinicians and patients, and it guided the eventual closedown of the initiative [57][58][59][60]. An assessment that sought to find out the successes and challenges of eHealth in Africa and developing countries [61] indicated that most of the initiatives lacked documentation and proper evaluation hence their overall success was uncertain, but led to recommendations that would guide future implementations to do well. All the above cases communicate how eHealth evaluation has been given attention in some countries and how the evaluation results have been useful to inform decisions.

Status of eHealth in Uganda
Uganda, like most developing countries, has employed eHealth applications to improve healthcare delivery and public health [16]. The growth in ICT created a fertile environment for new innovations whose application into the Uganda health industry has yielded positive results, especially in disease control and prevention through disease surveillance [62]. Some of the famous eHealth systems implemented in Uganda include DHIS2 which supports routine health data reporting from the district level to the national level [63], mTrac an SMS, USSD and web-based data collection tool for health workers at district health centers to submit weekly HMIS reports related to disease outbreaks and stock outs of essential medicines [64], and OpenMRS an electronic medical records application to support records management functions especially in HIV-care health facilities [63] among others.
Although Uganda experiences various eHealth projects, most of them are pilot projects, operated in silos and lack sustainability [64,65]. For example, there were approximately 23 of 36 mHealth initiatives in 2008 and 2009 that did not move beyond the pilot phase [66]. Such situations have been criticized as 'pilotitis', an expression of dissatisfaction 9 from donors and governments because of the isolated eHealth initiatives that are successful in one context, but not rolled out [67]. With such a large number of uncoordinated pilot projects, the Government of Uganda imposed a moratorium on new eHealth activities, which demanded new eHealth initiatives to be approved on condition that they met the existing Ministry of Health requirements [68]. The National eHealth Policy and Strategy [69] were also developed to guide the development and implementation of eHealth in the country.

Methods
We used qualitative methods [70] (3) were not contacted due to limitations to access their offices and contact details in the data collection period. Entry contacts to twenty-two (22) institutions were contacted, we explained the study objectives and asked to nominate their most appropriate staff that were involved in eHealth implementation or evaluation to participate in the interviews. Of the contacted, eighteen (18) institutions responded positively and nominated a staff to attend to the interview. Four (4) institutions did not participate because the nominated staff did not provide to researchers interview appointments.
Verbal consent to participate in the study was obtained from twenty-two (22) participants in eighteen (18) institutions, and face-to-face interviews were conducted on separate days at scheduled time at each participant's place of work. The first author (JA) conducted the interviews in English, each lasting between 60 -90 minutes. Participants' responses were recorded as written extensive notes, responses on each question were reviewed with each of the participants to ensure that no wrong data was carried over; and more field notes were also written immediately after each interview. The analysis of the interviews was done using the thematic content analysis approach [43] where both authors/researchers read all the notes to familiarise themselves with the text, then identified codes, categorised the codes and developed themes from the collected data. Quantitative information about the resultant codes and other quantitative responses were analysed using SPSS (Statistical Package for the Social Sciences) software. Feedback on the field findings was then shared with the MoH's DHI for review and identification of any obvious outliers in the collected data. The DHI did not identify any outliers; as such the findings presented in this paper reflect the true practices in eHealth implementation and evaluation in the country.

Results
A total of 22 interviews were conducted in 18 institutions. Out of the 22 respondents, 17 11 (77.3%) were males and 5 (22.7%) were females. Most of the respondents 12 (54.5%) were in the age bracket of 31-40 followed by 6 (27.3%) in the age bracket of 18-30 and 3 (13.6%) in the age bracket of 41-50. The respondents included a diverse range of cadres including programme managers, monitoring and evaluation officers, health informatics specialists, software developers, statisticians, and IT systems administrators.

Use of eHealth in work practices
All participants mentioned that their organisations used eHealth in their health related activities. 91% of the respondents indicated to be using eHealth to a great extent while only 9% indicated to be using eHealth to a certain extent in their activities (Table 1). Data collection and reporting 9 (41%) was the most common area of eHealth application followed by data analysis 4 (18%) and others as shown in Figure 2

Organisations' motivations for eHealth evaluation
Many participants reported that their institutions put efforts to evaluate the performance of eHealth, but also some organisations do not. 50% of the participants indicated that their organisations put efforts to a great extent, 18% to a certain extent, 23% to a very small extent and 9% not at all (Table 2) ".. to a very small extent because we do not normally conduct performance evaluations, but we sometimes want to ensure proper flow of system functionality to meet user requirements" -Participant 17 "… may be to some extent. We especially evaluate the system before implementation during user acceptance testing to ensure the system works as expected. The evaluation also reduces complaints from users" -Participant 10

Indicators monitored during eHealth evaluation
Participants reported various indicators that were currently considered during evaluations; most reported indicators being system availability, system response speed, interoperability, usability, scalability, and availability of human resources to implement the eHealth initiatives as shown in Figure 4  Discussion eHealth practices - Table 3 evidences that all institutions apply eHealth practices in some ways in the country [64,65]. From Figure 1 and 2, we observe that there are various areas of eHealth applications although few respondents use them; in other words, ehealth implementation in Uganda is not integrated but operated in silos [65]. Regards conducting eHealth evaluation, Table 4 indicates that only 50% of the respondents conduct ehealth evaluation to a great extent, while the rest it is done to a small extent (50%) or not at all.
This implies that there is no concerted culture of eHealth evaluation in Uganda. Looking into the reasons why evaluations are conducted (see Figure 3), most of the respondents provided reasons that are related to ensuring proper functionality of the eHealth initiatives. This is also reflected in the indicators measured in Figure 4, where system availability, response speed, interoperability, usability, scalability, and availability of human resources to implement the eHealth initiatives are the most measured indicators. World Health Organization [10] and WHO & ITU [12] categorise such indicators as process and output indicators that provide information and insight on the adoption of an eHealth initiative, and are more suitable for monitoring eHealth initiative implementation; however, they do not necessarily evaluate the performance of the eHealth initiative. This implies that even though more respondents conduct evaluations on their eHealth implementations, they majorly monitor eHealth deployment, functionality, and adoption rather than measuring outcome and impact that result from the eHealth implementations.  [61], and un-harmonised interpretation of ehealth performance indicators [9]. Other challenges reported by more than one respondent included limited resources (finances and time) to promote ehealth evaluation activities, unavailability of the definition of impact evaluation indicators [38] and stakeholders' attitude about the evaluation [71]. The challenges faced by implementers in conducting an evaluation of ehealth initiatives are more attributed to the fact that the country had no guidelines for eHealth implementation and evaluation, and implementers had not yet put efforts to building capacity that is relevant for the evaluation of eHealth implementations [38]. Though the country's National eHealth Policy and Strategy [69] was developed and launched in 2017, all existing eHealth initiatives in the country were implemented with no national guidance being followed. Worse still, there were no detailed guidelines for evaluating eHealth initiatives in the country.
Insights learned from eHealth implementations evaluation -From this study, we learned that implementers in Uganda undertake more of "monitoring" activities for their eHealth implementations as compared to their evaluation. That is, the implementers understood that such monitoring activities and efforts could also be used to evaluate the impact and contribution of the ehealth implementations to the main programme 18 objectives. This coincides with observations in Otto et al., [72] where only very few cases had their impact evaluation done out of the twelve eHealth cases studied across sixteen African countries. In their study, only Ethiopia's FrontLineSMS and Malawi's CommTrack were evaluated for impact; while for Uganda, both its RapidSMS and Trac FM were not evaluated [72]. Following guidance by WHO & ITU [12], activities and efforts for eHeath evaluation should consider observations and measurements beyond process and output indicators to also consider outcome and impact indicators for each of the eHealth implementation/initiative in question. In order to improve the practice of ehealth evaluation in Uganda, efforts are needed to support changing implementers' perspectives on eHealth evaluation; key effort being the development of an ehealth evaluation framework that will define the notion of "evaluation", its characteristics, and the indicators that should be measured with regards to the performance and impact of ehealth implementations in healthcare and service delivery for Uganda's health system.

Conclusions
In this study, we share findings from the exploratory study on eHealth implementations evaluation practices and challenges faced in Uganda. We learned that Uganda had implemented various ehealth initiatives that had neither followed the national ehealth implementing guidelines nor had they involved the government (i.e. Ministry of Health); despite its increased efforts such as the development of the eHealth Policy and Strategy to regulate and guide eHealth implementation in the country. We also learned that most of the ehealth implementations were actually "monitored", an activity that was erroneously used as "evaluation" of these initiatives. In other words, the process of evaluating the outcome and impact attributed to eHealth initiatives had not been a key activity with most of the eHealth implementers. This could be attributed to the fact that the country did not have guidelines on impact evaluation for eHealth implementations; as such, the ehealth implementers majorly monitored eHealth deployment, functionality, and adoption.
Notwithstanding the above challenges and lessons learned, our research findings can play a vital role in terms of providing the baseline situation on which health leaders and policymakers as well as the eHealth implementers can set improvement targets and action plans for strengthening and sustaining ehealth in Uganda. Accordingly, there is need for the development of ehealth evaluation guidelines and indicators that can be used to evaluate the outcome and impact of eHealth implementations in the country. Additionally, we advocate for the creation of awareness of the need to plan for eHealth evaluation in addition to monitoring activities during the planning of eHealth implementation programmes. This forms our next research steps, that is, establishing an all-encompassing ehealth evaluation framework to guide comprehensive evaluation for all Uganda's ehealth implementations in order to have a more sustainable digital health system for Uganda.

Declarations Ethics Approval and Consent to Participate
Following from the researchers' request and acceptance by the Ministry of Health, Uganda, verbal consent was obtained from all the participants. The verbal consent was accepted because majority of the participants are members of the eHealth Technical Working Group (EHTWG), which is chaired by the Permanent Secretary of the Ministry of Health.

Consent for Publication
Not applicable.

Availability of data and material
The materials/articles used in this review are available upon request from the corresponding author.

Competing Interests 20
The authors declare that they have no competing interests.  Not at all 2 9. Reasons for evaluating ICT application in the health sector.
32 Figure 4 Indicators measured during eHealth evaluation.
33 Figure 5 Challenges and limitations during eHealth evaluation.