Evaluating Feasibility and Acceptance of a Mobile Clinical Decision Support System in Botswana

doi:10.21203/rs.3.rs-1558725/v1

PDF

Research Article

Evaluating Feasibility and Acceptance of a Mobile Clinical Decision Support System in Botswana

https://doi.org/10.21203/rs.3.rs-1558725/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

The health workforce in Botswana consists mainly of medical officers and primary care nurses working in remote areas with limited training and insufficient reference materials to support diagnosis and management of diseases in dermatology and other subspecialties. This suggests a need for clinical decision support tools for these healthcare providers. VisualDx, a well-established mobile clinical decision support system, offers a promising solution with documented benefits. However, implementation of eHealth systems is commonly associated with challenges. To inform sustainable implementation of VisualDx in Botswana, it is important to evaluate intended users’ perceptions of the technology. This study aims to determine healthcare workers acceptance of VisualDx to gauge feasibility of future adoption in Botswana and other similar healthcare systems.

Methods

The study's design was informed by constructs of the technology acceptance model. A convergent mixed methods feasibility study involving surveys and semi-structured interviews was conducted. The Research Electronic Data Capture (REDCap) platform supported online data capture from March 2021 through August 2021. Twenty eight healthcare workers across Botswana participated in the study and their usage data was tracked over time. Descriptive statistics were generated and analyzed using Excel and REDCap. Thematic analysis of interview transcripts was performed using Delve.

Results

All initial survey participants (n = 28) expressed interest in using mHealth technology to support their daily work. Prior to VisualDx access, participants referenced textbooks, journal articles, and Google search engines among others. Overall, participants' survey responses showed their confidence in the VisualDx platform (94.7%); however, some barriers were noted. Frequently used features included generating a differential diagnosis through manual entry of patient symptoms (48.5%) or using the artificial intelligence feature (22.0%). Seventeen (60.7%) participated in virtual interview sessions at different times and ultimately 4 themes (governance, infrastructure, human resource capacity development and usability) were derived.

Conclusions

Participants' responses indicated acceptance of the VisualDx platform. The ability to access information quickly without internet connection is crucial in resource constrained environments. Select enhancements to VisualDx may further increase the feasibility of a national roll-out in Botswana, improving acceptance and successful adoption of the platform by general practitioners needing decision support for dermatology and other subspecialties.

VisualDx

eHealth

Technology Acceptance Model

Clinical Decision Support

Botswana

Dermatology

In recent years, Clinical Decision Support Systems (CDSS) have become increasingly popular. This trend is partly spurred by the current COVID-19 pandemic requiring effective and efficient use of clinical data to inform strategic decision making. Globally, health systems are beginning to embrace CDSS to augment the limited health human resource while reducing physical contact with physicians where possible. CDSS, defined as “information communication technologies (ICTs) that provide healthcare workers and patients with situation-specific advice that can inform their decision making”, [1] are popular in the developed world and are increasingly becoming popular in the developing world [2]. Botswana is one developing country in sub-Saharan Africa (SSA) that has embraced eHealth - “the use of ICTs for health” [3], as a means towards, 1) improving access to healthcare and provision of equitable healthcare in remote facilities; 2) achieving better customer satisfaction, improved patient outcomes and quality of care; 3) providing quick access to health information across the entire health sector; and 4) improving monitoring and evaluation of healthcare services [4].

Over the years, Botswana has been successful in achieving several health-related millennium development goals by reducing under five mortality, the spread of HIV, reversing the incidence of malaria and other major diseases, and improved access to safe drinking water and basic sanitation [5]. Although less severe than elsewhere in SSA, Botswana has a recognized inadequacy of health human resources, most significantly in primary healthcare [6]. A low doctor to patient ratio (3.8/10,000) as well as a shortage of medical specialists is also reported [7]. Access to dermatological services continues to be one of the major challenges across the public and private sectors in Botswana, with most available services clustered in the capital city of Gaborone [8, 9]. The number of dermatology specialist providers in Botswana’s public healthcare system has varied from none to most recently 2 full time Ministry of Health and Wellness (MOHW) employees and three contract specialists from Cuba. However, the demand for dermatology care in Botswana continues to be much higher than can be provided by the current dermatology specialists and waiting times for appointments can be six or more months [9]. Consequently, the MOHW has provided care to patients with skin diseases through a combination of providers who lack sufficient training and reference materials at the point of care to help with the diagnosis and management of skin conditions seen in this environment. The majority of patients with skin complaints are initially seen at their local clinic by a general nurse or physician with little experience or knowledge of dermatology. Anecdotal evidence has shown that without access to dermatology treatment guidelines or reference material for dermatology, healthcare workers are rarely able to adequately address patient’s skin concerns. Consequently this affects patients’ outcomes as they may be given no diagnosis or treatment, the wrong diagnosis and treatment or referred directly to dermatology and required to wait until their appointment to receive help. They may be trialed on a plethora of indiscriminate treatments to address their skin concerns. This can result in wasted time, money, exposure to unnecessary side effects and in some cases significant morbidity, particularly in the case of skin cancers that can go undiagnosed and grow to an untreatable size before the patient reaches dermatology [10].

The shortage of dermatology specialists in Botswana necessitates efficient use of the limited resources and continuous empowerment of those commonly engaged in the management of prevalent skin conditions [11]. This suggests a critical need for a CDSS to ameliorate current challenges within dermatology and other subspecialties in Botswana. Previous studies have demonstrated the promise of VisualDx CDSS as a platform which could contribute to increased provider confidence and a reduction in diagnostic errors in primary care settings [12, 13].

The VisualDx platform has been reported to have the potential to enhance diagnostic accuracy, aid therapeutic decisions, and improve patient safety [14]. It offers a unique user interface to support diagnosis based on visual signs which could reduce diagnostic errors, improve provider job satisfaction, and improve quality of care not only in dermatology but across all areas of medicine. To date, the value of VisualDx has been evidenced by a wide network of users around the world following its use at over 2,300 facilities including universities, hospitals and other clinical sites in over 100 countries. The platform combines machine learning algorithms and vision science with a structured clinical knowledge base to allow non-specialist healthcare providers to capture patient-specific findings, build custom differentials, and view images and treatment recommendations. An offline capability has recently been developed within VisualDx to further increase its potential utility for remote settings with limited to no internet connectivity.

Notwithstanding the potential benefits of the VisualDx CDSS, previous studies have demonstrated that the implementation of any eHealth system is commonly associated with challenges [15, 16]. There was limited prior data to inform the implementation of VisualDx as a CDSS to support dermatology services in Botswana. As such, a better understanding of the feasibility and acceptance of the tool by healthcare workers (HCWs) was needed to inform ways of adapting it to the Botswana context and to guide sustainable implementation approaches across the health sector. A review of technology acceptance and adoption models by Taherdoost et al., [17], identified several models and frameworks explaining user adoption of new technologies and presenting factors that can affect user acceptance of the technology. Similarly, previous studies have identified the Technology Acceptance Model (TAM) as prominent amongst key theoretical approaches used to understand people’s intentions to accept various forms of ICTs [18–20]. The TAM centers on three belief constructs that have been found to significantly influence an individual’s, 1) acceptance of (intention to engage) a technology: 2) perceived usefulness; and 3) perceived ease of use [21]. It contends that a relationship exists between one’s intention to use technology and their actual usage behavior [22].

Considering the novel use of the VisualDx CDSS in Botswana for dermatological services, and the need to evaluate its feasibility, this study aims to use the TAM to determine the acceptance of VisualDx and inform future adoption or adaptation strategies to avoid future implementation challenges. Study findings will inform policy decisions and next steps towards a national rollout of VisualDx in Botswana and similar developing countries.

Study participants

Healthcare workers supporting dermatology clinics and medical students participating in dermatology coursework or rotations at health facilities and universities across Botswana were sent an email and WhatsApp invitations to participate in the study through the eHealth Research Unit at the University of Botswana (UB). Consent forms were also provided via email to confirm participation. A total of 18 participants volunteered to take part in the study initially.

As COVID-19 restrictions were eased in Botswana, the Greater Gaborone District Health Management Team (DHMT) was engaged to recruit more nurses to participate for the remainder of the study duration. The DHMT is a local authority under MOHW tasked with overlooking management and staffing of primary care clinics. An additional 10 participants were enrolled with approximately three months remaining in the study period, resulting in a total of 28 participants enrolled from 20 sites (healthcare facilities and UB) in Botswana. Participants were based at 6 health districts (Greater Gaborone (21), Greater Palapye (1), Greater Phikwe (2), Greater Francistown (2), Maun (1) and Chobe (1)).

VisualDx Use

All participants used personal smartphones or tablet devices to download and install the VisualDx mobile application, with account credentials provided by VisualDx. They were offered mobile data vouchers to assist with the cost of data for the mobile application download and subsequent usage. Initial training with the original cohort of participants was conducted using the Zoom platform upon joining the study. Those recruited through the DHMT attended an in-person training session at the UB eHealth Research Unit. Training sessions covered information technology skills, demonstrations of Visual Dx application features and the practical application of Visual Dx to common dermatologic and general medical conditions seen in Botswana. All trainings were recorded and provided to participants who were unable to attend on the training day. Throughout the study duration, six case-based training sessions were provided to demonstrate the successful use of VisualDx to guide the clinical reasoning process.

Participants used VisualDx at their own discretion throughout the study period. A WhatsApp group was created to offer a platform for sharing announcements and seeking support related to the study.

VisualDx CDSS features

The VisualDx CDSS was developed by VisualDx and designed to be used on a mobile phone/tablet running an Android (Google, Inc., Mountain View, CA, USA) or iOS operating systems. For the purposes of this study, the system was used for decision support in varying settings for reducing diagnostic errors and suggesting management options for dermatological and other conditions. Offline capability was recently developed for Android devices and utilized to further increase the potential utility of VisualDx in areas of limited internet connectivity.

VisualDx features used include those allowing for 1) Searching directly for any of 4,000+ diseases for clinical information including therapy options, best tests, and management pearls (Figure 1), 2) Building a custom differential diagnosis based on chief complaints across all fields of medicine (Figure 2), and 3) Taking photos of a skin condition, and through in-built artificial intelligence techniques, receive a list of possible conditions matching the images provided (Figure 3).

Screenshots of the CDSS features

Data collection

A convergent mixed methods design [23] was used to assess the feasibility and acceptance of VisualDx as a CDSS tool in Botswana. Quantitative data was collected from a series of three surveys delivered at the beginning, middle and end of the study. The survey was designed in 3 parts to assess any changes over time in participants’ acceptance as they became familiar with the system. RG and NS created the first draft of the survey questions and the interview script. Survey questions were then reviewed and edited by all authors to avoid noted ambiguities. After survey questions were configured in REDCap, pre-testing of the surveys was conducted by RG, NS, and KN before being enhanced through improved branching logic.

VisualDx mobile application usage data was also collected from the VisualDx servers through existing event tracking mechanisms. Qualitative data from semi-structured interviews was collected (one interview with each participant, conducted concurrently with ongoing study activities). Both survey and interview tools were used by the authors to meet the study objective. Interviews further assisted in gaining in depth insight from participants.

All surveys were administered through the Research Electronic Data Capture (REDCap, (Vanderbilt University) system [24], with links provided to the participants for access on their personal or work devices. REDCap is a secure (HIPAA and GDPR compliant) system for supporting electronic data capture for research and operational support projects. The first survey was distributed to participants immediately following their initial mobile application training (March 2021). The second survey was delivered in the third month of the study period (May 2021). For the cohort of participants that started midway through the study, this survey was delivered after one month of participation (July 2021). The final survey was completed at the end of the study period (August 2021). Surveys had closed-ended questions (dichotomous, multiple choice and Likert scale) and open-ended questions.

Starting three months into the study period in June 2021, participants were contacted individually via WhatsApp to schedule semi-structured interviews. All interviews were conducted remotely via Zoom platform, with each interview recorded, transcribed verbatim, and reviewed by all researchers.

Sequential Query Language (SQL) statements were executed against the VisualDx database to obtain usage data associated with the study participants’ user accounts.

Data analysis

Quantitative data were summarized using descriptive statistics and the mean, median, range, and standard deviation calculated using the REDCap system. Interview transcripts were uploaded to Delve software for coding. Qualitative interview data was analyzed using Braun and Clarke’s widely accepted principles of thematic analysis [25] to categorize data into key themes which were later aligned to TAM constructs. Iterative transcript review and deductive coding [26] was performed independently by NS, RG, and MM. A predefined list of descriptive codes were developed and later discussed by all authors: “clinical decision support”, “eHealth”, “ease of use/usability”, “continuing education”, “internet connectivity”, “electronic health record”, “national implementation feasibility”, “technology acceptance”, “usage facilitators”, and “usage barriers”. Sub-codes were created to further categorize interview responses in more detail (shown in Appendix 3). The codes were later grouped into four broader themes: “governance”, “technology infrastructure”, “human resource capacity development”, and “usability”.

Usage data associated with the study participants’ user accounts was analyzed in Excel to generate basic descriptive statistics related to frequency of use, mode of access, and features most commonly utilized.

Integration of the mixed methods was achieved through a weaving narrative report, connecting themes from the quantitative and qualitative data collected.

The surveys, interviews, and usage data analysis were guided by the TAM constructs of perceived acceptance, usefulness, and ease of use of the VisualDx CDSS.

Ethics approval

The study protocol was approved by the University of Botswana’s IRB (UB: UBR/RES/IRB/BIO/223) and the Botswana Ministry of Health and Wellness (MOHW: HPDME: 13/18/1) in December 2020. The approved protocol was implemented over six months from March 2021 through August 2021.

Of the 28 study participants, 9 (32.1%) were in the age range 20-29 years, 14 (50.0%) in the range 30-39 years, and 4 (14.3%) aged 40-49 years. One participant did not specify their age range. Twelve (42.8%) of participants were doctors, another 12 (42.8%) were nurses, and 4 (14.3%) were medical students. Fourteen (50.0%) of participants specified their primary place of work as an outpatient clinic, 35.7% worked in a primary hospital, 3.6% in both clinics and hospitals, and 10.7% did not specify. Twenty (71.4%) of participants practiced general or family medicine, 7.1% specialized in dermatology, 7.1% specialized in pediatrics, and 14.3% practiced another medical specialty. Appendix 1 and Figure 4 shows total number of study participants per location and the geographical representation of study sites on the Botswana map, respectively.

Seven (25.0%) of participants successfully downloaded and used VisualDx in offline mode. This includes those participants who did not have access to offline mode (i.e. participants using iOS devices). Of participants who had access to offline functionality, 28.0% (7 of 25) were able to successfully download and use VisualDx’s offline features. Figure 5 and 6 shows the weekly usage summary and usage by operating system, type of use case, as well as offline and online use, respectively.

Surveys

Pre-pilot Survey Responses

The first survey was distributed before the pilot began in March 2021 and had a 100% response rate (n = 28 out of 28 total participants). Questions were primarily focused on participants’ interest level in eHealth technology (Table 1), any existing familiarity with VisualDx, and existing tools currently used for diagnosing and treatment of dermatological conditions (Table 2).

Table 1: Participants interest in using mHealth or eHealth technologies (TAM related element from the pre-pilot survey)

Question

Mean (SD)

Median

Mode

How interested are you in using mHealth or eHealth technology to support your daily work? (Likert scale: 1: Not interested at all, 5: Very interested)

4.7 +/- 0.53

Table 2: Tools currently used for clinical decision support

Tools currently used for clinical decision support	No. of participants for each tool used
Textbooks	23
Journal articles	11
Google search Engine	24
Other tools	Medscape, Uptodate Accessmedicine AMBOSS, Osmosis VisualDx, U-central EMGuidance Merck manuals Programme guidelines and Protocols

Mid-pilot Survey Responses

The second survey was distributed in early June 2021 and had 78.6% response rate (n = 22 out of 28 total participants). The survey identified participants' barriers to using VisualDx as well as their perceived ease of use and usefulness of the VisualDx platform (Tables 3).

During the mid-pilot survey, VisualDx was either used every day by 2 (9.1%), a few times a week by 13 (59.1%), a few times a month by 4 (18.2%), or once every couple of months by 3 (13.6%) participants. Fifteen (68.2%) of survey participants used VisualDx during a patient encounter, 11 (50.0%) immediately before or after a patient encounter, 18 (81.8%) as a studying or educational tool outside of work while other uses contributed 2 (9.1%).

Fifteen (68.2%) survey respondents indicated that they have not encountered any issues or barriers when trying to use VisualDx, while 7 (31.8%) highlighted the following barriers;

Lack of reliable internet: 4 (18.2%),

I find VisualDx difficult to use (1, 4.5%),

I could not find a good time to use VisualDx in my clinical workflow: 4 (18.2%)

I do not trust the results or content in VisualDx: 1 (4.5%)

Table 3: Mid-pilot perceived ease of use and perceived usefulness (TAM related elements from the mid-pilot survey)

TAM Construct	Question	Mean (SD)	Median	Mode
Perceived ease of use	How easy is it to use VisualDx? (1 = Very difficult, 5 = Very easy)	4.3 +/- 0.83	4.5	5
Perceived usefulness	Rate the quality and relevance of the diagnosis content (text and images) in VisualDx (1= Very irrelevant) (5= Very relevant)	4.4 +/- 0.73	4.5	5

Survey participants further reported VisualDx to be useful in the following ways; 1) Changing a patient’s treatment plan based on information in VisualDx 7 (36.8%), 2) Diagnosing or treating a patient 14 (73.7%), Educating a patient on their condition or sharing photos with them 10 (52.6%), and 3) Confirming a suspected diagnosis 11 (57.9%).

Other useful scenarios for VisualDx during mid-pilot survey are noted in Appendix 2.

Post-pilot Survey Responses

The final survey was distributed at the end of August 2021 and had a 67.8% response rate (n = 19). Some questions from the second survey were repeated to assess change over time, and additional 1-5 Likert scale questions and dichotomous questions were asked to assess participants’ perceptions of VisualDx. These were aligned to the TAM constructs (Table 4).

The third survey showed that VisualDx was either used every day by 1 (5.3%), a few times a week by 10 (52.6%), a few times a month by 4 (21.1%), or once every couple of months by 3 (15.8%) and 1 (5.3%) participants rarely or never used VisualDx.

Fifteen (68.2%) of survey participants used VisualDx during a patient encounter, 11 (50.0%) immediately before or after a patient encounter, 18 (81.8%) as a studying or educational tool outside of work while other uses contributed 2 (9.1%).

The third survey highlighted 8 (42.1%) participants having not encountered any issues or barriers when trying to use VisualDx, while 11 (57.9%) reported the following barriers while trying to use VisualDx;

Lack of reliable internet: 7 (36.8%)

I find VisualDx difficult to use: 1 (5.2%)

I could not find a good time to use VisualDx in my clinical workflow 4 (21.1%)

Other barriers, 2 (10.5%)

Eighteen (94.7%) survey participants found ‘VisualDx easy to access and easy to use’, while 1 (5.3%) disagreed with the statement.

Table 4: VisualDx Perceived Ease of use, Usefulness and Acceptance during the post-pilot survey (TAM related elements from the post-pilot survey)

TAM Construct	Question (likert scale)	Mean (SD)	Median	Mode
Perceived ease of use	How easy is it to use VisualDx? (1 = very difficult, 5 = very easy)	4.4 +/- 0.77	5	5
Perceived usefulness	Rate the quality and relevance of the medical content in VisualDx. (1 = not relevant at all, 5 = very relevant)	4.4 +/- 0.68	4	5
Acceptance	How would you rate your comfort level with diagnosis and treating dermatology issues? (1: not comfortable at all, 5: very comfortable)	3.8 +/- 0.90	4	3

Perceived usefulness of the VisualDx platform was further highlighted during the third and final survey as summarized in Tables 5 and 6 respectively.

Table 5: VisualDx Perceived Usefulness during the post-pilot survey (Agree/Disagree/Neither Agree nor Disagree) (TAM related elements from the post-pilot survey)

Question (multiple choice)	Agree	Disagree	Neither
I feel that the diagnosis content and images in VisualDx are relevant to my practice.	18 (94.7%)	0 (0.0%)	1 (5.3%)
VisualDx has helped me educate patients and build patient trust.	16 (84.2%)	0 (0.0%)	3 (15.8%)
VisualDx does not go far enough in recommending next steps.	3 (15.8%)	15 (78.9%)	1 (5.3%)

Table 6: VisualDx Perceived Usefulness scenarios during the post-pilot survey (TAM related elements from the post-pilot survey)

Question (multiple choice)	Yes	No	Not sure
Have you encountered any scenarios where VisualDx provided a clear benefit to you or your patient?	16 (84.2%)	3 (15.8%)	0 (0.0%)
Do you feel that the information you gain from VisualDx helps you to make more accurate diagnosis?	17 (89.5%)	0 (0.0%)	2 (10.5%)
Has VisualDx made your clinician work easier?	17 (89.5%)	0 (0.0%)	2 (10.5%)
Has VisualDx helped you diagnose and manage skin disease?	18 (94.7%)	0 (0.0%)	1 (5.3%)
Has VisualDx improved your ability to manage non-dermatologic conditions?	15 (78.9%)	1 (5.3%)	3 (15.8%)

At the end of the study period, participants demonstrated overall acceptance of the VisualDx platform (Table 7).

Table 7: VisualDx Perceived Acceptance (Agree/Disagree/Neither Agree nor Disagree) (TAM related elements from the post-pilot survey)

Question (multiple choice)	Agree	Disagree	Neither
I feel more confident in my work knowing that I have VisualDx available as a reference tool.	18 (94.7%)	1 (5.3%)	0 (0.0%)
I found it challenging to find the right time in my clinical workflow to use VisualDx.	10 (52.6%)	5 (26.3%)	4 (21.1%)
I will continue to use VisualDx in my day-to-day work.	19 (100.0%)	0 (0.0%)	0 (0.0%)

Interviews

Seventeen of the 28 participants (60.7%) participated in virtual interview sessions conducted at different times throughout the study period. Interviewees described their daily work, challenges encountered on a regular basis, and overall experiences/perceptions with VisualDx. Participants’ responses were categorized into 4 themes (governance, infrastructure, human resource capacity development and usability), which highlighted possible factors that could affect the sustainable implementation of the VisualDx platform in Botswana (Table 8).

Table 8: Thematic presentation of factors affecting the sustainable implementation of VisualDx (Themes deduced from survey open ended questions and interviews)

Theme	Example Quotes
Governance	“There's a lot of lack of resources in Botswana. So we couldn't necessarily do all the tests that they recommended. And in terms of management, a lot of the medication that they recommend is not exactly here, we need to find something similar in that drug class.” “I think this is a great idea and innovation. So my thinking is…if all doctors can be able to get the app and use it, it will make a difference. Because since I got the VisualDx, I don't remember the last time I pulled in experts to ask about a skin problem. I don't remember. So I think it can make a difference.”
Technology Infrastructure	“When I was downloading it - when we were in UB - it was taking time to download. I think it was the network issues but I managed to download (VisualDx offline content). But then I think it was within maybe 30 to 45 minutes or so.” “My other problem is my camera...when someone presents with a skin condition I got a problem to take a photo because of my camera.”
Human Resource Capacity Development	“You know, I'm not an electronic oriented person.” “So for me, things like VisualDx is a way to keep refreshing my brain as a practicing doctor as to the things that I may have forgotten because unfortunately in Botswana we only have ourselves sometimes to depend on, especially in the district other than the call away physician.”
Usability	“…when you look into management of certain cases, for example, if I look into B12 deficiency and then it gives you management, it's not very precise in how you have to manage it in terms of dosage, or the frequency of doing blood tests and what not. And I felt that was a bit lacking and in management section for a couple of things that I looked up.” “So when you see a case, like the first three suggestions, it doesn't speak to our population.” “And once more, it is easy and friendly to use in setups where you are consulting many patients, it doesn't take too long to use the tool.” “You can even get the answers right there with the patient.” “So it's really useful to have offline. And it's advisable in Botswana to always have for all applications an offline mode.” “And you get more specific diagnosis, I do not know if specific is the right word, but maybe tailored diagnosis like more towards what you are looking for, especially with the skin of colour option, it really made a huge difference for me.” “[VisualDx] is phenomenal, I love it. And it has just really made a huge difference.” “I think it's an excellent app. Especially for the dermatological cases.”

Overall, this study demonstrated the potential for acceptance of the VisualDx platform by HCWs in Botswana. All participants in the initial survey (28/28), expressed interest in using mHealth or eHealth technology to support their daily work (Table 1). The willingness to use or learn mHealth was previously identified as an important factor towards technology acceptance [27], in addition to perceived ease of use and perceived usefulness, as outlined in the TAM. Majority of responses showed positive statements towards TAM constructs (Table 3 (22/28; mode: 5), and Table 4 (19/28; mode: 5)). Based on our findings, successful acceptance of the VisualDx platform by healthcare workers in Botswana could be assumed; however, other factors that could influence acceptance of VisualDx were noted and organized into themes - governance, technology infrastructure, human resource capacity development and usability (Table 8). These are essential towards influencing technology acceptance and also highlight an organization’s readiness to adopt or adapt a new technology. Similarly, previous studies have associated failure of eHealth system implementations with the lack of eHealth readiness (the preparedness of healthcare institutions or communities for the anticipated change brought by programmes related to information and communications technology) [28, 29]. Moreover, constraints to the adoption of eHealth in Africa have been previously reported by the WHO to include low ICT budgets, poor infrastructure for communication, erratic electricity supply and inadequate human resource capacity [30], all of which are reflections of lack of readiness.

Barriers to implementation of the VisualDx CDSS were reported and aligned to the identified themes. Some of the barriers encountered when trying to use VisualDx are consistent with those found in previous studies of MCDSS, including perceived irrelevance of information by some participants, lack of technical skill or savvy, and lack of access to technology or internet connectivity [31, 32]. A recent study by Zakerabasali et al., also highlighted the importance of understanding barriers to adoption of mHealth applications among providers and engaging them in the adoption process as that is essential for their successful implementation [33].

In essence, the acceptance of VisualDx CDSS can be greatly influenced by its perceived ease of use and perceived usefulness. These factors have been previously documented as positive influences towards technology adoption [34].

Perceived Ease of Use

Results relating to the perceived ease of use of Visual Dx CDSS were largely positive, as evidenced by the modal score of 5/5 on the Likert scale indicating “very easy to use” on the second and third surveys (Tables 3 and 4).

The majority of interviews highlighted that VisualDx is easy to use due to user friendly interfaces enabling quick information retrieval at point of care. The importance of supporting user-friendly interfaces with real-time feedback and decision support capabilities in mHealth solutions was also highlighted in previous studies [35-38].

Despite the reported perceived ease of use, 4 (23.5%) interviewees cited usability concerns about the VisualDx platform suggesting;

‘Differential diagnosis results as too broad/not enough recommendation of next steps’, for example,

“…when you look into management of certain cases, for example, if I look into B12 deficiency and then it gives you management, it's not very precise in how you have to manage it in terms of dosage, or the frequency of doing blood tests and what not. And I felt that was a bit lacking and in management section for a couple of things that I looked up.”

Others interviewees 3 (17.6%), reported ‘Perceived lack of relevant information in the app’, for example,

“So when you see a case, like the first three suggestions, it doesn't speak to our population.”

Barriers to using VisualDx including lack of technical savvy, and lack of reliable internet access were also reported, suggesting the need to strengthen health human resource capacity development as well as the provision of adequate technology infrastructure. This further suggests an ongoing need for eHealth providers to tailor solutions to the contexts in which they are being delivered, including medical content, delivery of training resources, and offline access. These measures were also previously suggested in dealing with inadequate technology infrastructure and lack of skilled personnel to support mHealth interventions [27]. Considering that VisualDx is designed to be adaptable to specific country context, participants’ responses could directly influence further improvements to medical content and features of the platform. Overall, responses related to the ease of use construct were positive (Table 3, Table 4).

Perceived Usefulness

Study participants identified the VisualDx CDSS to be useful overall, highlighting its potential to impact multiple areas. The range of features available in VisualDx lent to its perceived usefulness across the study population, as different providers found the application useful for different situations (during a patient encounter, immediately before or after a patient encounter, and as a studying or educational tool outside of work). The perceived relevance of VisualDx medical content, and the convenience of mobile and offline access are consistent with previous MCDSS studies [39].

VisualDx was perceived to be particularly useful in supporting point-of-care decision making, patient outcomes and engagement, access even when there is unreliable connectivity, reduction in referrals to specialists and support for continuing medical education and professional development.

By the end of the study, 89.5% of survey respondents reported that VisualDx generally helped them make more accurate diagnosis, and 94.7% indicated that VisualDx helped them diagnose and manage skin disease specifically (Table 6). These findings emphasize the perceived usefulness of the VisualDx CDSS and its relevance towards supporting diagnosis across all areas of medicine, especially dermatological conditions.

Contrary to a recent study where patients highlighted lack of confidence in the mHealth system [40], VisualDx was considered useful towards educating patients as well as building patients’ trust during encounters. Sixteen (84.2%) of surveyed participants in the post-pilot survey indicated that they had encountered a situation in which using VisualDx provided a clear benefit, and 84.2% also indicated that VisualDx helped to educate patients and build patient trust throughout encounters (Table 5, Table 6). The increased user confidence in the VisualDx CDSS could also be a result of some measures put in place by VisualDx Corporation to ensure data privacy and confidentiality. Notably, VisualDx collects only anonymized and generalized demographic information about the patient to provide a differential diagnosis. Even when using the ‘DermExpert’ AI tool, the image of the patient remains on the device at all times and is discarded immediately after the analysis is complete. This alleviates any data security concerns and allows the tool to conform to data protection standards such as the Health Insurance Portability and Accountability Act (HIPAA) [41] and the General Data Protection Regulation (GDPR) [42].

Of all VisualDx usage throughout the study, 21% was in offline mode. This includes those users who did not have access to the offline capability because they were using iOS devices (Figure 6). This suggests a significant portion of users encounter situations of limited connectivity regularly and choose to use VisualDx offline as a mitigation. Notably, however, users do still need internet connection to complete the download of offline content. 82.4% of interviewed participants said that lack of reliable internet was a barrier to using the VirtualDx CDSS (Table 8). Kabukye et al., highlighted the need to address inadequate computer infrastructure challenges prior to EHR implementation [43].

mHealth tools were previously reported to have the potential to upskill non-specialist healthcare workers, allowing them to address more issues than they otherwise might not be able to address without specialist guidance [44]. By the end of the study, case referrals to a specialist or another provider were reported less than one time per week by 8 (42.1%), 1-3 times per week by 6 (31.6%), 3-6 times per week by 3 (15.8%), and 7 or more times per week by 2 (10.5%) participants. At the beginning of the study, the modal response for frequency of referrals was 1-3 times per week (51.9%), while at the end of the study after using VisualDx the modal response to the same question was less than 1 time per week (42.1%). This is of particular significance in Botswana, where the healthcare system is experiencing a shortage of medical specialists and especially dermatologists [8,9].

Majority (81.8%) of survey respondents used VisualDx outside of their work as a studying or educational tool. Also, 23.5% of interviewed participants expressed that using VisualDx allowed them to stay up to date on the latest best medical practices and challenge the way that they have handled certain conditions in the past. The ability to keep users’ knowledge and skills fresh by using an eHealth application is a facilitator of use.

Actual usage statistics helped to more specifically identify the situations participants found VisualDx to be most useful. The use case for building a differential diagnosis accounted for 70.0% of usage, while 30.0% of usage came from participants searching directly for a specific condition. Of the differential diagnosis uses, 31.4% were generated using the ‘DermExpert’ machine learning algorithm, while 68.6% were differentials generated by manually entering a custom set of symptoms (Figure 6). This usage pattern suggests that users in Botswana perceive the differential diagnosis features of VisualDx to be most useful, whether they are using the tool in situations of uncertainty, looking for a second opinion, or confirming that they are not missing any diagnostic possibilities. The relatively low usage of the direct diagnosis search feature could suggest that users are well trained to handle the conditions that they are already familiar with, or they have other tools or references (for example, MedScape, UpToDate, and others listed in Table 2) that they tend to access for this use case.

Acceptance

Participants’ general acceptance of VisualDx as a MCDSS tool was confirmed by multiple data points. Sixteen (94.1%) of interviewed participants indicated positive overall user satisfaction (Appendix 3), and 52.9% expressed the importance of a nationwide rollout of VisualDx across Botswana to help upskill the country’s general practitioner workforce and reduce stress on referral hospitals (Table 8). Further, 100% of survey respondents indicated that they intend to continue using VisualDx after completion of the study (Table 7).

The study protocol required deviation from the initial plan primarily due to restrictions and delays caused by the COVID-19 pandemic. Most notably, almost all training sessions and interviews had to be conducted remotely via Zoom platform due to restrictions on in-person gatherings. Researchers were unable to visit clinics and facilities in person to provide support and further training. All outreach and follow-ups had to be completed through WhatsApp messenger.

Additionally, participation was limited as COVID-19 resulted in some participants being reassigned to efforts such as vaccine distribution or other scenarios which would not be applicable for VisualDx use. Additionally, compliance with filling in surveys and scheduling interviews was not 100%; some participants failed to complete these key data collection endpoints. This was likely due in part to the fact that almost all study activity was done remotely without in-person follow-up. Increased stress and workload due to COVID-19 surges in Botswana likely contributed to lack of compliance as well.

Future Direction

In line with the study findings, the authors have identified the following acceptance-related issues and associated mitigation strategies as essential to informing next steps:

Survey responses and themes from the interviews conducted pointed out areas of concern for widespread adoption such as the lack of locally specific guidance and content; lack of reliable internet connectivity (especially for iPhone users); uncertainty of when to utilize VisualDx in the clinical workflow; lack of technical savvy and the perception that VisualDx is only for dermatologic concerns.

At an individual level, the TAM is based on the assumption that when users perceive that a type of technology is useful and easy to use, they will be willing to use it. However, that alone may not adequately address the issues raised. Beyond end users’ individual acceptance, there are models that also recognise the role of organizational readiness for enhancing successful acceptance, and consequently, successful adoption of the VisualDx platform. Organizational issues such as leadership buy-in, change management strategies, and alignment of eHealth initiatives to organizational eHealth mandate/vision have been previously identified as important drivers for technology adoption [45–48]. Moreover, 7 leadership behaviors associated with successful outcomes in Health Information Technology adoption include (1) communicating clearly about visions and goals, (2) providing support, (3) establishing a governance structure, (4) establishing training, (5) identifying and appointing champions, (6) addressing work process change, and (7) following up [49]. A holistic strategy for rollout and adoption of mHealth tools such as VisualDx must account for both end user acceptance as well as these organizational and governance-related factors.

Further research in the following areas may build upon the findings of this study to further inform mHealth adoption strategies in Botswana and other contexts:

Similar TAM studies with healthcare providers at different levels and in different contexts to identify where VisualDx or other mHealth tools are more or less likely to be accepted by end users.
Studying the impact of VisualDx or other mHealth tools on patient outcomes. Demonstrating a positive impact on patient outcomes could help to build leadership buy-in.
Assessing the quality of the VisualDx application using the Mobile Application Rating Scale (MARS) [50] to better understand the strengths and areas to improve.

VisualDx was a well-received MCDSS tool amongst the study population and has the potential to upskill and empower general practitioners to do more at the point of care. Through widespread use of VisualDx it could be reasonably hypothesized that benefits such as improved patient outcomes, reduced stress on the medical system through reduced need for referrals, and improved continuing medical education could be realized. Barriers to use of VisualDx were identified as potential areas of improvement for this and other mHealth tools targeting healthcare workers in Botswana and other countries with similarly developing healthcare systems. To ensure successful widespread adoption of an mHealth tool, end user acceptance must be paired with organizational readiness to fully embed the solution into the existing context.

DHMT

District Health Management Team

eHealth

Electronic health

GDPR

General Data Protection Regulation

HCW

Healthcare worker

HIPAA

Health Insurance Portability and Accountability Act

ICT

Information communication technology

(M)CDSS

(Mobile) Clinical Decision Support Systems, a subset of CET.

mHealth

Mobile health

MOHW

Ministry of Health & Wellness

SSA

Sub-Saharan Africa

TAM

Technology Acceptance Model

University of Botswana

Ethics approval and consent to participate

The study was approved by the Ministry of Health and Wellness in Botswana and the University of Botswana Institutional Review Board. All study participants gave informed consent electronically. All methods were performed in accordance with the relevant guidelines and regulations.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed during the study are available from the corresponding author on reasonable request.

Competing interests

Nate Stein is an employee of VisualDx. Mike Annechino is a former employee of VisualDx. Kagiso Ndlovu, Mosadikhumo Monkge, Victoria Williams and Ruth Gaopelo are implementation partners and strategic advisors for VisualDx in Botswana. Mmoloki Molwantwa and Amy Forrestel have no competing interest to declare.

Funding

This work was supported, in whole or in part, by the Bill & Melinda Gates Foundation grant number INV-003773. Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission.

Authors’ contributions

All authors jointly conceived the study, and jointly contributed to its design, and development of the survey tools and interview guide. NS and RG provided technical support and coordinated day-to-day implementation activities. NS, MA, AF, M Monkge, and VW provided app training to the study participants. NS, KN, and RG performed surveys and interviews, completed initial data analysis and wrote the first draft of the manuscript. VW and M Molwantwa provided substantial editorial and intellectual input and contributed to subsequent revisions. All authors approved the final manuscript.

Acknowledgements

A great appreciation to the Bill & Melinda Gates Foundation for funding the pilot implementation in Botswana, the Ministry of Health & Wellness (MOHW Botswana), and the Gaborone District Health Management Team (DHMT), as well as our supportive pilot study participants for making the study successful.

Special thanks to the public and private facilities throughout Botswana for allowing us to conduct this pilot study at their sites.

We would also like to thank VisualDx, for securing the grant that funded this work and for supporting ongoing activities.

Musen MA, Middleton B, Greenes RA. Clinical decision-support systems. InBiomedical informatics 2021 (pp. 795–840). Springer, Cham.
Yu G, Chen Z, Wu J, Tan Y. Medical decision support system for cancer treatment in precision medicine in developing countries. Expert Systems with Applications. 2021 Dec 30;186:115725.
World Health Organization. Digital health [Internet]. Geneva, Switzerland; 2020. https://www.who.int/health-topics/digital-health#tab=tab_1. Accessed 11 April 2022.
Ministry of Health and Wellness Botswana. The eHealth Strategy of Botswana (2020–2024), “A healthy and productive nation that enjoys the benefits of high quality eHealth services.” Gaborone, Botswana; 2020. https://ehealth.ub.bw/bhdc/ehealthstrategy.html.
Botswana: Millennium Development Goals Status Report 2015 Sustaining progress to 2015 and beyond. http://www.bw.undp.org/content/dam/botswana/docs/Publications/Botswana%20NMDG%20Report.pdf. Accessed 9 April 2022.
Nkomazana O, Mash R, Shaibu S, Phaladze N. Stakeholders’ perceptions on shortage of healthcare workers in primary healthcare in Botswana: focus group discussions. PloS one. 2015 Aug 18;10(8):e0135846.
Ncube B, Mars M, Scott RE. The need for a telemedicine strategy for Botswana? A scoping review and situational assessment. BMC Health Services Research. 2020 Dec;20(1):1–8.
Mosojane KI, Giovanni D, Forrestel AK, Conic RZ, Kovarik C, Williams VL. Patterns of Skin Disease in the Context of a High Prevalence HIV Population in Botswana. Dermatologic Clinics. 2021 Jan 1;39(1):1–4.
Williams VL, Narasimhamurthy M, Rodriguez O, Mosojane K, Bale T, Kesalopa K, Kayembe MA, Grover S. Dermatology-Driven quality improvement interventions to decrease diagnostic delays for Kaposi sarcoma in Botswana. Journal of Global Oncology. 2019 Nov;5:1–7.
Anshelevich EE, Mosojane KI, Kenosi L, Nkomazana O, Williams VL. Factors Affecting Quality of Life for People Living with Albinism in Botswana. Dermatol Clin. 2021 Jan;39(1):129–145. https://doi.org/10.1016/j.det.2020.08.012. Epub 2020 Oct 31. PMID: 33228856.
Mosam A, Todd G. Dermatology Training in Africa: Successes and Challenges. Dermatol Clin. 2021 Jan;39(1):57–71. https://doi.org/10.1016/j.det.2020.08.006. PMID: 33228862.
Breitbart EW, Choudhury K, Andersen AD, Bunde H, Breitbart M, Sideri AM, et. al. Improved patient satisfaction and diagnostic accuracy in skin diseases with a Visual Clinical Decision Support System—A feasibility study with general practitioners. PloS one. 2020 Jul 29;15(7):e0235410.
Dulmage B, Tegtmeyer K, Zhang MZ, Colavincenzo M, Xu S. A Point-of-Care, Real Time Artificial Intelligence System to Support Clinician Diagnosis of a Wide Range of Skin Diseases. J Invest Dermatol. 2021 May 1;141(5):1230-5. https://doi.org/10.1016/j.jid.2020.08.027
VisualDx: Equity in Knowledge = Equity in Care. https://www.visualdx.com/. Accessed 14 April 2022
Fulgencio H. E-Health for developing countries: a theoretical model grounded on literature. InE-Systems for the 21st Century: Concept, Developments, and Applications, Volume 1: E-Commerce, E-Decision, E-Government, E-Health, and Social Networks 2019 Jul 10 (pp. 325–348). Apple Academic Press.
Mauco, K.L., Scott, R.E. & Mars, M. Validation of an e-health readiness assessment framework for developing countries. BMC Health Serv Res 20, 575 (2020). https://doi.org/10.1186/s12913-020-05448-3
Taherdoost H. A review of technology acceptance and adoption models and theories. Procedia manufacturing. 2018 Jan 1;22:960–7.
Kamal SA, Shafiq M, Kakria P. Investigating acceptance of telemedicine services through an extended technology acceptance model (TAM). Technology in Society. 2020 Feb 1;60:101212.
Ndlovu K, Mauco KL, Keetile M, Kadimo K, Senyatso RY, Ntebela D, Valela B, Murambi C. Acceptance of the District Health Information System Version 2 Platform for Malaria Case-Based Surveillance By Health Care Workers in Botswana: Web-Based Survey. JMIR Formative Research. 2022 Mar 15;6(3):e32722.
Rahimi B, Nadri H, Afshar HL, Timpka T. A systematic review of the technology acceptance model in health informatics. Applied clinical informatics. 2018 Jul;9(03):604–34.
Silva P. Davis' technology acceptance model (TAM)(1989). Information seeking behavior and technology adoption: Theories and trends. 2015:205 – 19.
Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly. 1989 Sep 1:319–40.
Morse JM. Mixed method design: Principles and procedures. Routledge; 2016 Jul 8.
Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, McLeod L, Delacqua G, Delacqua F, Kirby J, Duda SN. The REDCap consortium: Building an international community of software platform partners. Journal of biomedical informatics. 2019 Jul 1;95:103208. https://doi.org/10.1016/j.jbi.2019.103208.
Terry G, Hayfield N, Clarke V, Braun V. Thematic analysis. The SAGE handbook of qualitative research in psychology. 2017 Jun 30;2:17–37.
Linneberg MS, Korsgaard S. Coding qualitative data: A synthesis guiding the novice. Qualitative research journal. 2019 Jul 24. https://doi.org/10.1108/qrj-12-2018-0012
Cajita MI, Hodgson NA, Lam KW, Yoo S, Han HR. Facilitators of and barriers to mHealth adoption in older adults with heart failure. Computers, informatics, nursing: CIN. 2018 Aug;36(8):376.
Mauco KL, Scott RE, Mars M. Validation of an e-health readiness assessment framework for developing countries. BMC health services research. 2020 Dec;20(1):1–0.
Khoja S, Scott RE, Casebeer AL, Mohsin M, Ishaq AF, Gilani S. e-Health readiness assessment tools for healthcare institutions in developing countries. Telemedicine and e-Health. 2007 Aug 1;13(4):425 – 32.
World Health Organization. Atlas of African Health Statistics 2012: health situation analysis of the African Region. Geneva: WHO. https://www.afro.who.int/publications/atlas-african-health-statistics-2012-health-situation-analysis-african-region. Accessed 10 March 2022.
Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Academic Medicine. 2005 Feb 1;80(2):176 – 82.
Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians' clinical questions: obstacles and potential solutions. Journal of the American Medical Informatics Association. 2005 Mar 1;12(2):217 – 24.
Zakerabasali S, Ayyoubzadeh SM, Baniasadi T, Yazdani A, Abhari S. Mobile health technology and healthcare providers: systemic barriers to adoption. Healthcare Informatics Research. 2021 Oct 31;27(4):267–78.
Brown IT. Individual and technological factors affecting perceived ease of use of web-based learning technologies in a developing country. The Electronic Journal of Information Systems in Developing Countries. 2002 Jul;9(1):1–5.
El-Sappagh S, Ali F, Hendawi A, Jang J, Kwak K. A mobile health monitoring-and-treatment system based on integration of the SSN sensor ontology and the HL7 FHIR standard. BMC Med Inform Decis Mak 2019 May 10;19(1):97 – 36. https://doi.org/10.1186/s12911-019-0806-z.
Rubio Ó, Trigo JD, Alesanco Á, Serrano L, García J. Analysis of ISO/IEEE 11073 built-in security and its potential IHE-based extensibility. J Biomed Inform 2016 Apr;60:270–285. https://doi.org/10.1016/j.jbi.2016.02.006.
Ndlovu K, Mars M, Scott RE. Interoperability frameworks linking mHealth applications to electronic record systems. BMC Health Serv Res 2021 May 13;21(1):459 – 410. https://doi.org/10.1186/s12913-021-06473-6
Memon M, Wagner S, Pedersen C, Beevi F, Hansen F. Ambient assisted living healthcare frameworks, platforms, standards, and quality attributes. Sensors (Basel) 2014 Mar 04;14(3):4312–4341. https://doi.org/10.3390/s140304312.
Cook DA, Sorensen KJ, Hersh W, Berger RA, Wilkinson JM. Features of effective medical knowledge resources to support point of care learning: a focus group study. PLoS One. 2013 Nov 25;8(11):e80318.
De La Cruz Monroy MF, Mosahebi A. The use of smartphone applications (apps) for enhancing communication with surgical patients: a systematic review of the literature. Surgical innovation. 2019 Apr;26(2):244–59.
Contracting with Vendors that are NOT HIPAA Business Associates: Best Practices. [Internet]. 2022. https://www.hipaa.com/. Accessed 31 March 2022.
Complete guide to GDPR compliance. [Internet]. 2022. https://gdpr.eu/. Accessed 29 March 2022.
Kabukye JK, de Keizer N, Cornet R. Assessment of organizational readiness to implement an electronic health record system in a low-resource settings cancer hospital: A cross-sectional survey. PloS one. 2020 Jun 16;15(6):e0234711.
Ndlovu K, Littman-Quinn R, Park E, Dikai Z, Kovarik CL. Scaling up a mobile telemedicine solution in Botswana: keys to sustainability. Frontiers in public health. 2014 Dec 11;2:275.
Warth LL, Dyb K. eHealth initiatives; the relationship between project work and institutional practice. BMC Health Services Research. 2019 Dec;19(1):1–2.
Mlekus L, Bentler D, Paruzel A, Kato-Beiderwieden AL, Maier GW. How to raise technology acceptance: user experience characteristics as technology-inherent determinants. Gruppe. Interaktion. Organisation. Zeitschrift für Angewandte Organisationspsychologie (GIO). 2020 Sep;51(3):273–83.
Aziz F, Md Rami A, Razali F, Mahadi N. The influence of leadership style towards technology acceptance in organization. International Journal of Advanced Science and Technology. 2020;29(7):218–25.
Canada Health Infoway. A framework and toolkit for managing ehealth change: people and processes. Registered Nurses' Association of Ontario. 2013. [Internet]. https://bpgordersettoolkit.rnao.ca/sites/default/files/CHI_ChangeMgmtGuide_ENG.pdf Accessed 25 March 2022.
Ingebrigtsen T, Georgiou A, Clay-Williams R, Magrabi F, Hordern A, Prgomet M, Li J, Westbrook J, Braithwaite J. The impact of clinical leadership on health information technology adoption: systematic review. International journal of medical informatics. 2014 Jun 1;83(6):393–405.
Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth. 2015 Mar 11;3(1):e3422.

Competing interest reported. Nate Stein is an employee of VisualDx. Mike Annechino is a former employee of VisualDx. Kagiso Ndlovu, Mosadikhumo Monkge, Victoria Williams and Ruth Gaopelo are implementation partners and strategic advisors for VisualDx in Botswana. Mmoloki Molwantwa and Amy Forrestel have no competing interest to declare.

AppendicesVisualDxmanuscript.docx

PDF

Reviews received at journal
08 May, 2022
Reviewers agreed at journal
02 May, 2022
Reviewers invited by journal
01 May, 2022
Editor assigned by journal
01 May, 2022
Editor invited by journal
15 Apr, 2022
Submission checks completed at journal
15 Apr, 2022
First submitted to journal
14 Apr, 2022

You are reading this latest preprint version

Evaluating Feasibility and Acceptance of a Mobile Clinical Decision Support System in Botswana

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Data analysis

Results

Surveys

Pre-pilot Survey Responses

Discussion

Study Limitations

Future Direction

Conclusion

Abbreviations

Declarations

References

Competing Interests

Supplementary Files

Status:

Version 1