This study protocol is developed and reported in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guidelines(11) and submitted with the International Registration of Systematic Reviews (PROSPERO) to be made available to the public for comments.
We proposed to conduct a systematic review and meta-analysis to gather available evidence that fits pre-specified eligibility criteria on cost and cost-effectiveness of e-learning interventions for healthcare workers in low- and middle-income countries will be undertaken in line with the Cochrane Handbook for Systematic Reviews of Interventions guideline(12). The distinct steps consist of: Criteria for selecting studies for this review, types of studies, types of participants, types of interventions, types of outcome measures, search methods for identification of studies, Data collection and analysis as stipulated in the Cochrane Handbook for Systematic Reviews of Interventions will be carefully followed. Details of these steps will be discussed in the following subsections.
The PRISMA 2020 Flow Diagram (11)[Fig 1] will be used to show the flow of citations being reviewed, and the results of title searches from various databases consulted. The certainty of the evidence will be assessed by using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) guidelines (13). Lastly, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist will be followed for standardised reporting of this proposed systematic review and meta-analysis results.
Identifying the relevant research question.
The proposed systematic review study will address the following research questions under investigation:
-
What are the available eLearning strategies for Healthcare worker training in low middle-income countries?
-
What is the cost of eLearning in Healthcare worker training in low middle-income countries?
-
What is the cost- effectiveness of eLearning in Healthcare worker training in low middle-income countries?
To determine the above-mentioned relevant research questions eligibility for a systematic review project, we applied the PEO (Population, Exposure, and Outcomes) nomenclature for systematic reviews as recommended by the Joanna Briggs Institute, and illustrated in table1.
Table 1 PEO framework for determining the eligibility of the relevant research questions
Criteria
|
Determinant
|
Population (P)
|
Healthcare workers defined as workers involved in the provision of healthcare services to a user, for example; doctors or physicians, nurses, dentists, pharmacists, optometrists,optometrists, gynachologist and medical specialists.
|
Exposure (E)
|
eLearning as a teaching and learning approach that relies on electronic media and devices as tools to improve access to training and communication, for example e-learning, online learning, distance learning, web-base learning, blended learning, virtual learning, remote learning, and video-based learning.
|
Outcomes (O)
|
Cost, cost-effectiveness
|
Eligibility criteria for considering studies for inclusion in this review.
The development of the eligibility criteria will be informed by the relevant elements of the PEOd-T (Time) framework guidance for conducting a systematic review (12), as shown in Table2. This will be done to ensure that the proposed systematic review relevant research questions boundaries are well defined. We aim to pool all studies that contain cost, cost-effectiveness, cost-benefit and cost-utility of eLearning for healthcare workers. Two independent authors will screen abstract, full text and extract data from eligible studies. Further, we intend to do meta-analysis by statistically pooling appropriate studies with comparable eLearning intervention, study design setting and outcomes, through determining the heterogeneity for included studies.
Table 2 Eligibility criteria according to the PEOd-T nomenclature criteria
Criteria
|
Inclusion
|
Exclusion
|
Population (P)
|
Studies presenting evidence on healthcare workers in low middle-income countries
|
Studies presenting evidence done in high- income countries
|
Exposure (E)
|
Studies presenting evidence in eLearning including various forms of training, using electronic media and devices as tools to improve access to training and communication, for example e-learning, online learning, distance learning, web-base learning, blended learning, virtual learning, remote learning, and video-based learning for essential healthcare worker development, college -level training, and various other training required for medical healthcare workers.
|
Studies presenting evidence that does not include eLearning for healthcare worker training
|
Outcome (O)
|
Studies presenting evidence on costs and cost-effectiveness eLearning for healthcare workers
|
Studies showing evidence of descriptive statistics but lacking detailed cost, cost-effectiveness, measures concerning the outcomes of interest.
|
Design (Study Design)
|
Studies presenting evidence on using different study designs, such as economic evaluations, qualitative research, quantitative descriptive studies, randomised controlled trials, non-randomised studies and mixed methods studies.
|
Studies presenting evidence in the form of government reports, single case reports, strategic documents, and strategic documents
|
Time frame (T)
|
No year of publication restrictions
|
No year of publication restrictions
|
Information sources and search strategy for the identification of relevant studies.
We liaised with the public health university of KwaZulu-Natal librarian expert to develop a robust search strategy. The search strategy will include systematic literature searches conducted using EBSCOhost platform ( Academic Search Complete, ERIC, health source: nursing/academic edition, MEDLINE with Full Text, OpenDissertations), Google Scholar, and the following databases: Web of Science, PubMed and ProQuest databases, in search for studies presenting evidence on the cost and cost-effectiveness of eLearning interventions for healthcare worker training.
The literature search strategy
The PEOd-T (Time) guided search strategy terms development for the proposed systematic review study will be used. A search strategy was developed using four key domains: cost, cost-effectiveness, eLearning, and health workers. A draft search strategy was established and piloted as shown in table 3 to ensure that the search strategy was effective. The final search strategies will be reported in the complete review. Keywords will be combined into a phrase including Boolean (AND, OR) terms, as in the sample demonstrated and will be documented in the analysis and write up. Healthcare workers OR medical students OR health professionals AND eLearning OR Blended learning OR Online learning OR web-based learning AND Costs AND Cost-Effectiveness.
Table 3 Search strategy pilot search result from PubMed database
Date of Search
|
The search engine used (Database)
|
Keyword search used
|
No. of articles retrieved
|
19 July 2021
|
PubMed
|
Search: ((((((cost) AND (cost-effectiveness)) AND (eLearning)) OR (digital learning)) OR (online learning)) OR (blended learning)) AND (health workers)
|
3530
|
Electronic search
In this study relevant electronic data sources: EBSCOhost platform ( Academic Search Complete, ERIC, health source: nursing/academic edition, MEDLINE with Full Text, OpenDissertations), Google Scholar, and the following databases: Web of Science; PubMed and ProQuest databases, will be thoroughly searched using a search strategy evaluating the cost and cost-effectiveness of e-learning interventions for healthcare workers in low- and middle-income countries, reference-checking and searching relevant papers. To ensure the search is thorough, a librarian expert will also be invited for contributions towards retrieving appropriate literature for this particular study.
Searching other resources – including hand searches
The principal investigator will further hand (manually) search additional studies, for example, google scholar and so forth. This would allow for additional pertinent studies that could not have been retrieved using the search strategy to also be included based on their eligibility for this study. Data selection will be assessed and carried out by two independent researchers using the key data variables in order to extract the most specific literature for the review guided by the PRISMA statement.
Data management
Studies and articles searched and retrieved will be managed using the EndNote reference manager. Thereafter, all possible duplicates will be removed, and all articles records that will be used through screening and data collection will be kept in the designated EndNote library.
Study selection and eligibility
The principal investigator (one screener) will do title screening and select abstracts eligible for screening. Two trained independent screeners will screen abstract and full article screening from the included studies retrieved from title screening. Rayyan – Intelligent Systematic Review Application will be used as a screening tool. Screening will be will be settled based on PEO nomenclature. Disputes between reviewers for abstract will be settled through a discussion until consensus is reached. Should reviewers (screeners) have discrepancies at the full article screening stage, a third screener will be invited to resolve the discrepancies. Studies eligible for inclusion at full article screening will then be noted for use in data extraction. Skilled language interpreters will be consulted should there be a need to translate some non-English studies. As shown in PRISMA flowchart Figure 1 through the different phases of a systematic review, the flow of information guide will be followed.
Data extraction
Data extraction from included studies will be conducted by two independent reviewers using piloted standardised data extraction tools such as excel. For all studies eligible up to this stage, the data extraction sheet will contain the following: bibliographic information; aim and objectives; country; participant characteristics; intervention; and different study outcomes as shown in table 4. Both reviewer's final data extracted will be compared, any discrepancies will be solved over discussion until consensus is reached.
Table 4 Data Extraction Content
Author & year of publication
|
Title of study
|
Aim of study
|
Type of publication (Journal/ Review/ `Other)
|
Study design
|
Study population
|
Sample size
|
Evidence of type(s) of eLearning intervention investigated
|
Evidence concerning costs of eLearning intervention investigated
|
Evidence concerning the cost-effectiveness of eLearning intervention investigated
|
Main findings
|
Recommendations
|
Other comments
|
Dealing with missing data
-
The main author will be contacted requesting missing data
-
Assumptions to cope with unavailable data will be disclosed.
-
The possible effect of unavailable data will be addressed in the results section and discussed later.
-
The sensitivity of results to assumptions will be analysed.
Assessment of study quality and Risk of bias
To assess of risk of bias in included studies the authors will assess and judge the quality of the selected papers using the Cochrane risk of bias tool(14). Allocation of sequence generalisation, blinding of data extractors, blinding of outcome evaluation, comprehensiveness of outcome data and discriminatory reporting will be used to assess the risk of bias.
Quality of strength of evidence
The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach will be used to assess the quality of evidence across outcomes(13). The overall quality of evidence (or certainty in the findings) for each outcome collected will be assessed using the five GRADE criteria (trial bias, consistency of effect, imprecision, indirectness, and publication bias). Across trials, the quality rating will be assigned to one of four levels: high, moderate, low, or very low(13). Randomised controlled trials (RCTs) are classified as high quality, but they can be downgraded based on the above criteria.
Data Synthesis
We aim to pool the findings of studies together for a meta-analysis, by determining heterogeneity in the content of included studies. Data of included studies will be analysed using Meta- Easy Excel software tools. Additionally, the generic inverse method will be used for meta-analysis. We will use OR and 95% CIs as measures of effect for dichotomous outcomes(15). As for continuous outcomes, we will use standardised mean differences and 95% CIs. The data reported in medians will be converted to means if possible(15). We expect to find at least two studies in this study that will produce a single estimate of the desired outcomes. If we are unable to find at least two studies with a single estimate of the cost and cost-effectiveness desired outcomes for this study, meta-analysis will be ruled out, and instead, subgroup analysis and sensitivity analysis will be employed.
Assessment of heterogeneity
Forest plots for overlapping CIs will be used to assess the difference in results due to differences in population, selection, study design and intervention delivery known as heterogeneity. To indicate sensible levels of heterogeneity, a χ2 test for heterogeneity with a 10% level of statistical significance and an I2 statistic value greater than 40% will be used(12).
Subgroup analysis and investigation of heterogeneity
Should heterogeneity be found, and a minimum of two studies to produce a single outcome are not found, subgroup analysis will be done. Statistical heterogeneity test with χ2 will be supported. We will conduct a subgroup to examine the level of heterogeneity of all included studies. Low r value will prove heterogeneity of intervention.
Sensitivity analysis
Sensitivity analyses will be carried out to assess the meta-analysis result’s stability and confirm the robustness of the study findings(15). Should the value of results be unsatisfactory or unclear, researchers will repeat analysis to make sure that results from the systematic review are robust from decisions made in the procedure to attaining them. Comprehensive data will be tabulated to clarify method quality of each study together with facts concerning the intervention. Reviewers will then use the data in the evidence tables by grouping, coding and organising data into organised themes to address research aims and objectives. Should there be necessary data missing, relevant authors will be contacted to provide the information required. A draft report will be made. Key findings will be reviewed by stakeholders and study research will be disseminated using different fitting resources.
Ethics and dissemination
This systematic review does not require ethical approval because this study will make use of data extracted from peer-reviewed published studies and will not include the participation of humans and animals. However, requested ethical exemption from the University of KwaZulu-Natal. Findings will be disseminated by the publication of the results manuscript and policy brief in an appropriate journal and shared with the relevant stakeholders through conference presentations, discussions and seminars.