See the published version of this article at BMC Health Services Research.
This is a preprint. Preprints are preliminary reports that have not undergone peer review. They should not be considered conclusive, used to inform clinical practice, or referenced by the media as validated information.
Learn more about In Review
Research article
Economic Impact of Clinical Decision Support Interventions Based on Electronic Health Records
Daniel Lewkowicz
Hasso Plattner Institute
Corresponding Author
ORCiD: https://orcid.org/0000-0002-3167-8741
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published version at BMC Health Services Research.
Background
Unnecessary healthcare utilization, non-adherence to current clinical guidelines or insufficient personalized care are perpetual challenges and remain potential major cost-drivers for healthcare systems around the world. Implementing decision support systems into clinical care is promised to increase quality of care and thereby yield substantial effects onreducing healthcare expenditure. Inthis articleweevaluate the economic impact of clinical decision support (CDS) interventions which are based on electronic health records (EHR).
Methods
We searched for studies published after 2014 using MEDLINE, CENTRAL, WEB OF SCIENCE, EBSCO, and TUFTS CEA registrydatabases that encompass an economic evaluation or consider cost outcome measures of EHR based CDS interventions. Thereupon, we identified best practiceapplication areas and categorized the investigated interventions according to an existing taxonomy of front-endCDS tools.
Results and discussion
Twenty-sevenstudies are investigated in this review. Of those, twenty-twostudies indicate a reduction of healthcare expenditure after implementingan EHRbased CDS system, especiallytowardsprevalent application areas, such as unnecessarylaboratory testing, duplicate order entry, efficient transfusion practice or reduction of antibiotic prescriptions.On the contrary, order facilitators and undiscovered malfunctions revealed to be threats and could lead to new cost drivers in healthcare. While high upfront and maintenance cost of CDS systems are a worldwide implementation barrier, most studies do not consider implementation cost. Finally, four included economic evaluation studies report mixed monetary outcome results and thus highlight the importance for further high quality economic evaluations for these CDS systems.
Conclusion
Current research studies lack to consider comparative cost-outcome metrics as well as detailed cost-components in their analyses. Nonetheless, thepositive economic impact of EHR based CDS interventions especially with regard to reducing waste in healthcare is highly promising.
Keywords
economic evaluation, electronic health record, clinical decision support, behavioral economics
Figure 1
As stated in the 2017 OECD health report, the annual average growth rate in per capita health expenditure continued to increase 1,7% in Germany and 2,1% in the US in real terms since 2009.[1] Accordingly, healthcare expenditure per capita was estimated to be $5,551 in Germany, but was yet outspent by the United States with almost 80% higher spending per person.[1] The latest OECD Health Statistics 2019 report reconfirms these numbers on rising healthcare expenditure and yet reveals an increase of spending per person to $5,986 in Germany and $10,586 in the US, which is equal to 11.2% and 16.9% of total GDP, respectively.[2]
Unnecessary healthcare utilization, non-adherence to current clinical guidelines or insufficient personalized care are perpetual challenges and remain potential major cost-drivers for healthcare systems around the world.[3, 4] For instance, a recent review estimated the annual cost of waste in the US healthcare system to range between $760 billion and $935 billion, which accounts for 25% of total healthcare spending.[3] Furthermore, Shrank et al.[3] approximated that $191 billion to $282 billion could be saved annually with the use of systematic interventions that address the reduction of waste in healthcare.
The benefits of electronic health records (EHR) culminate in the integration of computerized provider order entry (CPOE) systems and real-time, point of care clinical decision support (CDS) interventions. Introducing decision support systems into clinical care is promised to increase quality of care and thereby yield substantial effects on reducing healthcare expenditure.[5] In addition, the rising field of behavioral economics explores how different interventions, such as nudges or best-practice-alerts (BPA), influence and improve clinical decision making through various applicable concepts.[6, 7]
The goal of this study is to understand the economic impact of EHR based CDS interventions and to identify a coherent research best practice approach for these clinical interventions from a cost outcome perspective. Finally, we seek to examine application areas for different medical risk factors that have meanwhile been explored to be cost-saving or cost-effective.
Search strategy
We conducted a systematic literature review to identify the current research progress regarding the economic impact and benefits of clinical decision support interventions which are based on EHR. Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement,[8] we searched English-language literature indexed in the following databases (1) PubMed, (2) Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, (3) Web of Science, (4) EBSCO Business Source Complete, and (5) CEA Registry Tufts Medical Center Library. Additionally, we screened the reference lists of all included studies for eligibility. The final loop of the literature screening process was completed on January 10th 2020.
In order to achieve high sensitivity and precision we developed each search query based on three main pillars: (a) economic outcome (b) electronic health record (c) clinical decision support. These main terms are then further extended with specific terminology and synonyms using Boolean operators to complete the search strategy. Furthermore, we used MeSH terms for the search in PubMed (1) and comparable search terms in databases (2)–(4). For the basic CEA registry search(5) only rudimentary key search terms were used. A detailed summary of the developed search queries is listed in an additional file [see Additional file 2].
Inclusion Criteria
We included all trials in which a monetary economic outcome of an implemented EHR based CDS system is reported. Thus, we considered all analyses of inpatient or ambulatory financial data measures as well as trial-based modelling predictions. With regard to decision-analytic modelling approaches, we also included all kinds of economic perspectives, i.e. societal, health insurances, health systems or user-centered perspective, in order to identify the complete economic dimension of an EHR based CDS intervention. We summarized the detailed inclusion and exclusion criteria of this systematic economic review in Table 1.
| Inclusion criteria | Exclusion criteria | |
|---|---|---|
| Type of decision support intervention | Any real-time and near real-time (point-of-care) computerized clinical decision intervention based on an EHR | - Decision support via e-mail, telephone contact, expert training or workshop, non-computerized education materials, or other behavioral economics interventions, such as accountable justification, i.e., free text entry, or peer comparison via e-mail - Retrospectively generated EHR based CDS alerts, e.g., for retrospective comparison or estimation - Basic CPOE without any decision stewardship - Cost or price display in order to facilitate cost-consciousness - BPA for EHR based patient recruitment for clinical trials - CDS for transitional care to improve post-discharge utilization and discharge management, i.e., process management - CDS usage for resource management, e.g., nurse staffing - EHR based CDS usage support through pay4performance incentives |
| Economic outcome | Monetary outcome data reported through quantitative cost-calculations or estimated through clinical trial-based modelling techniques | Other economic outcome measures, e.g., length of stay, amount of emergency department visits or primary care consultations |
Pre-search showed that the timeframe until year 2014 is adequately elaborated by prior systematic reviews, and we therefore include only studies published from 2014 to present. During our search process, we found that the number of studies meeting the inclusion criteria increased tremendously in the past years with partially overlapping the present research question.[9–11] The most recent review by Jacob et al.[9] examined the cost and economic benefits of CDS systems restricted to cardiovascular disease prevention, but were unable to conclude whether this intervention was either cost-beneficial or cost-effective. Moja et al.[10] reviewed randomized controlled trials (RCTs) that examined the effectiveness of EHR based CDS systems with regard to mortality, morbidity and economic outcomes, and the authors report that EHR based CDS interventions resulted in only small differences in costs and health service utilization.
Front-end Clinical Decision Support Interventions
Wright et al.[12] developed a taxonomy of front-end CDS interventions available to EHR users which we adopt into this study. Front-end CDS tools, in contrast to back-end system capabilities, are defined by the authors as “the intervention types available to end-users created using specific clinical knowledge bases and application logic”.[12] Their taxonomy consists of fifty-three designed CDS front-end tools, i.e. interventions, that were further categorized into six categories:[12]
We use this taxonomy in order to identify a current best practice approach in terms of exploiting application areas that yield potentials for major cost-savings. For this, we aim to prioritize and weight different EHR based CDS tools based upon their economic benefit, so that policy makers, clinic managers and other healthcare provider, who intent to implement equal health information technology, will gain a better understanding of valuable EHR based CDS interventions and their application areas.
We screened in total 1,309 publications of which twenty-seven studies meet our inclusion criteria for this economic review.[5, 13–38] The process of our literature search as well as the reasons for excluding a number of studies is provided within the PRISMA flow-diagram in Fig. 1. Furthermore, an overview of the characteristics of included studies is listed in Table 2.
Category Number of studies (% of total, rounded)
Country
UnitedStates 24 (89%) [5,13-18,20,22,23,25,26,27-38]
Canada 3 (11%) [19,21,24]
Year published
2019 6 (22%) [13-18]
2018 6 (22%) [5,19-23]
2017 5 (19%) [24-28]
2016 2 (7%) [29,30]
2015 3 (11%) [31,33,34]
2014 5 (19%) [32,35-38]
Study design
Cluster randomized trial 4 (15%) [5,19,26,33]
Cross-sectional 1 (4%) [28]
Retroprospective 9 (33%) [15,17,18,20,21,27,32,36,38]
Quasi-experimental 5 (19%) [14,16,22,24,35]
Comparative 1 (4%) [31]
Observational 1 (4%) [23]
Pre-post-intervention 6 (22%) [13,25,29,30,34,37]
Setting
Inpatient 14 (51%) [14,16,17,20,22-25,27,31,32,34,36,38]
Outpatient 8 (30%) [5,13,15,19,26,30,33,34]
Inpatient & Outpatient 4 (15%) [21,28,29,37]
Emergency department 1 (4%) [18]
Type of economic evaluation
Basic cost calculation 23 (85%) [13-25,27-31,32,34,36-38]
Modelapproach 4 (15%) [5,26,33,35]
Table 2:
Characteristics of included studies (n = 27)
Generally, twenty-two studies (81%) [5, 13–16, 18, 20–25, 28–31, 32, 34–38] out of the included twenty-seven studies report cost savings after implementing an EHR based CDS intervention. Three studies (11%) [17, 26, 33] report a rise in cost expenditure, one study (4%) [19] did not detect significant differences in cost outcomes, and one study (4%) [27] compares the economic outcome of two similar and subsequently explored EHR based CDS interventions. Furthermore, in the majority of included studies the main cost outcome measures were laboratory test cost.[15–17, 20, 21, 25, 28, 29, 31, 32, 38]
Exploitation Of Different Front-end Cds Intervention Categories
According to the taxonomy by Wright et al.[12] we identified twelve (44%) studies [5, 13, 15, 20, 22, 23, 26, 31, 32, 36–38] which explored EHR based CDS interventions based on point-of-care alerts or reminders (category 3). In addition, three interventions (11%) [17, 27, 34] were order facilitators (category 2), two studies (7%) [19, 30] investigated medication dosing support (category 1), while relevant information display as well as expert systems (category 4 and 5) were each reported only once from an economic perspective (4%).[18, 24] In the remaining five studies [14, 16, 28, 33, 35] interventions from two different categories were explored in combination. Finally, we found three studies [21, 25, 29] in which the option to place a certain order or test, e.g. a laboratory test, was removed from the EHR CPOE system or the clinician’s laboratory ordering preference list. That was also partly explored along with the so called “hard-stop” alert, which requires a clicking response from the physician before being able to move forward.[14] These restrictive frond-end CDS intervention types were not yet mentioned in the pre-defined categories by Wright et al.[12]. Thus, for this study we extend their taxonomy by a new category
7. Restriction of choice[39]
The removal of an order option ultimately resulted in less laboratory tests ordered, and therefore in a reduction of healthcare expenditure in all five studies it was implemented [14, 21, 25, 28, 29].
Economic Impact For Prevalent Application Areas
In Table 3, we summarized our findings and created an overview of application areas and cost outcome measures in relation to the applied CDS intervention types. Due to the heterogeneity of included studies with regard to different types of cost outcomes reported and different intervention duration it was not possible to conduct a subgroup analysis considering the economic impact of each CDS-front end category. A detailed evidence synthesis of all included twenty-seven studies as well as a brief description of their intervention types, their application area and the resulting economic impact is provided in an additional file [see Additional file 3].
| Study | Size¹ | Application area | CDS intervention period (in month) | Cost outcome (per year, in US$, if not other stated) ² ³ |
|---|---|---|---|---|
| 1. Medication (dosing) support | ||||
| Tamblyn [19] | Medium | Reduce out-of-pocket costs for patients with uncomplicated hypertension | 60 | No difference⁴ |
| Stenner [30] | Large | ePrescribing tool for therapeutic interchange prescribing | 18 | - $812,956 |
| 2. Order facilitator | ||||
| Bolles [17] | Small | Inappropriate test ordering for specialized HIV laboratory testing | 6 | + $14,000 to + $92,000 |
| Schnaus [27] | Large | The order “complete blood count without differential” unintentionally changed to “complete blood count with differential” | 23 days | + $87,275⁵ |
| Shaha [34] | Small | CDS order sets for managing new-onset stroke patients | 6 | - $460,000 to - $1,130,000 |
| 3. Point of care alerts or reminders | ||||
| Chen D [13] | Large | Reduce unnecessary imaging studies in patients with low back pain | 12 | -$1,872,000 |
| Chin [15] | Large | Decrease routine testing for 25(OH) vitamin D levels | 12 | -$300,000 |
| Gong [5] | Medium | Inappropriate antibiotic prescribing for acute respiratory infection | 18 | -$500,000 per 30 years and 100,000 individuals ⁸ |
| Bejjanki [20] | Large | Reduce 17 frequently used duplicate laboratory tests | 17 | - $51,206 |
| Chen JR [22] | Small | Directing the physician to order penicillin allergy testing for patients receiving aztreonam | 9 | -$673.73 per patient |
| Heekin [23] | Large | Adherence to 18 different Choosing Wisely (CW) alerts | 36 | -$944 per patient |
| Sharifi [26] | Small | Clinical childhood obesity intervention | 12 | + $175mill. in 10 years ⁸ |
| Procop (a) [31] | Medium | Unnecessary duplicate laboratory testing | 12 | - $94,225 (Hard-Stop) - $45,681 (Smart-Alert) |
| Procop (b) [32] | Large | Reduce unnecessary, same day duplicate orders | 24 | - $91,793 |
| Goodnough [36] | Large | Reduce overutilization in blood transfusion procedure | 36 | - $1,620,000 |
| Razavi [37] | Small | Reduce unnecessary waste in transfusion practice and blood use of cardiothoracic surgeons | 12 | - $62,715 |
| Bridges [38] | Small | Reduce unnecessary acute hepatitis profile laboratory tests | 3 | - $13,580 |
| 4. Relevant information display | ||||
| Fertel [18] | Small | Reduce the amount of frequent or high emergency department utilizers | 24 | - $3,306 average monthly per patient care plan |
| 5. Expert systems | ||||
| Nault [24] | Large | Antimicrobial stewardship that facilitates the post-prescription review process | 36 | - CAD $116,666 |
| 6. Workflow support | ||||
| none | - | - | - | - |
| 7. Restriction of choice | ||||
| MacMillan [21] | Large | Reduce unnecessary frequent red blood cell folate tests | 43 | - CAD $78,180 |
| Sadowski [25] | Medium | Reduce admission order sets, which allowed multiple routine tests to be ordered repetitively | 2 | - $152,496 |
| Konger [29] | Large | Define order frequency rules and reduce duplicate tests | 24 | - $157,782 |
| Studies with combined multiple CDS intervention categories | ||||
| 3. Point of care alerts or reminders & 7. Restriction of choice | ||||
| Marcelin [14] | Large | Reduce inappropriate gastrointestinal pathogen panel testing | 15 | -$51,600 |
| Felcher [28] | Medium | Reduce unnecessary Vitamin D testing | 6 | - $2,800,000 |
| 2. Order facilitator & 3. Point of care alerts | ||||
| Goetz [16] | Large | Decrease serum folate laboratory testing | 12 | -$26,719 |
| 2. Order facilitator & 6. Workflow support | ||||
| Michaelidis [33] | Medium | Reduce inappropriate antibiotic prescribing for acute bronchitis | 6 | +$34 ⁷ ⁸ |
| 1. Medication (dosing) support & 3. Point of care alerts or reminders | ||||
| Forrester [35] | Medium | CPOE CDS vs. paper-based prescribing in reducing medication errors and adverse drug events (ADE) | 10 | - $18 mill. per 10.000 Monte Carlo simulations ⁸ |
| ¹ Size is defined as the following: | ||||
| Number of patients or encounters involved | ||||
| 0-999 small size | ||||
| 1,000–10,000 medium size | ||||
| > 10,000 large size | ||||
| If patient count was not reported, we applied this range of criteria to the amount of triggered alerts in total | ||||
| ² Total cost outcome in the EHR based CDS intervention group compared to the control group or pre-implementation phase. | ||||
| ³ All cost outcomes were scaled and thereupon calculated to overall cost outcome per year. Values are rounded to full integer numbers. Because of the predominantly short CDS intervention period time range, a discount factor is not used for calculation. The original reported cost data is mentioned in an additional file [see Additional file 3]. | ||||
| ⁴ No statistically significant differences between control and intervention group regarding out-of-pocket costs per patient: CAD$252.6 (control) and CAD$261.5 (intervention) / patient year. Similar values for currently treated patients as well as mean annual costs of antihypertensive treatment (CAD$370.90 control and CAD$385.70 intervention) | ||||
| ⁵ Comparative study: Expenditure increase resulted from the unintentional change within the EHR based CDS system | ||||
| ⁷ 5-year societal cost per five cases of acute bronchitis | ||||
| ⁸ Cost estimation based on model | ||||
Application areas for cost-savings
Thereupon, we identified four main application areas based on their investigated prevalence that resulted in cost-savings after EHR based CDS implementation. Firstly, two studies report on essentially reducing unnecessary Vitamin D routine testing that led to a decrease of laboratory test cost of $300,000[15] and $1,4mill.[28] per year.Secondly, two studies addressed the economic outcomes of the reduction of waste in transfusion practice and red blood cell usage.[36, 37] Acquisition product cost of red cell units were decreased with the help of EHR based CDS and resulted in cost savings of in total $4,821,000 within three years[36] and about $62,715 within one year[37] after implementation, respectively.
Thirdly, two cost-effectiveness-analyses modeled the cost outcome of reducing antibiotic prescriptions for acute respiratory infection as well as for acute bronchitis.[5, 33] Gong et al.[5] include a full accounting of costs into their Markov model and explore that the implemented CDS intervention, called “suggested alternatives”, yielded more quality adjusted life years (QALYs) at a lower cost of $500,000 per 100.000 individuals over thirty years of implementation. Michaelidis et al.[33] on the other hand report a small increase in costs compared to a printed decision support system, i.e. posters. However, the outcome of the latter mainly results from a cost difference between the direct costs of poster printing and the computer programming cost.
Lastly, five studies[20, 29, 31, 32, 38] report on the potential for cost savings through reducing duplicate orders or laboratory tests by using hard-stops[32] or applying order frequency rules[20] to prevent ordering the same test within a certain timeframe. Reducing laboratory duplicate tests resulting in savings of $3,395 in three months for a small patient size cohort[38] and up to $315,565 within twenty-four month for a large patient size cohort.[29]
Application areas resulting in cost increase
Furthermore, we also identified risk areas, which possibly lead to a further increase in healthcare expenditure. One study found that after implementing a CPOE system with default settings, specialized HIV laboratory test cost increased by $14,000-$96,000 within six months.[17] Another study reports that an unplanned change of a pre-selected default order for ‘complete blood count’ to ‘complete blood count with differential’ lead to an average cost increase of $293.11 per day.[27] Finally, the implementation of order sets as decision facilitators possibly entail negative economic effects. One study found that only after the uncoupling of Vitamin B12 and serum folate joint orders within predefined order sets, laboratory test cost decreased by about $26,719 per year.[16] Similarly, another study removed the option to order daily routine tests from automated admission order sets and found savings of $26,416 after two months.[25]Cost-effectiveness-analyses Models
Table 4 encompasses an overview of studies which conducted a cost-effectiveness-analysis (CEA) of EHR based CDS interventions considering various cost data as well as economic outcome measures, such as the incremental cost-effectiveness ratio (ICER), which depicts the incremental change in costs divided by the incremental change in health outcome or effect.
| Study | Model time horizon (years) | Choice of model | Implementation and maintenance cost | Total budget impact | ICER |
|---|---|---|---|---|---|
| Gong et al. [5] | 30 | Markov model | $1.91 base case for 100,000 individuals [preexisting EHR] | CDS intervention $17.32 mill. Control $17.82 mill. | $99.8 per QALY in base case scenario Not directly reported |
| Sharifi et al. [26] | 10 | Monte Carlo micro-simulation | $23,542 per PCP group [preexisting EHR] | CDS intervention +$239 mill. | $237 per BMI unit reduction |
| Michaelidis et al. [33] | 5 | Decision analytic tree | $18 base case - medical record programming [preexisting EHR] | CDS intervention $2,802* Control (usual care) $2,768* | $51.51 per antibiotic prescription safely avoided Not directly reported |
| Forrester et al. [35] | 5 | Decision analytic tree | $1,773,000 five years CPOE system costs | CDS CPOE system $25 mill. Control (paper system) $43mill. | $110 per ADE averted† |
| * Cumulative 5-year societal cost per five cases of acute bronchitis | |||||
| † Documented only for the explored modelling scenario no. 2: The Everett Clinic achieved no reduction in paper chart pulls throughout the 5-year time horizon, to explore the effect of inefficiency from running a paper and electronic system in parallel | |||||
Cost-effectiveness-analyses aim to reveal the trade-offs in resource-allocation decisions.[40] In this context, it is essential to investigate when and to what extend upfront and maintenance cost for an EHR based CDS system will be amortized by its benefits, which again can be measured either in health outcomes, such as quality adjusted life years (QALYs) saved or in reduction of unnecessary healthcare utilization.
Generally, two studies report an increase in healthcare expenditure from a societal perspective, [26, 33] while the other two report cost savings from a societal perspective as well as the medical group’s perspective.[5, 35] Notably, the measurement of effectiveness was single study-based estimates in all four studies.
Regarding the consideration of upfront implementation cost, Gong et al.[5] include only base case consolidated cost data of $1.91 for a cohort of 100.000 individuals based on expert opinions. Sharifi et al.[26] include intervention start-up cost for EHR modification of $2.7mill. as well as other direct cost, such as professional care provider training. Michaelidis et al.[33] report implementation and maintenance cost data, which is physician education per hour and medical record and CDS programming per patient of $18 in the base case. Lastly, Forrester et al.[35] report CPOE CDS system cost as hardware, software and maintenance cost starting from $373,000 in year one to $92,000 after five years, as well as personnel, $555,000 in year one, and indirect cost as 3% of the total cost. Interestingly, the latter also include the HITECH Meaningful Use incentives in their model in order to simulate the financial incentives by the Centers for Medicare & Medicaid Services in the US.
Lack Of Considering All Cost Components
Despite revealing major potentials for cost-savings, we could not asses the quality of included studies, because of the lack of cost information provided, or predominantly the lack of considering all relevant cost components. According to the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, most of the reported recommendations were not satisfied.[41] All twenty-three non-model studies (85%) only calculate the economic outcome based on financial data reported before and after intervention implementation, which for instance, ultimately results from the computation of price per healthcare resource utilization times the quantity of used healthcare resources or services. Thus, even though it was not intended in those studies, it is necessary to mention that only four of them adhered to sound economic evaluations as recommended by CHEERS.[5, 35]
The challenge of heterogeneity for the CEA is also aggravated by considering different cost outcomes considered. Two studies do not directly report an incremental cost effectiveness ratio (ICER) for a predefined threshold, nor include comparative metrics.[33, 35] Other standardized metrics, such as the return on investment or net present value, were also not examined in the included studies. Only one study reported the net monetary benefit (NMB) of the intervention in relation to a predefined threshold.[5, 42]
Additional Studies Worth Mentioning
Notably, five more studies [43–47] meet most of our inclusion criteria, but were excluded due to various, although little, deviations. Three studies [43–45] report cost-savings after a bundle of information technology was implemented simultaneously, but the economic benefit could not solely be attributed to the EHR based CDS intervention. The fourth publication is a NHS health technology assessment (HTA) report.[46] In this HTA, a RCT was conducted in 79 general practices in the UK in which a multicomponent intervention was installed using electronic health records in order to reduce antibiotic prescribing for respiratory infections. The authors perform a basic cost-analysis on whether the cost of healthcare utilization, that is the number of provider consultations, will increase during the time of the trial under the CDS intervention arm, and if patients more often re-consult the physician when not given a prescription. However, the authors explored no difference in cost outcome between the intervention and control period.
The last study worth mentioning compared retrospectively generated alerts by an advanced machine learning CDS system with alerts triggered through the home-grown EHR based CDS system.[47] The authors calculated the healthcare costs of potentially prevented adverse drug events and medication errors, and found that by using the advanced machine learning CDS system 68,2% of alerts were only fired by that new system resulting in cost savings of $60.67 per alert.[47] After extrapolating these results to an local patient population of 747,985 over five years they estimated savings of $1,294,457.[47]
Evaluating the economic impact of EHR based CDS interventions and its potential to increase value in healthcare remains a great challenge. Even though we found that twenty-two studies report cost savings, most of them do not include developing or maintaining costs. Therefore, we could not draw a sound conjunction between the costs of vendor-purchased or home-grown systems to their economic benefit. Nonetheless, this study reveals several use cases with coherent CDS tools that have proven to be cost-saving, and could therefore be eligible for other healthcare providers, clinic managers and researchers for implementation or further exploration.
With the majority of implemented CDS interventions based on point-of-care alerts the question remains how more algorithm-based expert systems and multiple interventions will have synergy effects on the economic impact. Considering then the amount of alerts and a healthcare provider’s time expense, a process-cost analysis, such as the time-driven activity based costing approach, could be combined with the CEA to achieve a better understanding of the whole cost cycle as well as productivity effects for healthcare entities.[48, 49] Generally, cost outcome measures continue to require comparative metrics, for instance, as used by Mathias et al., the cost per useful alert.[49] In a simple model the authors introduce this measure to analyze how different parameter affect the cost of implementing EHR based CDS alerts for genomic precision medicine.[50] However, for future economic evaluation of EHR based CDS interventions a more specific approach with regard to individual application areas or medical risk factor focus might result in better cost and outcome comparisons in order to draw a meaningful conclusion.[9]
Another economic challenge to consider are CPOE systems with default lists or opt-out options of orderable tests as well as predefined order sets. The increasing rate of unnecessary lab tests might be fostered due to a ‘button clicking syndrome’, which explains the inducement of moving along inattentively.[17] In times, when reducing alert fatigue is gaining momentum, automation of orders through order sets or joint-order options could ultimately lead to a decrease of value.[51, 52] Additionally, one study also reported an increase in cost after an unplanned change of the CDS system had occurred.[27] Thus, malfunctions or unintended errors by newly integrated health information technology, also called ‘e-Iatrogenesis’[53], may also lead to yet another cost-driver and possibly cause unpredictable economic damage.[54, 55]
Transferability For Other Countries
All included studies where based on cost data and trials from the United States or Canada. Consequently, current research progress on the economic potentials of EHR based CDS systems on rising healthcare expenditure in Europe or worldwide cannot be derived. We found recent studies evaluating the cost-effectiveness of a stand-alone CPOE CDS system in the Netherlands,[56] or comparing the effectiveness of an EHR based CDS intervention in the US, UK, Republic of Korea and Belgium,[57] while another RCT explored the effectiveness of an EHR based CDS intervention for patients with atrial fibrillation and high risk of stroke in Sweden.[58] Yet we found no study that evaluated the potential for increasing value in this present highly promising field of health information technology outside of North America.
However, this study reveals promising cost savings for already implemented health information technology. Even though implementation cost was not considered, on a long-term view these results reveal the potential for cost-savings once implementation costs are amortized. Therefore, the sooner broad health information technology systems will be implemented in other countries around the world and evaluated economically, the earlier cost-benefits and return on investments can be realized.
Support From Policy Makers Could Accelerate Economic Benefits
Interestingly, Forrester et al.[35] include monetary incentives as provided by the Meaningful Use Initiative in the US in their CEA. This financial support covered only a small percentage of total implementation cost, as in their developed model incentive-eligible prescribers received $42,000 over five years, but this nevertheless contributed to the investigated cost-effectiveness of an EHR based CDS intervention compared to paper-based prescribing. Therefore, how policy makers worldwide intent to financially support EHR adoption and incentivize usage of embedded CDS systems is a critical factor for the economic success of such systems. High upfront implementation cost is a major burden for healthcare entities, especially for smaller to middle size practices and hospitals.[59]
Finally, achieving a decrease in healthcare expenditure should never influence a patient’s quality of life nor disease treatment in a negative way. Even though eliminating a laboratory order option from a CPOE system led to cost-savings, the value and health outcome of each patient is of highest importance and should be individually assessed. Rather than proving to have effectively reduced laboratory test costs, future economic evaluation of EHR based CDS systems should focus more on the potentials of health benefits that could be achieved, such as through reduced antibiotic prescriptions or adverse drug events. In the end, competing on shifting costs will not change anything about the main goal of decision stewardship, and that is to increase value in healthcare.[60]
There are some limitations to this study. Firstly, since we only considered English language literature in our economic review, we might not have included international publications in other languages that indeed report on the information technology progress made by other countries regarding the linkage of CDS systems to an existing EHR. Another limitation is the exclusion of EHR cost and price display interventions. Physicians’ literacy towards costs in healthcare hold furthers possibilities to increase the economic benefit CDS interventions, although a systematic review recently found that cost display in EHR CPOE systems does not affect the efficiency and effectiveness domain of healthcare quality.[61] We also excluded other non-monetary impact measures, such as length of stay, which essentially also refer to the economic impact of EHR based CDS implementation.
Overall, the results of our findings might be biased, since we included all types of studies as well as all kinds of monetary outcomes reported. Due to the lack of economic evaluations, included studies tended to declare high cost-savings, but did only consider little to none cost components of implementing such a complex information system. In addition, authors might have been tempted to calculated cost-savings only when the intervention proved to be effective. Finally, based on our inclusion criteria we cannot argue to have involved all existing studies that meet the inclusion criteria of this review. While conducting the study, we found that studies not necessarily report calculated economic outcomes in the title or abstract of their publication, and thus by the nature of following the PRISMA guidelines leading to an exclusion of that study.
Clinical decision support interventions based on electronic health records have an overall positive economic impact. Predominantly point-of-care alerts with regard to unnecessary laboratory testing, efficient transfusion practice, or reduction of antibiotic prescription emerged as application areas with already promising potentials for high cost-savings. Nonetheless, lack of cost data consideration as well as the need for comparative metrics continue to be the reason why the economic dimension of EHR based CDS interventions need to be further explored. Therefore, high quality cost-effectiveness or cost-utility analyses, which include more extensive cost data and consider different economic perspectives, are needed in order to draw a sound conclusion. Finally, introducing personalized health services based on peoples’ electronic health records is yet another promising research field with great potentials for further increasing value in healthcare, and should therefore receive more attention in future research.
Adverse drug events; BPA:Best-practice-alerts; CDS:Clinical decision support; CEA:Cost-effectiveness-analysis; CHEERS:Consolidated Health Economic Evaluation Reporting Standards; CPOE:Computerized provider order entry; CW:Choosing Wisely; EHR:Electronic health record; HTA:Health technology assessment; ICER:Incremental cost-effectiveness ratio; NMB:Net monetary benefit; QALY:Quality adjusted life years; RCT:Randomized controlled trial; PRISMA:Preferred Reporting Items for Systematic Reviews and Meta Analyses;
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and materials
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Funding
This work has received funding from the European Union’s Horizon 2020 research and innovation program under the grant agreement No. 826117 Smart4Health - Building a citizen-centered EU-EHR exchange for personalized health. The funding body had no role in the design of the study, collection or data analysis.
Acknowledgements
Not applicable.
Authors' contributions
DL conceptualized the study. DL and AW conducted the literature searches, literature screening as wellas the analyses. DL drafted the manuscript, and all authors contributed to refining all sections and critically editing the paper.
- OECD10.1787/health_glance-2017-enOECD. (2017), Health at a Glance 2017: OECD Indicators, Publishing OECD. Paris. http://dx.doi.org/10.1787/health_glance-2017-en (accessed 1 October 2019).
- OECD. (2019), OECD Health Statistics 2019. https://www.oecd.org/health/health-data.htm (accessed 1 December 2019).
- Shrank WH, Rogstad TL, Parekh N. Waste in the US Health Care System: Estimated Costs and Potential for Savings. JAMA. 2019;322(15):1501–9. doi:10.1001/jama.2019.13978.
- McGinn T. Putting Meaning into Meaningful Use: A Roadmap to Successful Integration of Evidence at the Point of Care. JMIR Med Inform. 2016;4(2):e16. doi:10.2196/medinform.4553.
- Gong CL, Zangwill KM, Hay JW, et al. Behavioral Economics Interventions to Improve Outpatient Antibiotic Prescribing for Acute Respiratory Infections: a Cost-Effectiveness Analysis. J Gen Intern Med. 2019;34(6):846–54. doi:10.1007/s11606-018-4467-x.
- Atkins D. So Many Nudges, So Little Time: Can Cost-effectiveness Tell Us When It Is Worthwhile to Try to Change Provider Behavior? J Gen Intern Med. 2019;34(6):783–4. doi:10.1007/s11606-019-04871-5.
- Cho I, Bates DW. Behavioral Economics Interventions in Clinical Decision Support Systems. Yearb Med Inform. 2018;27(1):114–21. doi:10.1055/s-0038-1641221.
- Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi:10.1371/journal.pmed.1000097.
- Jacob V, Thota AB, Chattopadhyay SK, et al. Cost and economic benefit of clinical decision support systems for cardiovascular disease prevention: a community guide systematic review. J Am Med Inform Assoc. 2017;24(3):669–76. doi:10.1093/jamia/ocw160.
- Moja L, Kwag KH, Lytras T, et al. Effectiveness of computerized decision support systems linked to electronic health records: A systematic review and meta-analysis. Am J Public Health. 2014;104(12):e12–22. https://doi.org/10.2105/AJPH.2014.302164.
- Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: A systematic review. Ann Intern Med. 2012;157(1):29–43. https://doi.org/10.7326/0003-4819-157-1-201207030-00450.
- Wright A, Sittig DF, Ash JS, et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc. 2011;18(3):232–42. doi:10.1136/amiajnl-2011-000113.
- 10.1016/j.wneu.2019.11.03Chen D, Bhambhvani HP, Hom J, et al. Effect of Electronic Clinical Decision Support on Imaging for the Evaluation of Acute Low Back Pain in the Ambulatory Care Setting. World Neurosurgery 2019;pii:S1878-8750(19)32867-0. doi:10.1016/j.wneu.2019.11.03.
- Marcelin J, Brewer C, Beachy M, et al. Hardwiring diagnostic stewardship using electronic ordering restrictions for gastrointestinal pathogen testing. Infection Control Hospital Epidemiology. 2019;40(6):668–73. doi:10.1017/ice.2019.78.
- Chin KK, Hom J, Tan M, et al. Effect of Electronic Clinical Decision Support on 25(OH) Vitamin D Testing. J Gen Intern Med. 2019;34(9):1697–9. https://doi.org/10.1007/s11606-019-05057-9.
- Goetz C, Di Capua J, Lee I, et al. A Student-Led, Multifaceted Intervention to Decrease Unnecessary Folate Ordering in the Inpatient Setting. J Healthc Qual. 2019;41(5):e54–60. doi:10.1097/JHQ.0000000000000177.
- Bolles K, Woc-Colburn L, Hamill RJ, et al. Ordering patterns and costs of specialized laboratory testing by hospitalists and house staff in hospitalized patients with HIV at a county hospital: An opportunity for diagnostic stewardship. Open Forum Infect Dis. 2019;6(6):ofz158. doi:10.1093/ofid/ofz158.
- Fertel BS, Podolsky SR, Mark J, et al. Impact of an individual plan of care for frequent and high utilizers in a large healthcare system. Am J Emerg Med. 2019;37(11):2039–42. doi:10.1016/j.ajem.2019.02.032.
- Tamblyn R, Winslade N, Qian CJ, et al. What is in your wallet? A cluster randomized trial of the effects of showing comparative patient out-of-pocket costs on primary care prescribing for uncomplicated hypertension. Implement Sci. 2018;13(1):7. doi:10.1186/s13012-017-0701-x.
- Bejjanki H, Mramba LK, Beal SG, et al. The role of a best practice alert in the electronic medical record in reducing repetitive lab tests. Clinicoecon Outcomes Res 2018;10:611–618. doi:10.2147/CEOR.S167499.
- MacMillan TE, Gudgeon P, Yip PM, et al. Reduction in unnecessary red blood cell folate testing by restricting computerized physician order entry in the electronic health record. Am J Med. 2018;131(8):939–44. doi:10.1016/j.amjmed.2018.03.044.
- Chen JR, Tarver SA, Alvarez KS, et al. Improving aztreonam stewardship and cost through a penicillin allergy testing clinical guideline. Open Forum Infect Dis. 2018;5(6):ofy106. doi:10.1093/ofid/ofy106.
- Heekin AM, Kontor J, Sax HC, et al. Choosing Wisely clinical decision support adherence and associated inpatient outcomes. Am J Manag Care. 2018;24(8):361–6.
- Nault V, Pepin J, Beaudoin M, et al. Sustained impact of a computer-assisted antimicrobial stewardship intervention on antimicrobial use and length of stay. J Antimicrob Chemother. 2017;72(3):933–40. doi:10.1093/jac/dkw468.
- Sadowski BW, Lane AB, Wood SM, et al. High-value, cost-conscious care: Iterative systems-based interventions to reduce unnecessary laboratory testing. Am J Med. 2017;130(9):1112.e. 1-1112.e7. doi:10.1016/j.amjmed.2017.02.029.
- Sharifi M, Franz C, Horan CM, et al. Cost-effectiveness of a clinical childhood obesity intervention. Pediatrics 2017;140(5). pii: e20162998. doi:10.1542/peds.2016-2998.
- Schnaus MJ, Michalik M, Skarda P. Effects of electronic medical record display on provider ordering behavior: Leveraging the EMR to improve quality and costs. Am J Med. 2017;130(12):1366–71. doi:10.1016/j.amjmed.2017.08.019.
- Felcher AH, Gold R, Mosen DM, et al. Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc. 2017;24(4):776–80. doi:10.1093/jamia/ocw182.
- Konger RL, Ndekwe P, Jones G, et al. Reduction in Unnecessary Clinical Laboratory Testing Through Utilization Management at a US Government Veterans Affairs Hospital. Am J Clin Pathol. 2016;145(3):355–64. doi:10.1093/ajcp/aqv092.
- Stenner SP, Chakravarthy R, Johnson KB, et al. ePrescribing: Reducing Costs through In-Class Therapeutic Interchange. Appl Clin Inform. 2016;7(4):1168–81.
- 10.1309/AJCPJOJ3HKEBD3TUProcop GW, Keating C, Stagno P, et al (a). Reducing duplicate testing: a comparison of two clinical decision support tools. Am J Clin Pathol 2015;143(5):623-6. doi:10.1309/AJCPJOJ3HKEBD3TU.
- 10.1309/AJCPOWHOIZBZ3FRWProcop GW, Yerian LM, Wyllie R, et al (b). Duplicate laboratory test reduction using a clinical decision support tool. Am J Clin Pathol 2014;141(5):718 – 23. doi:10.1309/AJCPOWHOIZBZ3FRW.
- Michaelidis CI, Kern MS, Smith KJ. Cost-Effectiveness of Decision Support Strategies in Acute Bronchitis. J Gen Intern Med. 2015;30(10):1505–10. doi:10.1007/s11606-015-3289-3.
- Shaha SH, Gilbert-Bradley D. Improved Immediate and Continued-Care Outcomes for Stroke through Community-Wide Data. Stud Health Technol Inform. 2015;209:140–6.
- Forrester SH, Hepp Z, Roth JA, et al. Cost-effectiveness of a computerized provider order entry system in improving medication safety ambulatory care. Value Health. 2014;17(4):340–9. doi:10.1016/j.jval.2014.01.009.
- Goodnough LT, Maggio P, Hadhazy E, et al. Restrictive blood transfusion practices are associated with improved patient outcomes. Transfusion. 2014;54:2753–9. doi:10.1111/trf.12723.
- Razavi SA, Carter AB, Puskas JD, et al. Reduced red blood cell transfusion in cardiothoracic surgery after implementation of a novel clinical decision support tool. J Am Coll Surg. 2014;219(5):1028–36. doi:10.1016/j.jamcollsurg.2014.06.012.
- Bridges SA, Papa L, Norris AE, et al. Duplicated laboratory tests: evaluation of a computerized alert intervention abstract. J Healthc Qual. 2014;36(3):46–53. doi:10.1111/j.1945-1474.2012.00219.x.
- Ubel PA, Rosenthal MB. Beyond Nudges - When Improving Health Calls for Greater Assertiveness. N Engl J Med. 2019;380(4):309–11. doi:10.1056/NEJMp1806371.
- Neumann PJ, Sanders GD. Cost-Effectiveness Analysis 2.0. N Engl J Med. 2017;376(3):203–5. doi:10.1056/NEJMp1612619.
- Husereau D, Drummond M, Petrou S, et al. Consolidated health economic evaluation reporting standards (CHEERS)—explanation and elaboration: a report of the ISPOR Health Economic Evaluations Publication Guidelines Good Reporting Practices Task Force. Value Health. 2013;16(2):231–50.
- Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making. 1998;18:68–80.
- Graven M, Allen P, Smith I, et al. Decline in mortality with the Belize Integrated Patient-Centred Country Wide Health Information System (BHIS) with embedded program management. Int J Med Inform. 2013;82(10):954–63. doi:10.1016/j.ijmedinf.2013.06.003.
- Riahi S, Fischler I, Stuckey MI, et al. The Value of electronic medical record implementation in mental health care: A Case Study. JMIR Med Inform. 2017;5(1):e1.
- Olchanski N, Dziadzko MA, Tiong IC, et al. Can a novel ICU data display positively affect patient outcomes and save lives? J Med Syst. 2017;41(11):171. doi:10.1007/s10916-017-0810-8.
- Gulliford MC, Juszczyk D, Prevost AT, et al. Electronically delivered interventions to reduce antibiotic prescribing for respiratory infections in primary care: cluster RCT using electronic health records and cohort study. Health Technol Assess. 2019;23(11):1–70. doi:10.3310/hta23110.
- Rozenblum R, Rodriguez-Monguio R, Volk LA, et al. Using a Machine Learning System to Identify and Prevent Medication Prescribing Errors: A Clinical and Cost Analysis Evaluation. Jt Comm J Qual Patient Saf. 2020;46(1):3–10. doi:10.1016/j.jcjq.2019.09.008.
- Keel G, Savage C, Rafiq M, et al. Time-driven activity-based costing in health care: A systematic review of the literature. Health Policy. 2017;121(7):755–63. doi:10.1016/j.healthpol.2017.04.013.
- Campanale C, Cinquini L, Tenucci A. Time-driven activity-based costing to improve transparency and decision making in healthcare. Management. 2014;11:165–86.
- Mathias PC, Tarczy-Hornoch P, Shirts BH. Modeling the costs of clinical decision support for genomic precision medicine. AMIA Jt Summits Transl Sci Proc 2016;2016:60 – 4.eCollection 2016..
- Hussain MI, Reynolds TL, Zheng K, et al. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc. 2019;26(10):1141–9. doi:10.1093/jamia/ocz095.
- Hakim I, Hathi S, Nair A, et al. Electronic health records and the frequency of diagnostic test orders. Am J Manag Care. 2017;23(1):e16–23.
- Payne TH, Bates DW, Berner ES, et al. Healthcare information technology and economics. J Am Med Inform Assoc. 2013;20(2):212–7.
- Wright A, Ai A, Ash J, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc. 2018;25(5):496–506. doi:10.1093/jamia/ocx106.
- Aaron S, McEvoy DS, Ray S, et al. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc. 2019;26(1):37–43. doi:10.1093/jamia/ocy139.
- Vermeulen KM, van Doormaal JE, Zaal RJ, et al. Cost-effectiveness of an electronic medication ordering system (CPOE/CDSS) in hospitalized patients. Int J Med Inform. 2014;83(8):572–80. doi:10.1016/j.ijmedinf.2014.05.003.
- Cornu P, Phansalkar S, Seger DL, et al. High-priority and low-priority drug-drug interactions in different international electronic health record systems: A comparative study. Int J Med Inform. 2018;111:165–71. doi:10.1016/j.ijmedinf.2017.12.027.
- Karlsson LO, Nilsson S, Bång M, et al. A clinical decision support tool for improving adherence to guidelines on anticoagulant therapy in patients with atrial fibrillation at risk of stroke: A cluster-randomized trial in a Swedish primary care setting (the CDS-AF study). PLoS Med. 2018;15(3):e1002528. doi:10.1371/journal.pmed.1002528.
- Slight SP, Quinn C, Avery AJ, et al. A qualitative study identifying the cost categories associated with electronic health record implementation in the UK. J Am Med Inform Assoc. 2014;21(e2):e226-31. doi:10.1136/amiajnl-2013-002404.
- Porter ME. What is value in health care? N Engl J Med 2010;363(26):2477-81. doi:0.1056/NEJMp1011024.
- Mummadi SR, Mishra R. Effectiveness of provider price display in computerized physician order entry (CPOE) on healthcare quality: a systematic review. J Am Med Inform Assoc. 2018;25(9):1228–39. doi:10.1093/jamia/ocy076.
- Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: A randomized clinical trial. JAMA. 2016;315(6):562–70. doi:10.1001/jama.2016.0275.
- Taveras EM, Marshall R, Kleinman KP, et al. Comparative effectiveness of childhood obesity interventions in pediatric primary care: a cluster-randomized clinical trial. JAMA Pediatr. 2015;169(6):535–42. doi:10.1001/jamapediatrics.2015.0182.
- Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267–73. doi:10.1001/jamainternmed.2013.1589.
- 10.1007/s11606-015-3431-2Black JT. Capsule Commentary on Michaelidis et al., Cost-Effectiveness of decision support strategies in acute bronchitis. J Gen Intern Med 2016;31(3):328. doi:10.1007/s11606-015-3431-2.
- Devine EB, Hansen RN, Wilson-Norton JL, et al. The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc. 2010;17(1):78–84. doi:10.1197/jamia.M3285.
This is a list of supplementary files associated with this preprint. Click to download.
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
Published
On 15 Sep, 2020
No community comments so far
Submission checks complete
On 26 Aug, 2020
Editorial decision: Accept
On 24 Aug, 2020
No comments provided
Editorial decision: Minor revision
On 18 Aug, 2020
Editor assigned
On 16 Aug, 2020
Submission checks complete
On 16 Aug, 2020
Editor invited
On 16 Aug, 2020
No comments provided
Editorial decision: Minor revision
On 07 Aug, 2020
Review #2 received
Received 06 Aug, 2020
Review #1 received
Received 06 Aug, 2020
Reviewer #2 agreed
On 18 Jul, 2020
Reviewer #1 agreed
On 15 Jul, 2020
Editor assigned
On 12 Jul, 2020
Reviewers invited
Invitations sent on 12 Jul, 2020
Submission checks complete
On 12 Jul, 2020
Editor invited
On 12 Jul, 2020
Version 1
Posted 20 May, 2020
No comments provided
Editorial decision: Minor revision
On 25 Jun, 2020
Review #2 received
Received 23 Jun, 2020
Review #1 received
Received 13 Jun, 2020
Reviewer #2 agreed
On 31 May, 2020
Reviewer #1 agreed
On 29 May, 2020
Reviewers invited
Invitations sent on 25 May, 2020
Editor assigned
On 18 May, 2020
Submission checks complete
On 18 May, 2020
Editor invited
On 18 May, 2020
First submitted
On 14 May, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Learn more about our company.
This preprint is under consideration at BMC Health Services Research. Preprints are preliminary reports that have not undergone peer review. They should not be considered conclusive, used to inform clinical practice, or referenced by the media as validated information.
Learn more about In Review
Research article
Economic Impact of Clinical Decision Support Interventions Based on Electronic Health Records
Daniel Lewkowicz
Hasso Plattner Institute
Corresponding Author
ORCiD: https://orcid.org/0000-0002-3167-8741
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
Published
On 15 Sep, 2020
No community comments so far
Submission checks complete
On 26 Aug, 2020
Editorial decision: Accept
On 24 Aug, 2020
No comments provided
Editorial decision: Minor revision
On 18 Aug, 2020
Editor assigned
On 16 Aug, 2020
Submission checks complete
On 16 Aug, 2020
Editor invited
On 16 Aug, 2020
No comments provided
Editorial decision: Minor revision
On 07 Aug, 2020
Review #2 received
Received 06 Aug, 2020
Review #1 received
Received 06 Aug, 2020
Reviewer #2 agreed
On 18 Jul, 2020
Reviewer #1 agreed
On 15 Jul, 2020
Editor assigned
On 12 Jul, 2020
Reviewers invited
Invitations sent on 12 Jul, 2020
Submission checks complete
On 12 Jul, 2020
Editor invited
On 12 Jul, 2020
Version 1
Posted 20 May, 2020
No comments provided
Editorial decision: Minor revision
On 25 Jun, 2020
Review #2 received
Received 23 Jun, 2020
Review #1 received
Received 13 Jun, 2020
Reviewer #2 agreed
On 31 May, 2020
Reviewer #1 agreed
On 29 May, 2020
Reviewers invited
Invitations sent on 25 May, 2020
Editor assigned
On 18 May, 2020
Submission checks complete
On 18 May, 2020
Editor invited
On 18 May, 2020
First submitted
On 14 May, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published version at BMC Health Services Research.
Background
Unnecessary healthcare utilization, non-adherence to current clinical guidelines or insufficient personalized care are perpetual challenges and remain potential major cost-drivers for healthcare systems around the world. Implementing decision support systems into clinical care is promised to increase quality of care and thereby yield substantial effects onreducing healthcare expenditure. Inthis articleweevaluate the economic impact of clinical decision support (CDS) interventions which are based on electronic health records (EHR).
Methods
We searched for studies published after 2014 using MEDLINE, CENTRAL, WEB OF SCIENCE, EBSCO, and TUFTS CEA registrydatabases that encompass an economic evaluation or consider cost outcome measures of EHR based CDS interventions. Thereupon, we identified best practiceapplication areas and categorized the investigated interventions according to an existing taxonomy of front-endCDS tools.
Results and discussion
Twenty-sevenstudies are investigated in this review. Of those, twenty-twostudies indicate a reduction of healthcare expenditure after implementingan EHRbased CDS system, especiallytowardsprevalent application areas, such as unnecessarylaboratory testing, duplicate order entry, efficient transfusion practice or reduction of antibiotic prescriptions.On the contrary, order facilitators and undiscovered malfunctions revealed to be threats and could lead to new cost drivers in healthcare. While high upfront and maintenance cost of CDS systems are a worldwide implementation barrier, most studies do not consider implementation cost. Finally, four included economic evaluation studies report mixed monetary outcome results and thus highlight the importance for further high quality economic evaluations for these CDS systems.
Conclusion
Current research studies lack to consider comparative cost-outcome metrics as well as detailed cost-components in their analyses. Nonetheless, thepositive economic impact of EHR based CDS interventions especially with regard to reducing waste in healthcare is highly promising.
Figure 1
As stated in the 2017 OECD health report, the annual average growth rate in per capita health expenditure continued to increase 1,7% in Germany and 2,1% in the US in real terms since 2009.[1] Accordingly, healthcare expenditure per capita was estimated to be $5,551 in Germany, but was yet outspent by the United States with almost 80% higher spending per person.[1] The latest OECD Health Statistics 2019 report reconfirms these numbers on rising healthcare expenditure and yet reveals an increase of spending per person to $5,986 in Germany and $10,586 in the US, which is equal to 11.2% and 16.9% of total GDP, respectively.[2]
Unnecessary healthcare utilization, non-adherence to current clinical guidelines or insufficient personalized care are perpetual challenges and remain potential major cost-drivers for healthcare systems around the world.[3, 4] For instance, a recent review estimated the annual cost of waste in the US healthcare system to range between $760 billion and $935 billion, which accounts for 25% of total healthcare spending.[3] Furthermore, Shrank et al.[3] approximated that $191 billion to $282 billion could be saved annually with the use of systematic interventions that address the reduction of waste in healthcare.
The benefits of electronic health records (EHR) culminate in the integration of computerized provider order entry (CPOE) systems and real-time, point of care clinical decision support (CDS) interventions. Introducing decision support systems into clinical care is promised to increase quality of care and thereby yield substantial effects on reducing healthcare expenditure.[5] In addition, the rising field of behavioral economics explores how different interventions, such as nudges or best-practice-alerts (BPA), influence and improve clinical decision making through various applicable concepts.[6, 7]
The goal of this study is to understand the economic impact of EHR based CDS interventions and to identify a coherent research best practice approach for these clinical interventions from a cost outcome perspective. Finally, we seek to examine application areas for different medical risk factors that have meanwhile been explored to be cost-saving or cost-effective.
Search strategy
We conducted a systematic literature review to identify the current research progress regarding the economic impact and benefits of clinical decision support interventions which are based on EHR. Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement,[8] we searched English-language literature indexed in the following databases (1) PubMed, (2) Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, (3) Web of Science, (4) EBSCO Business Source Complete, and (5) CEA Registry Tufts Medical Center Library. Additionally, we screened the reference lists of all included studies for eligibility. The final loop of the literature screening process was completed on January 10th 2020.
In order to achieve high sensitivity and precision we developed each search query based on three main pillars: (a) economic outcome (b) electronic health record (c) clinical decision support. These main terms are then further extended with specific terminology and synonyms using Boolean operators to complete the search strategy. Furthermore, we used MeSH terms for the search in PubMed (1) and comparable search terms in databases (2)–(4). For the basic CEA registry search(5) only rudimentary key search terms were used. A detailed summary of the developed search queries is listed in an additional file [see Additional file 2].
Inclusion Criteria
We included all trials in which a monetary economic outcome of an implemented EHR based CDS system is reported. Thus, we considered all analyses of inpatient or ambulatory financial data measures as well as trial-based modelling predictions. With regard to decision-analytic modelling approaches, we also included all kinds of economic perspectives, i.e. societal, health insurances, health systems or user-centered perspective, in order to identify the complete economic dimension of an EHR based CDS intervention. We summarized the detailed inclusion and exclusion criteria of this systematic economic review in Table 1.
| Inclusion criteria | Exclusion criteria | |
|---|---|---|
| Type of decision support intervention | Any real-time and near real-time (point-of-care) computerized clinical decision intervention based on an EHR | - Decision support via e-mail, telephone contact, expert training or workshop, non-computerized education materials, or other behavioral economics interventions, such as accountable justification, i.e., free text entry, or peer comparison via e-mail - Retrospectively generated EHR based CDS alerts, e.g., for retrospective comparison or estimation - Basic CPOE without any decision stewardship - Cost or price display in order to facilitate cost-consciousness - BPA for EHR based patient recruitment for clinical trials - CDS for transitional care to improve post-discharge utilization and discharge management, i.e., process management - CDS usage for resource management, e.g., nurse staffing - EHR based CDS usage support through pay4performance incentives |
| Economic outcome | Monetary outcome data reported through quantitative cost-calculations or estimated through clinical trial-based modelling techniques | Other economic outcome measures, e.g., length of stay, amount of emergency department visits or primary care consultations |
Pre-search showed that the timeframe until year 2014 is adequately elaborated by prior systematic reviews, and we therefore include only studies published from 2014 to present. During our search process, we found that the number of studies meeting the inclusion criteria increased tremendously in the past years with partially overlapping the present research question.[9–11] The most recent review by Jacob et al.[9] examined the cost and economic benefits of CDS systems restricted to cardiovascular disease prevention, but were unable to conclude whether this intervention was either cost-beneficial or cost-effective. Moja et al.[10] reviewed randomized controlled trials (RCTs) that examined the effectiveness of EHR based CDS systems with regard to mortality, morbidity and economic outcomes, and the authors report that EHR based CDS interventions resulted in only small differences in costs and health service utilization.
Front-end Clinical Decision Support Interventions
Wright et al.[12] developed a taxonomy of front-end CDS interventions available to EHR users which we adopt into this study. Front-end CDS tools, in contrast to back-end system capabilities, are defined by the authors as “the intervention types available to end-users created using specific clinical knowledge bases and application logic”.[12] Their taxonomy consists of fifty-three designed CDS front-end tools, i.e. interventions, that were further categorized into six categories:[12]
We use this taxonomy in order to identify a current best practice approach in terms of exploiting application areas that yield potentials for major cost-savings. For this, we aim to prioritize and weight different EHR based CDS tools based upon their economic benefit, so that policy makers, clinic managers and other healthcare provider, who intent to implement equal health information technology, will gain a better understanding of valuable EHR based CDS interventions and their application areas.
We screened in total 1,309 publications of which twenty-seven studies meet our inclusion criteria for this economic review.[5, 13–38] The process of our literature search as well as the reasons for excluding a number of studies is provided within the PRISMA flow-diagram in Fig. 1. Furthermore, an overview of the characteristics of included studies is listed in Table 2.
Category Number of studies (% of total, rounded)
Country
UnitedStates 24 (89%) [5,13-18,20,22,23,25,26,27-38]
Canada 3 (11%) [19,21,24]
Year published
2019 6 (22%) [13-18]
2018 6 (22%) [5,19-23]
2017 5 (19%) [24-28]
2016 2 (7%) [29,30]
2015 3 (11%) [31,33,34]
2014 5 (19%) [32,35-38]
Study design
Cluster randomized trial 4 (15%) [5,19,26,33]
Cross-sectional 1 (4%) [28]
Retroprospective 9 (33%) [15,17,18,20,21,27,32,36,38]
Quasi-experimental 5 (19%) [14,16,22,24,35]
Comparative 1 (4%) [31]
Observational 1 (4%) [23]
Pre-post-intervention 6 (22%) [13,25,29,30,34,37]
Setting
Inpatient 14 (51%) [14,16,17,20,22-25,27,31,32,34,36,38]
Outpatient 8 (30%) [5,13,15,19,26,30,33,34]
Inpatient & Outpatient 4 (15%) [21,28,29,37]
Emergency department 1 (4%) [18]
Type of economic evaluation
Basic cost calculation 23 (85%) [13-25,27-31,32,34,36-38]
Modelapproach 4 (15%) [5,26,33,35]
Table 2:
Characteristics of included studies (n = 27)
Generally, twenty-two studies (81%) [5, 13–16, 18, 20–25, 28–31, 32, 34–38] out of the included twenty-seven studies report cost savings after implementing an EHR based CDS intervention. Three studies (11%) [17, 26, 33] report a rise in cost expenditure, one study (4%) [19] did not detect significant differences in cost outcomes, and one study (4%) [27] compares the economic outcome of two similar and subsequently explored EHR based CDS interventions. Furthermore, in the majority of included studies the main cost outcome measures were laboratory test cost.[15–17, 20, 21, 25, 28, 29, 31, 32, 38]
Exploitation Of Different Front-end Cds Intervention Categories
According to the taxonomy by Wright et al.[12] we identified twelve (44%) studies [5, 13, 15, 20, 22, 23, 26, 31, 32, 36–38] which explored EHR based CDS interventions based on point-of-care alerts or reminders (category 3). In addition, three interventions (11%) [17, 27, 34] were order facilitators (category 2), two studies (7%) [19, 30] investigated medication dosing support (category 1), while relevant information display as well as expert systems (category 4 and 5) were each reported only once from an economic perspective (4%).[18, 24] In the remaining five studies [14, 16, 28, 33, 35] interventions from two different categories were explored in combination. Finally, we found three studies [21, 25, 29] in which the option to place a certain order or test, e.g. a laboratory test, was removed from the EHR CPOE system or the clinician’s laboratory ordering preference list. That was also partly explored along with the so called “hard-stop” alert, which requires a clicking response from the physician before being able to move forward.[14] These restrictive frond-end CDS intervention types were not yet mentioned in the pre-defined categories by Wright et al.[12]. Thus, for this study we extend their taxonomy by a new category
7. Restriction of choice[39]
The removal of an order option ultimately resulted in less laboratory tests ordered, and therefore in a reduction of healthcare expenditure in all five studies it was implemented [14, 21, 25, 28, 29].
Economic Impact For Prevalent Application Areas
In Table 3, we summarized our findings and created an overview of application areas and cost outcome measures in relation to the applied CDS intervention types. Due to the heterogeneity of included studies with regard to different types of cost outcomes reported and different intervention duration it was not possible to conduct a subgroup analysis considering the economic impact of each CDS-front end category. A detailed evidence synthesis of all included twenty-seven studies as well as a brief description of their intervention types, their application area and the resulting economic impact is provided in an additional file [see Additional file 3].
| Study | Size¹ | Application area | CDS intervention period (in month) | Cost outcome (per year, in US$, if not other stated) ² ³ |
|---|---|---|---|---|
| 1. Medication (dosing) support | ||||
| Tamblyn [19] | Medium | Reduce out-of-pocket costs for patients with uncomplicated hypertension | 60 | No difference⁴ |
| Stenner [30] | Large | ePrescribing tool for therapeutic interchange prescribing | 18 | - $812,956 |
| 2. Order facilitator | ||||
| Bolles [17] | Small | Inappropriate test ordering for specialized HIV laboratory testing | 6 | + $14,000 to + $92,000 |
| Schnaus [27] | Large | The order “complete blood count without differential” unintentionally changed to “complete blood count with differential” | 23 days | + $87,275⁵ |
| Shaha [34] | Small | CDS order sets for managing new-onset stroke patients | 6 | - $460,000 to - $1,130,000 |
| 3. Point of care alerts or reminders | ||||
| Chen D [13] | Large | Reduce unnecessary imaging studies in patients with low back pain | 12 | -$1,872,000 |
| Chin [15] | Large | Decrease routine testing for 25(OH) vitamin D levels | 12 | -$300,000 |
| Gong [5] | Medium | Inappropriate antibiotic prescribing for acute respiratory infection | 18 | -$500,000 per 30 years and 100,000 individuals ⁸ |
| Bejjanki [20] | Large | Reduce 17 frequently used duplicate laboratory tests | 17 | - $51,206 |
| Chen JR [22] | Small | Directing the physician to order penicillin allergy testing for patients receiving aztreonam | 9 | -$673.73 per patient |
| Heekin [23] | Large | Adherence to 18 different Choosing Wisely (CW) alerts | 36 | -$944 per patient |
| Sharifi [26] | Small | Clinical childhood obesity intervention | 12 | + $175mill. in 10 years ⁸ |
| Procop (a) [31] | Medium | Unnecessary duplicate laboratory testing | 12 | - $94,225 (Hard-Stop) - $45,681 (Smart-Alert) |
| Procop (b) [32] | Large | Reduce unnecessary, same day duplicate orders | 24 | - $91,793 |
| Goodnough [36] | Large | Reduce overutilization in blood transfusion procedure | 36 | - $1,620,000 |
| Razavi [37] | Small | Reduce unnecessary waste in transfusion practice and blood use of cardiothoracic surgeons | 12 | - $62,715 |
| Bridges [38] | Small | Reduce unnecessary acute hepatitis profile laboratory tests | 3 | - $13,580 |
| 4. Relevant information display | ||||
| Fertel [18] | Small | Reduce the amount of frequent or high emergency department utilizers | 24 | - $3,306 average monthly per patient care plan |
| 5. Expert systems | ||||
| Nault [24] | Large | Antimicrobial stewardship that facilitates the post-prescription review process | 36 | - CAD $116,666 |
| 6. Workflow support | ||||
| none | - | - | - | - |
| 7. Restriction of choice | ||||
| MacMillan [21] | Large | Reduce unnecessary frequent red blood cell folate tests | 43 | - CAD $78,180 |
| Sadowski [25] | Medium | Reduce admission order sets, which allowed multiple routine tests to be ordered repetitively | 2 | - $152,496 |
| Konger [29] | Large | Define order frequency rules and reduce duplicate tests | 24 | - $157,782 |
| Studies with combined multiple CDS intervention categories | ||||
| 3. Point of care alerts or reminders & 7. Restriction of choice | ||||
| Marcelin [14] | Large | Reduce inappropriate gastrointestinal pathogen panel testing | 15 | -$51,600 |
| Felcher [28] | Medium | Reduce unnecessary Vitamin D testing | 6 | - $2,800,000 |
| 2. Order facilitator & 3. Point of care alerts | ||||
| Goetz [16] | Large | Decrease serum folate laboratory testing | 12 | -$26,719 |
| 2. Order facilitator & 6. Workflow support | ||||
| Michaelidis [33] | Medium | Reduce inappropriate antibiotic prescribing for acute bronchitis | 6 | +$34 ⁷ ⁸ |
| 1. Medication (dosing) support & 3. Point of care alerts or reminders | ||||
| Forrester [35] | Medium | CPOE CDS vs. paper-based prescribing in reducing medication errors and adverse drug events (ADE) | 10 | - $18 mill. per 10.000 Monte Carlo simulations ⁸ |
| ¹ Size is defined as the following: | ||||
| Number of patients or encounters involved | ||||
| 0-999 small size | ||||
| 1,000–10,000 medium size | ||||
| > 10,000 large size | ||||
| If patient count was not reported, we applied this range of criteria to the amount of triggered alerts in total | ||||
| ² Total cost outcome in the EHR based CDS intervention group compared to the control group or pre-implementation phase. | ||||
| ³ All cost outcomes were scaled and thereupon calculated to overall cost outcome per year. Values are rounded to full integer numbers. Because of the predominantly short CDS intervention period time range, a discount factor is not used for calculation. The original reported cost data is mentioned in an additional file [see Additional file 3]. | ||||
| ⁴ No statistically significant differences between control and intervention group regarding out-of-pocket costs per patient: CAD$252.6 (control) and CAD$261.5 (intervention) / patient year. Similar values for currently treated patients as well as mean annual costs of antihypertensive treatment (CAD$370.90 control and CAD$385.70 intervention) | ||||
| ⁵ Comparative study: Expenditure increase resulted from the unintentional change within the EHR based CDS system | ||||
| ⁷ 5-year societal cost per five cases of acute bronchitis | ||||
| ⁸ Cost estimation based on model | ||||
Application areas for cost-savings
Thereupon, we identified four main application areas based on their investigated prevalence that resulted in cost-savings after EHR based CDS implementation. Firstly, two studies report on essentially reducing unnecessary Vitamin D routine testing that led to a decrease of laboratory test cost of $300,000[15] and $1,4mill.[28] per year.Secondly, two studies addressed the economic outcomes of the reduction of waste in transfusion practice and red blood cell usage.[36, 37] Acquisition product cost of red cell units were decreased with the help of EHR based CDS and resulted in cost savings of in total $4,821,000 within three years[36] and about $62,715 within one year[37] after implementation, respectively.
Thirdly, two cost-effectiveness-analyses modeled the cost outcome of reducing antibiotic prescriptions for acute respiratory infection as well as for acute bronchitis.[5, 33] Gong et al.[5] include a full accounting of costs into their Markov model and explore that the implemented CDS intervention, called “suggested alternatives”, yielded more quality adjusted life years (QALYs) at a lower cost of $500,000 per 100.000 individuals over thirty years of implementation. Michaelidis et al.[33] on the other hand report a small increase in costs compared to a printed decision support system, i.e. posters. However, the outcome of the latter mainly results from a cost difference between the direct costs of poster printing and the computer programming cost.
Lastly, five studies[20, 29, 31, 32, 38] report on the potential for cost savings through reducing duplicate orders or laboratory tests by using hard-stops[32] or applying order frequency rules[20] to prevent ordering the same test within a certain timeframe. Reducing laboratory duplicate tests resulting in savings of $3,395 in three months for a small patient size cohort[38] and up to $315,565 within twenty-four month for a large patient size cohort.[29]
Application areas resulting in cost increase
Furthermore, we also identified risk areas, which possibly lead to a further increase in healthcare expenditure. One study found that after implementing a CPOE system with default settings, specialized HIV laboratory test cost increased by $14,000-$96,000 within six months.[17] Another study reports that an unplanned change of a pre-selected default order for ‘complete blood count’ to ‘complete blood count with differential’ lead to an average cost increase of $293.11 per day.[27] Finally, the implementation of order sets as decision facilitators possibly entail negative economic effects. One study found that only after the uncoupling of Vitamin B12 and serum folate joint orders within predefined order sets, laboratory test cost decreased by about $26,719 per year.[16] Similarly, another study removed the option to order daily routine tests from automated admission order sets and found savings of $26,416 after two months.[25]Cost-effectiveness-analyses Models
Table 4 encompasses an overview of studies which conducted a cost-effectiveness-analysis (CEA) of EHR based CDS interventions considering various cost data as well as economic outcome measures, such as the incremental cost-effectiveness ratio (ICER), which depicts the incremental change in costs divided by the incremental change in health outcome or effect.
| Study | Model time horizon (years) | Choice of model | Implementation and maintenance cost | Total budget impact | ICER |
|---|---|---|---|---|---|
| Gong et al. [5] | 30 | Markov model | $1.91 base case for 100,000 individuals [preexisting EHR] | CDS intervention $17.32 mill. Control $17.82 mill. | $99.8 per QALY in base case scenario Not directly reported |
| Sharifi et al. [26] | 10 | Monte Carlo micro-simulation | $23,542 per PCP group [preexisting EHR] | CDS intervention +$239 mill. | $237 per BMI unit reduction |
| Michaelidis et al. [33] | 5 | Decision analytic tree | $18 base case - medical record programming [preexisting EHR] | CDS intervention $2,802* Control (usual care) $2,768* | $51.51 per antibiotic prescription safely avoided Not directly reported |
| Forrester et al. [35] | 5 | Decision analytic tree | $1,773,000 five years CPOE system costs | CDS CPOE system $25 mill. Control (paper system) $43mill. | $110 per ADE averted† |
| * Cumulative 5-year societal cost per five cases of acute bronchitis | |||||
| † Documented only for the explored modelling scenario no. 2: The Everett Clinic achieved no reduction in paper chart pulls throughout the 5-year time horizon, to explore the effect of inefficiency from running a paper and electronic system in parallel | |||||
Cost-effectiveness-analyses aim to reveal the trade-offs in resource-allocation decisions.[40] In this context, it is essential to investigate when and to what extend upfront and maintenance cost for an EHR based CDS system will be amortized by its benefits, which again can be measured either in health outcomes, such as quality adjusted life years (QALYs) saved or in reduction of unnecessary healthcare utilization.
Generally, two studies report an increase in healthcare expenditure from a societal perspective, [26, 33] while the other two report cost savings from a societal perspective as well as the medical group’s perspective.[5, 35] Notably, the measurement of effectiveness was single study-based estimates in all four studies.
Regarding the consideration of upfront implementation cost, Gong et al.[5] include only base case consolidated cost data of $1.91 for a cohort of 100.000 individuals based on expert opinions. Sharifi et al.[26] include intervention start-up cost for EHR modification of $2.7mill. as well as other direct cost, such as professional care provider training. Michaelidis et al.[33] report implementation and maintenance cost data, which is physician education per hour and medical record and CDS programming per patient of $18 in the base case. Lastly, Forrester et al.[35] report CPOE CDS system cost as hardware, software and maintenance cost starting from $373,000 in year one to $92,000 after five years, as well as personnel, $555,000 in year one, and indirect cost as 3% of the total cost. Interestingly, the latter also include the HITECH Meaningful Use incentives in their model in order to simulate the financial incentives by the Centers for Medicare & Medicaid Services in the US.
Lack Of Considering All Cost Components
Despite revealing major potentials for cost-savings, we could not asses the quality of included studies, because of the lack of cost information provided, or predominantly the lack of considering all relevant cost components. According to the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, most of the reported recommendations were not satisfied.[41] All twenty-three non-model studies (85%) only calculate the economic outcome based on financial data reported before and after intervention implementation, which for instance, ultimately results from the computation of price per healthcare resource utilization times the quantity of used healthcare resources or services. Thus, even though it was not intended in those studies, it is necessary to mention that only four of them adhered to sound economic evaluations as recommended by CHEERS.[5, 35]
The challenge of heterogeneity for the CEA is also aggravated by considering different cost outcomes considered. Two studies do not directly report an incremental cost effectiveness ratio (ICER) for a predefined threshold, nor include comparative metrics.[33, 35] Other standardized metrics, such as the return on investment or net present value, were also not examined in the included studies. Only one study reported the net monetary benefit (NMB) of the intervention in relation to a predefined threshold.[5, 42]
Additional Studies Worth Mentioning
Notably, five more studies [43–47] meet most of our inclusion criteria, but were excluded due to various, although little, deviations. Three studies [43–45] report cost-savings after a bundle of information technology was implemented simultaneously, but the economic benefit could not solely be attributed to the EHR based CDS intervention. The fourth publication is a NHS health technology assessment (HTA) report.[46] In this HTA, a RCT was conducted in 79 general practices in the UK in which a multicomponent intervention was installed using electronic health records in order to reduce antibiotic prescribing for respiratory infections. The authors perform a basic cost-analysis on whether the cost of healthcare utilization, that is the number of provider consultations, will increase during the time of the trial under the CDS intervention arm, and if patients more often re-consult the physician when not given a prescription. However, the authors explored no difference in cost outcome between the intervention and control period.
The last study worth mentioning compared retrospectively generated alerts by an advanced machine learning CDS system with alerts triggered through the home-grown EHR based CDS system.[47] The authors calculated the healthcare costs of potentially prevented adverse drug events and medication errors, and found that by using the advanced machine learning CDS system 68,2% of alerts were only fired by that new system resulting in cost savings of $60.67 per alert.[47] After extrapolating these results to an local patient population of 747,985 over five years they estimated savings of $1,294,457.[47]
Evaluating the economic impact of EHR based CDS interventions and its potential to increase value in healthcare remains a great challenge. Even though we found that twenty-two studies report cost savings, most of them do not include developing or maintaining costs. Therefore, we could not draw a sound conjunction between the costs of vendor-purchased or home-grown systems to their economic benefit. Nonetheless, this study reveals several use cases with coherent CDS tools that have proven to be cost-saving, and could therefore be eligible for other healthcare providers, clinic managers and researchers for implementation or further exploration.
With the majority of implemented CDS interventions based on point-of-care alerts the question remains how more algorithm-based expert systems and multiple interventions will have synergy effects on the economic impact. Considering then the amount of alerts and a healthcare provider’s time expense, a process-cost analysis, such as the time-driven activity based costing approach, could be combined with the CEA to achieve a better understanding of the whole cost cycle as well as productivity effects for healthcare entities.[48, 49] Generally, cost outcome measures continue to require comparative metrics, for instance, as used by Mathias et al., the cost per useful alert.[49] In a simple model the authors introduce this measure to analyze how different parameter affect the cost of implementing EHR based CDS alerts for genomic precision medicine.[50] However, for future economic evaluation of EHR based CDS interventions a more specific approach with regard to individual application areas or medical risk factor focus might result in better cost and outcome comparisons in order to draw a meaningful conclusion.[9]
Another economic challenge to consider are CPOE systems with default lists or opt-out options of orderable tests as well as predefined order sets. The increasing rate of unnecessary lab tests might be fostered due to a ‘button clicking syndrome’, which explains the inducement of moving along inattentively.[17] In times, when reducing alert fatigue is gaining momentum, automation of orders through order sets or joint-order options could ultimately lead to a decrease of value.[51, 52] Additionally, one study also reported an increase in cost after an unplanned change of the CDS system had occurred.[27] Thus, malfunctions or unintended errors by newly integrated health information technology, also called ‘e-Iatrogenesis’[53], may also lead to yet another cost-driver and possibly cause unpredictable economic damage.[54, 55]
Transferability For Other Countries
All included studies where based on cost data and trials from the United States or Canada. Consequently, current research progress on the economic potentials of EHR based CDS systems on rising healthcare expenditure in Europe or worldwide cannot be derived. We found recent studies evaluating the cost-effectiveness of a stand-alone CPOE CDS system in the Netherlands,[56] or comparing the effectiveness of an EHR based CDS intervention in the US, UK, Republic of Korea and Belgium,[57] while another RCT explored the effectiveness of an EHR based CDS intervention for patients with atrial fibrillation and high risk of stroke in Sweden.[58] Yet we found no study that evaluated the potential for increasing value in this present highly promising field of health information technology outside of North America.
However, this study reveals promising cost savings for already implemented health information technology. Even though implementation cost was not considered, on a long-term view these results reveal the potential for cost-savings once implementation costs are amortized. Therefore, the sooner broad health information technology systems will be implemented in other countries around the world and evaluated economically, the earlier cost-benefits and return on investments can be realized.
Support From Policy Makers Could Accelerate Economic Benefits
Interestingly, Forrester et al.[35] include monetary incentives as provided by the Meaningful Use Initiative in the US in their CEA. This financial support covered only a small percentage of total implementation cost, as in their developed model incentive-eligible prescribers received $42,000 over five years, but this nevertheless contributed to the investigated cost-effectiveness of an EHR based CDS intervention compared to paper-based prescribing. Therefore, how policy makers worldwide intent to financially support EHR adoption and incentivize usage of embedded CDS systems is a critical factor for the economic success of such systems. High upfront implementation cost is a major burden for healthcare entities, especially for smaller to middle size practices and hospitals.[59]
Finally, achieving a decrease in healthcare expenditure should never influence a patient’s quality of life nor disease treatment in a negative way. Even though eliminating a laboratory order option from a CPOE system led to cost-savings, the value and health outcome of each patient is of highest importance and should be individually assessed. Rather than proving to have effectively reduced laboratory test costs, future economic evaluation of EHR based CDS systems should focus more on the potentials of health benefits that could be achieved, such as through reduced antibiotic prescriptions or adverse drug events. In the end, competing on shifting costs will not change anything about the main goal of decision stewardship, and that is to increase value in healthcare.[60]
There are some limitations to this study. Firstly, since we only considered English language literature in our economic review, we might not have included international publications in other languages that indeed report on the information technology progress made by other countries regarding the linkage of CDS systems to an existing EHR. Another limitation is the exclusion of EHR cost and price display interventions. Physicians’ literacy towards costs in healthcare hold furthers possibilities to increase the economic benefit CDS interventions, although a systematic review recently found that cost display in EHR CPOE systems does not affect the efficiency and effectiveness domain of healthcare quality.[61] We also excluded other non-monetary impact measures, such as length of stay, which essentially also refer to the economic impact of EHR based CDS implementation.
Overall, the results of our findings might be biased, since we included all types of studies as well as all kinds of monetary outcomes reported. Due to the lack of economic evaluations, included studies tended to declare high cost-savings, but did only consider little to none cost components of implementing such a complex information system. In addition, authors might have been tempted to calculated cost-savings only when the intervention proved to be effective. Finally, based on our inclusion criteria we cannot argue to have involved all existing studies that meet the inclusion criteria of this review. While conducting the study, we found that studies not necessarily report calculated economic outcomes in the title or abstract of their publication, and thus by the nature of following the PRISMA guidelines leading to an exclusion of that study.
Clinical decision support interventions based on electronic health records have an overall positive economic impact. Predominantly point-of-care alerts with regard to unnecessary laboratory testing, efficient transfusion practice, or reduction of antibiotic prescription emerged as application areas with already promising potentials for high cost-savings. Nonetheless, lack of cost data consideration as well as the need for comparative metrics continue to be the reason why the economic dimension of EHR based CDS interventions need to be further explored. Therefore, high quality cost-effectiveness or cost-utility analyses, which include more extensive cost data and consider different economic perspectives, are needed in order to draw a sound conclusion. Finally, introducing personalized health services based on peoples’ electronic health records is yet another promising research field with great potentials for further increasing value in healthcare, and should therefore receive more attention in future research.
Adverse drug events; BPA:Best-practice-alerts; CDS:Clinical decision support; CEA:Cost-effectiveness-analysis; CHEERS:Consolidated Health Economic Evaluation Reporting Standards; CPOE:Computerized provider order entry; CW:Choosing Wisely; EHR:Electronic health record; HTA:Health technology assessment; ICER:Incremental cost-effectiveness ratio; NMB:Net monetary benefit; QALY:Quality adjusted life years; RCT:Randomized controlled trial; PRISMA:Preferred Reporting Items for Systematic Reviews and Meta Analyses;
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and materials
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Funding
This work has received funding from the European Union’s Horizon 2020 research and innovation program under the grant agreement No. 826117 Smart4Health - Building a citizen-centered EU-EHR exchange for personalized health. The funding body had no role in the design of the study, collection or data analysis.
Acknowledgements
Not applicable.
Authors' contributions
DL conceptualized the study. DL and AW conducted the literature searches, literature screening as wellas the analyses. DL drafted the manuscript, and all authors contributed to refining all sections and critically editing the paper.
- OECD10.1787/health_glance-2017-enOECD. (2017), Health at a Glance 2017: OECD Indicators, Publishing OECD. Paris. http://dx.doi.org/10.1787/health_glance-2017-en (accessed 1 October 2019).
- OECD. (2019), OECD Health Statistics 2019. https://www.oecd.org/health/health-data.htm (accessed 1 December 2019).
- Shrank WH, Rogstad TL, Parekh N. Waste in the US Health Care System: Estimated Costs and Potential for Savings. JAMA. 2019;322(15):1501–9. doi:10.1001/jama.2019.13978.
- McGinn T. Putting Meaning into Meaningful Use: A Roadmap to Successful Integration of Evidence at the Point of Care. JMIR Med Inform. 2016;4(2):e16. doi:10.2196/medinform.4553.
- Gong CL, Zangwill KM, Hay JW, et al. Behavioral Economics Interventions to Improve Outpatient Antibiotic Prescribing for Acute Respiratory Infections: a Cost-Effectiveness Analysis. J Gen Intern Med. 2019;34(6):846–54. doi:10.1007/s11606-018-4467-x.
- Atkins D. So Many Nudges, So Little Time: Can Cost-effectiveness Tell Us When It Is Worthwhile to Try to Change Provider Behavior? J Gen Intern Med. 2019;34(6):783–4. doi:10.1007/s11606-019-04871-5.
- Cho I, Bates DW. Behavioral Economics Interventions in Clinical Decision Support Systems. Yearb Med Inform. 2018;27(1):114–21. doi:10.1055/s-0038-1641221.
- Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi:10.1371/journal.pmed.1000097.
- Jacob V, Thota AB, Chattopadhyay SK, et al. Cost and economic benefit of clinical decision support systems for cardiovascular disease prevention: a community guide systematic review. J Am Med Inform Assoc. 2017;24(3):669–76. doi:10.1093/jamia/ocw160.
- Moja L, Kwag KH, Lytras T, et al. Effectiveness of computerized decision support systems linked to electronic health records: A systematic review and meta-analysis. Am J Public Health. 2014;104(12):e12–22. https://doi.org/10.2105/AJPH.2014.302164.
- Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: A systematic review. Ann Intern Med. 2012;157(1):29–43. https://doi.org/10.7326/0003-4819-157-1-201207030-00450.
- Wright A, Sittig DF, Ash JS, et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc. 2011;18(3):232–42. doi:10.1136/amiajnl-2011-000113.
- 10.1016/j.wneu.2019.11.03Chen D, Bhambhvani HP, Hom J, et al. Effect of Electronic Clinical Decision Support on Imaging for the Evaluation of Acute Low Back Pain in the Ambulatory Care Setting. World Neurosurgery 2019;pii:S1878-8750(19)32867-0. doi:10.1016/j.wneu.2019.11.03.
- Marcelin J, Brewer C, Beachy M, et al. Hardwiring diagnostic stewardship using electronic ordering restrictions for gastrointestinal pathogen testing. Infection Control Hospital Epidemiology. 2019;40(6):668–73. doi:10.1017/ice.2019.78.
- Chin KK, Hom J, Tan M, et al. Effect of Electronic Clinical Decision Support on 25(OH) Vitamin D Testing. J Gen Intern Med. 2019;34(9):1697–9. https://doi.org/10.1007/s11606-019-05057-9.
- Goetz C, Di Capua J, Lee I, et al. A Student-Led, Multifaceted Intervention to Decrease Unnecessary Folate Ordering in the Inpatient Setting. J Healthc Qual. 2019;41(5):e54–60. doi:10.1097/JHQ.0000000000000177.
- Bolles K, Woc-Colburn L, Hamill RJ, et al. Ordering patterns and costs of specialized laboratory testing by hospitalists and house staff in hospitalized patients with HIV at a county hospital: An opportunity for diagnostic stewardship. Open Forum Infect Dis. 2019;6(6):ofz158. doi:10.1093/ofid/ofz158.
- Fertel BS, Podolsky SR, Mark J, et al. Impact of an individual plan of care for frequent and high utilizers in a large healthcare system. Am J Emerg Med. 2019;37(11):2039–42. doi:10.1016/j.ajem.2019.02.032.
- Tamblyn R, Winslade N, Qian CJ, et al. What is in your wallet? A cluster randomized trial of the effects of showing comparative patient out-of-pocket costs on primary care prescribing for uncomplicated hypertension. Implement Sci. 2018;13(1):7. doi:10.1186/s13012-017-0701-x.
- Bejjanki H, Mramba LK, Beal SG, et al. The role of a best practice alert in the electronic medical record in reducing repetitive lab tests. Clinicoecon Outcomes Res 2018;10:611–618. doi:10.2147/CEOR.S167499.
- MacMillan TE, Gudgeon P, Yip PM, et al. Reduction in unnecessary red blood cell folate testing by restricting computerized physician order entry in the electronic health record. Am J Med. 2018;131(8):939–44. doi:10.1016/j.amjmed.2018.03.044.
- Chen JR, Tarver SA, Alvarez KS, et al. Improving aztreonam stewardship and cost through a penicillin allergy testing clinical guideline. Open Forum Infect Dis. 2018;5(6):ofy106. doi:10.1093/ofid/ofy106.
- Heekin AM, Kontor J, Sax HC, et al. Choosing Wisely clinical decision support adherence and associated inpatient outcomes. Am J Manag Care. 2018;24(8):361–6.
- Nault V, Pepin J, Beaudoin M, et al. Sustained impact of a computer-assisted antimicrobial stewardship intervention on antimicrobial use and length of stay. J Antimicrob Chemother. 2017;72(3):933–40. doi:10.1093/jac/dkw468.
- Sadowski BW, Lane AB, Wood SM, et al. High-value, cost-conscious care: Iterative systems-based interventions to reduce unnecessary laboratory testing. Am J Med. 2017;130(9):1112.e. 1-1112.e7. doi:10.1016/j.amjmed.2017.02.029.
- Sharifi M, Franz C, Horan CM, et al. Cost-effectiveness of a clinical childhood obesity intervention. Pediatrics 2017;140(5). pii: e20162998. doi:10.1542/peds.2016-2998.
- Schnaus MJ, Michalik M, Skarda P. Effects of electronic medical record display on provider ordering behavior: Leveraging the EMR to improve quality and costs. Am J Med. 2017;130(12):1366–71. doi:10.1016/j.amjmed.2017.08.019.
- Felcher AH, Gold R, Mosen DM, et al. Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc. 2017;24(4):776–80. doi:10.1093/jamia/ocw182.
- Konger RL, Ndekwe P, Jones G, et al. Reduction in Unnecessary Clinical Laboratory Testing Through Utilization Management at a US Government Veterans Affairs Hospital. Am J Clin Pathol. 2016;145(3):355–64. doi:10.1093/ajcp/aqv092.
- Stenner SP, Chakravarthy R, Johnson KB, et al. ePrescribing: Reducing Costs through In-Class Therapeutic Interchange. Appl Clin Inform. 2016;7(4):1168–81.
- 10.1309/AJCPJOJ3HKEBD3TUProcop GW, Keating C, Stagno P, et al (a). Reducing duplicate testing: a comparison of two clinical decision support tools. Am J Clin Pathol 2015;143(5):623-6. doi:10.1309/AJCPJOJ3HKEBD3TU.
- 10.1309/AJCPOWHOIZBZ3FRWProcop GW, Yerian LM, Wyllie R, et al (b). Duplicate laboratory test reduction using a clinical decision support tool. Am J Clin Pathol 2014;141(5):718 – 23. doi:10.1309/AJCPOWHOIZBZ3FRW.
- Michaelidis CI, Kern MS, Smith KJ. Cost-Effectiveness of Decision Support Strategies in Acute Bronchitis. J Gen Intern Med. 2015;30(10):1505–10. doi:10.1007/s11606-015-3289-3.
- Shaha SH, Gilbert-Bradley D. Improved Immediate and Continued-Care Outcomes for Stroke through Community-Wide Data. Stud Health Technol Inform. 2015;209:140–6.
- Forrester SH, Hepp Z, Roth JA, et al. Cost-effectiveness of a computerized provider order entry system in improving medication safety ambulatory care. Value Health. 2014;17(4):340–9. doi:10.1016/j.jval.2014.01.009.
- Goodnough LT, Maggio P, Hadhazy E, et al. Restrictive blood transfusion practices are associated with improved patient outcomes. Transfusion. 2014;54:2753–9. doi:10.1111/trf.12723.
- Razavi SA, Carter AB, Puskas JD, et al. Reduced red blood cell transfusion in cardiothoracic surgery after implementation of a novel clinical decision support tool. J Am Coll Surg. 2014;219(5):1028–36. doi:10.1016/j.jamcollsurg.2014.06.012.
- Bridges SA, Papa L, Norris AE, et al. Duplicated laboratory tests: evaluation of a computerized alert intervention abstract. J Healthc Qual. 2014;36(3):46–53. doi:10.1111/j.1945-1474.2012.00219.x.
- Ubel PA, Rosenthal MB. Beyond Nudges - When Improving Health Calls for Greater Assertiveness. N Engl J Med. 2019;380(4):309–11. doi:10.1056/NEJMp1806371.
- Neumann PJ, Sanders GD. Cost-Effectiveness Analysis 2.0. N Engl J Med. 2017;376(3):203–5. doi:10.1056/NEJMp1612619.
- Husereau D, Drummond M, Petrou S, et al. Consolidated health economic evaluation reporting standards (CHEERS)—explanation and elaboration: a report of the ISPOR Health Economic Evaluations Publication Guidelines Good Reporting Practices Task Force. Value Health. 2013;16(2):231–50.
- Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making. 1998;18:68–80.
- Graven M, Allen P, Smith I, et al. Decline in mortality with the Belize Integrated Patient-Centred Country Wide Health Information System (BHIS) with embedded program management. Int J Med Inform. 2013;82(10):954–63. doi:10.1016/j.ijmedinf.2013.06.003.
- Riahi S, Fischler I, Stuckey MI, et al. The Value of electronic medical record implementation in mental health care: A Case Study. JMIR Med Inform. 2017;5(1):e1.
- Olchanski N, Dziadzko MA, Tiong IC, et al. Can a novel ICU data display positively affect patient outcomes and save lives? J Med Syst. 2017;41(11):171. doi:10.1007/s10916-017-0810-8.
- Gulliford MC, Juszczyk D, Prevost AT, et al. Electronically delivered interventions to reduce antibiotic prescribing for respiratory infections in primary care: cluster RCT using electronic health records and cohort study. Health Technol Assess. 2019;23(11):1–70. doi:10.3310/hta23110.
- Rozenblum R, Rodriguez-Monguio R, Volk LA, et al. Using a Machine Learning System to Identify and Prevent Medication Prescribing Errors: A Clinical and Cost Analysis Evaluation. Jt Comm J Qual Patient Saf. 2020;46(1):3–10. doi:10.1016/j.jcjq.2019.09.008.
- Keel G, Savage C, Rafiq M, et al. Time-driven activity-based costing in health care: A systematic review of the literature. Health Policy. 2017;121(7):755–63. doi:10.1016/j.healthpol.2017.04.013.
- Campanale C, Cinquini L, Tenucci A. Time-driven activity-based costing to improve transparency and decision making in healthcare. Management. 2014;11:165–86.
- Mathias PC, Tarczy-Hornoch P, Shirts BH. Modeling the costs of clinical decision support for genomic precision medicine. AMIA Jt Summits Transl Sci Proc 2016;2016:60 – 4.eCollection 2016..
- Hussain MI, Reynolds TL, Zheng K, et al. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc. 2019;26(10):1141–9. doi:10.1093/jamia/ocz095.
- Hakim I, Hathi S, Nair A, et al. Electronic health records and the frequency of diagnostic test orders. Am J Manag Care. 2017;23(1):e16–23.
- Payne TH, Bates DW, Berner ES, et al. Healthcare information technology and economics. J Am Med Inform Assoc. 2013;20(2):212–7.
- Wright A, Ai A, Ash J, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc. 2018;25(5):496–506. doi:10.1093/jamia/ocx106.
- Aaron S, McEvoy DS, Ray S, et al. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc. 2019;26(1):37–43. doi:10.1093/jamia/ocy139.
- Vermeulen KM, van Doormaal JE, Zaal RJ, et al. Cost-effectiveness of an electronic medication ordering system (CPOE/CDSS) in hospitalized patients. Int J Med Inform. 2014;83(8):572–80. doi:10.1016/j.ijmedinf.2014.05.003.
- Cornu P, Phansalkar S, Seger DL, et al. High-priority and low-priority drug-drug interactions in different international electronic health record systems: A comparative study. Int J Med Inform. 2018;111:165–71. doi:10.1016/j.ijmedinf.2017.12.027.
- Karlsson LO, Nilsson S, Bång M, et al. A clinical decision support tool for improving adherence to guidelines on anticoagulant therapy in patients with atrial fibrillation at risk of stroke: A cluster-randomized trial in a Swedish primary care setting (the CDS-AF study). PLoS Med. 2018;15(3):e1002528. doi:10.1371/journal.pmed.1002528.
- Slight SP, Quinn C, Avery AJ, et al. A qualitative study identifying the cost categories associated with electronic health record implementation in the UK. J Am Med Inform Assoc. 2014;21(e2):e226-31. doi:10.1136/amiajnl-2013-002404.
- Porter ME. What is value in health care? N Engl J Med 2010;363(26):2477-81. doi:0.1056/NEJMp1011024.
- Mummadi SR, Mishra R. Effectiveness of provider price display in computerized physician order entry (CPOE) on healthcare quality: a systematic review. J Am Med Inform Assoc. 2018;25(9):1228–39. doi:10.1093/jamia/ocy076.
- Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: A randomized clinical trial. JAMA. 2016;315(6):562–70. doi:10.1001/jama.2016.0275.
- Taveras EM, Marshall R, Kleinman KP, et al. Comparative effectiveness of childhood obesity interventions in pediatric primary care: a cluster-randomized clinical trial. JAMA Pediatr. 2015;169(6):535–42. doi:10.1001/jamapediatrics.2015.0182.
- Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267–73. doi:10.1001/jamainternmed.2013.1589.
- 10.1007/s11606-015-3431-2Black JT. Capsule Commentary on Michaelidis et al., Cost-Effectiveness of decision support strategies in acute bronchitis. J Gen Intern Med 2016;31(3):328. doi:10.1007/s11606-015-3431-2.
- Devine EB, Hansen RN, Wilson-Norton JL, et al. The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc. 2010;17(1):78–84. doi:10.1197/jamia.M3285.