False positive analysis of machine-learning based sepsis prediction

doi:10.21203/rs.3.rs-1746922/v1

Download PDF

Research Article

False positive analysis of machine-learning based sepsis prediction

https://doi.org/10.21203/rs.3.rs-1746922/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Despite the emergence of several promising machine learning models for prediction of patients at risk of sepsis, investigation of factors that contribute to false positive rates has not been performed. Here, we conducted a false positive analysis to determine the sources of false alerts and examine a mitigation methodology for reduction of false positive rates in sepsis prediction.

Methods

Analysis of false positive predictions from our sepsis prediction model was conducted and patient populations were stratified by underlying conditions that contribute to false positives. Sensitivity and specificity results for each subpopulation were calculated while adjusting the initial adjustment threshold. An optimal threshold was applied to an entire test set to observe the effect on performance.

Results

Certain underlying conditions mimic sepsis and contribute to false positive rates. The optimal threshold adjustment for the whole population resulted in greater reductions of false positives than true positives.

Conclusions

An optimal threshold adjustment to a whole population based on specific patient populations stratified by underlying conditions reduces the number of false positive cases while increasing the specificity and positive predictive value and still maintaining high sensitivity.

Preventive Medicine

false positive

sepsis prediction

false alarms

machine learning

Sepsis is one of the leading causes of admission to the intensive care unit and may be life-threatening if not diagnosed and treated promptly. In 2017, there were approximately 11 million sepsis-related deaths worldwide, accounting for 19.7% of all-cause deaths.(1) Early sepsis detection allows for timely implementation of interventional measures to reduce mortality rates.(2, 3) Machine learning (ML) algorithms have shown considerable potential in the risk-stratification of sepsis patients, early prediction, and detection of sepsis.(4) Despite the success seen with ML-based applications in the medical field, a high rate of false positive alerts bear clinical implications.(5, 6) False positive alerts create noise disturbance, desensitization, and a reduction in quality of patient care.(7) Alert fatigue due to false positive alerts poses a significant barrier to clinician adoption of clinical decision support tools.(8)

Current ML-based studies often do not examine underlying factors that contribute to false positive cases. Previous studies have briefly discussed the topic,(9–11) however, mitigation methods and their effects on false positive rates have yet to be fully investigated. In this retrospective study, we present a novel methodology for a detailed analysis of false positive cases resulting from our sepsis prediction algorithm. We examine specific patient populations stratified by underlying conditions found to contribute to sources of false positive rates and evaluate the effect of an optimized threshold adjustment on a whole population test set to achieve increases in specificity and positive predictive value while maintaining high sensitivity in overall model performance.

The data for this analysis were obtained from patient electronic health records (EHRs) of inpatient and emergency department encounters from six medical sites around the United States. The data were divided into training (60,272 patient encounters) and three testing sets (35,476). This analysis is based on the three testing sets: a holdout test set (16,754), temporal validation set (7,235), and external validation set (11,487). In compliance with the Health Insurance Portability and Accountability Act, all data were de-identified prior to extraction and retrospective analysis. The full descriptions of data and model design used in this analysis are presented in the Supplementary Material S1.

The false positive analysis began with an inspection of the false positive predictions from our sepsis prediction model on the holdout test set. Supplementary Figure S1 summarizes the study methodology. The sepsis definition used in the sepsis prediction algorithm to identify positive cases included two components: patients with sepsis or infection diagnosis who also obtained a Sequential Organ Failure Assessment (SOFA) score of two or greater (see Supplementary Material S1 for greater detail). The analysis showed that a majority of the false positives included patients with underlying conditions who met the SOFA criteria but a diagnosis for infection or sepsis was not identified, and thus these patients were not considered septic. These underlying conditions acted as sepsis mimics, prompting high SOFA scores that triggered the sepsis prediction algorithm. They have varied prevalence in the dataset; to ensure that adequate sample sizes were available for analysis, only conditions with a prevalence > 5% on the holdout test set were further analyzed. The full list of underlying conditions is presented in Supplementary Table S1.

The performance results including sensitivity, specificity and positive predictive value (PPV) of the sepsis prediction algorithm were analyzed for each test set. The combination of sensitivity and specificity produce different operating points. An initial adjustment threshold was obtained with a sensitivity and specificity of 0.88 and 0.83, respectively. An analysis of the tradeoff between the sensitivity and specificity for each subpopulation with an underlying condition was then performed as the initial adjustment threshold varied. Subpopulations with underlying conditions in which variation of the adjustment threshold allowed for an increase in specificity while the sensitivity did not decrease below 0.85 were considered ideal candidates for further analysis. The optimal adjustment threshold changes for each subpopulation with an underlying condition were determined and applied separately to an entire test population, and the effects of the changes on each subpopulation in comparison to the entire test set were observed. The threshold changes from the individual subpopulations with underlying conditions were then applied together in the form of an optimal threshold adjustment to an entire test population, and the effects of the changes were observed. The optimal threshold adjustment on an entire test set was obtained by testing all possible combinations of new thresholds from the subpopulation results while maintaining sensitivity on the entire test set ≥ 0.85.

The resultant effects of reducing the true and false positive cases due to the threshold changes in each subpopulation were calculated. The reduction in true positive cases versus the false positive cases were calculated for each subpopulation with an underlying condition in comparison to the effect of each subpopulation on the entire hold test set. The reduction in true and false positive cases due to optimal threshold adjustments were also calculated for all three test sets.

Inspection of the model performance results showed that the patients with false positive predictions had a higher likelihood of a high SOFA score as compared to patients with true negative predictions (Supplementary Table S2), indicating that the majority of false positives were due to underlying conditions triggering the SOFA criteria and acting as sepsis mimics. These underlying conditions with high SOFA scores but without sepsis or infection identified were associated with seven major disease areas with prevalence > 5%. The underlying conditions included: age ≥ 70; history of acute kidney injury; history of cardiovascular disease; history of cerebrovascular disease; history of diabetes; history of pulmonary disease; and pneumonia (Table 1). While pneumonia had a prevalence > 5%, it was not associated with high SOFA scores in the false positive cases and thus not a candidate for threshold adjustment.

Table 1

Characteristics of the patient subpopulations with underlying conditions in each test set. The total number of patients and prevalence (%), as well as the number of sepsis positive cases and sepsis negative cases in each test set are presented. Acute kidney injury (AKI)
	Holdout test set			Temporal test set			External validation set
	Number of patients (%)	Sepsis +	Sepsis -	Number of patients (%)	Sepsis +	Sepsis -	Number of patients (%)	Sepsis +	Sepsis -
Totals in testing sets	16754	1545	15209	7235	706	6529	11487	1105	10382
Age > 70	4664 (27.84%)	698	3966	2772 (38.31%)	416	2356	2171 (18.88%)	1811	360
AKI	939 (5.60%)	338	601	805 (11.12%)	226	579	473 (4.12%)	436	37
Cardio- vascular	1461 (23.90%)	659	3340	3209 (44.35%)	404	2805	2127 (18.52%)	2049	78
Cerebro- vascular	1461 (8.71%)	171	1290	1088 (15.04%)	131	957	618 (5.38%)	602	16
Diabetes	1247 (7.44%)	266	981	986 (13.62%)	161	825	1037 (9.02%)	996	41
Pulmonary	2006 (12.00%)	420	1586	1577 (21.80%)	289	1288	647 (5.63%)	612	35

The percent changes in sensitivity, specificity, and PPV due to threshold adjustments in each subpopulation with an underlying condition are presented at the subpopulation level and whole population level on the hold-out test set in Table 2. The percent changes from optimal threshold adjustments based on underlying conditions for each of the three test sets are also shown.

Table 2

(A) The percent changes in sensitivity, specificity, and positivity predictive value (PPV) due to threshold adjustments of subpopulations with underlying conditions versus the whole population on the holdout test set. (B) The percent changes in performances of the three testing sets due to optimal threshold adjustments from all underlying conditions.
A.
Subpopulation vs whole population	Adjusted threshold	Sensitivity	% Change in sensitivity	Specificity	% Change in specificity	PPV	% Change in PPV
Subpopulation with underlying condition: Age ≥ 70
Subpopulation	0.48	0.898	-5.35	0.722	6.65	0.363	8.54
Subpopulation	0.48	0.85	-5.35	0.77	6.65	0.394	8.54
Whole population	0.53	0.880	-1.93	0.830	1.45	0.340	4.71
Whole population	0.53	0.863	-1.93	0.842	1.45	0.356	4.71
Subpopulation with underlying condition: Acute kidney injury
Subpopulation	0.48	0.932	-8.26	0.614	15.15	0.576	7.99
Subpopulation	0.48	0.855	-8.26	0.707	15.15	0.622	7.99
Whole population	0.56	0.880	-1.25	0.830	0.36	0.340	1.76
Whole population	0.56	0.869	-1.25	0.833	0.36	0.346	1.76
Subpopulation with underlying condition: Cardiovascular disease
Subpopulation	0.48	0.914	-6.24	0.734	7.08	0.404	9.16
Subpopulation	0.48	0.857	-6.24	0.786	7.08	0.441	9.16
Whole population	0.54	0.880	-2.16	0.830	1.20	0.340	4.12
Whole population	0.54	0.861	-2.16	0.840	1.20	0.354	4.12
Subpopulation with underlying condition: Cerebrovascular disease
Subpopulation	0.48	0.901	-5.22	0.806	4.09	0.381	8.66
Subpopulation	0.48	0.854	-5.22	0.839	4.09	0.414	8.66
Whole population	0.53	0.880	0.00	0.830	0.24	0.340	2.06
Whole population	0.53	0.880	0.00	0.832	0.24	0.347	2.06
Subpopulation with underlying condition: Diabetes
Subpopulation	0.48	0.883	-3.74	0.721	6.52	0.462	7.79
Subpopulation	0.48	0.85	-3.74	0.768	6.52	0.498	7.79
Whole population	0.53	0.880	0.00	0.830	0.24	0.340	2.06
Whole population	0.53	0.880	0.00	0.832	0.24	0.347	2.06
Subpopulation with underlying condition: Pulmonary disease
Subpopulation	0.48	0.929	-7.21	0.673	11.0	0.429	10.49
Subpopulation	0.48	0.862	-7.21	0.747	11.0	0.474	10.49
Whole population	0.56	0.880	-1.48	0.830	0.84	0.340	3.24
Whole population	0.56	0.867	-1.48	0.837	0.84	0.351	3.24
B.
Test set		Sensitivity	% Change in sensitivity	Specificity	% Change in specificity	PPV	% Change in PPV
Holdout test set		0.885	-3.95	0.829	2.29	0.345	4.93
Holdout test set		0.85	-3.95	0.848	2.29	0.362	4.93
Temporal test set		0.887	-4.17	0.795	3.52	0.319	6.9
Temporal test set		0.85	-4.17	0.823	3.52	0.341	6.9
External validation set		0.944	-0.74	0.754	1.86	0.29	3.79
External validation set		0.937	-0.74	0.768	1.86	0.301	3.79

The curves of sensitivity, specificity, and PPV in response to variance in the adjustment threshold are presented in Fig. 1 for the subpopulations with the underlying conditions of age ≥ 70 and cardiovascular disease. Changes in these specific subpopulations and the changes in the whole population of the holdout test set are shown. The figures depicting these changes in the other subpopulations with underlying conditions are presented in Supplementary Figure S2.

The percent change of the sensitivity, specificity, and PPV due to threshold adjustments for each subpopulation with an underlying condition on the performance of the entire holdout test are shown in Fig. 2. The comparison of percent changes in sensitivity, specificity, and PPV for the three test sets due to optimal threshold adjustments from all underlying conditions are also shown.

The reduction in true positive cases versus false positive cases were analyzed and results for the effect on the holdout test set from each subpopulation with an underlying condition are shown in Fig. 3 as well as the effect on all three test sets. Detailed results of the reduction in true and false positive analysis are presented in Supplementary Table S3.

This study presents an in-depth analysis of the false positive cases resulting from a sepsis prediction algorithm and suggests a methodology to mitigate false alarms. To our knowledge, this is the first focused false positive analysis for ML sepsis prediction model. By examining patients with underlying conditions that register high SOFA scores but without any sepsis or infection identified, we were able to identify major contributors to false alarms. The methodology we propose resulted in percent change increases in PPV up to 6.9% and specificity up to 3.52% while the sensitivity decreased by 4.17% but still maintained above the 0.85 threshold. Importantly, we demonstrate that this methodology facilitated up to a 10.89% reduction in false positives while the reduction in true positives was only 3.95% on the test sets.

Artificial intelligence-based sepsis prediction models have high false positive rates, ranging between 15–47%.(4, 12, 13) When utilizing machine learning, causes of false alerts may include site-to-site variation,(14) differences in hospital workflow and practices,(15) and variation in data missingness.(16) Brief analyses of false alarms have been conducted in other prediction algorithms.(9, 17) However, they did not provide a methodology to mitigate false positives nor provide evidence of successful reduction in false alerts. One important aspect recognized in prior studies is the effect of sepsis mimics via high SOFA scores on false positives.(18–20) To explore this in more detail, we examined several comorbidities that give rise to high SOFA scores, but do not necessarily lead to sepsis. By focusing on these individual subpopulations, and adjusting their thresholds in the ML model, we devised a method to reduce false positives for our sepsis prediction algorithm while having a fixed sensitivity of > 0.85. Our findings suggest this methodology could be used in broader application to other machine learning prediction algorithms to reduce false alarms.

The substantial reduction in false positives in comparison to true positives suggests this methodology could prove useful in clinical settings. However, to implement reductions in false positive cases in a prospective setting would require adjustments. One manner to achieve this is to perform scaling of the weight of specific underlying conditions during model development.(21) Essentially, underlying conditions that substantially contribute to more false positive cases could be weighted to reduce the effect on model prediction results. This proposed method would reduce the number of false positive cases before model output in a streamlined manner. Another method to achieve a reduction in false positives would be to introduce a notification caveat when an alert is generated if a patient is identified as having a known underlying condition. Completely excluding patients with known underlying conditions that mimic sepsis could be dangerous as missed detections can cause considerable harm, and those patients are still potentially quite sick. Producing a notification caveat would enable the health care provider to use their best judgment regarding patient status and potential treatment actions. These methods of implementing reductions in false positives would need to be fully analyzed to ascertain the potential benefit in mitigating clinician workload as well as improving patient outcomes.

Limitations

This study has several limitations. It is retrospective in nature and thus the performance of the algorithm and the underlying conditions affecting the false positive cases may be different in a prospective clinical setting. External validation in prospective clinical settings are necessary to determine the consistency of the model output and the analysis of false positive cases. This would also be necessary to determine how health care providers may respond to alerts for patients at risk of sepsis and also assess the potential benefit in sepsis patient outcomes.

Future perspectives

This detailed analysis of false positive cases highlights potential false alerts that could be reduced. Future development of sepsis prediction algorithms that include false positive analysis, either in algorithm design and development or as post hoc analysis of algorithm output, may provide pre-alert notification stratification informed by end-user preferences.

Reductions in false alarms are important to relieve clinician workload burden while maintaining accurate identification of septic patients. We present a methodology that demonstrates the number of false positive cases may be reduced while increasing the specificity and PPV and still maintaining high sensitivity of our sepsis prediction algorithm. The clinical usefulness of our proposed methodology warrants further investigation in prospective settings to potentially improve sepsis-related outcomes.

COMPETING INTERESTS

All authors are or have been employees or contractors of Dascena, Inc.

ETHICS

The data acquisition and subsequent study were determined to not constitute human subjects research due to the use of de-identified data. The use of de-identified retrospective data is classified as a non-human subject study and exempt from Institutional Review Board approval, consistent with 45 Code of Federal Regulations 46.102.

FUNDING

The authors declare that no public funding has been used in this study.

DATA AVAILABILITY

The data that support the findings of this study have restrictions that apply to the availability of these data and are not publicly available. The MLA code developed in this study is proprietary and not publicly available.

Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. The Lancet. 2020 Jan 18;395(10219):200–11.
Burrell AR, McLaws ML, Fullick M, Sullivan RB, Sindhusake D. SEPSIS KILLS: early intervention saves lives. Med J Aust. 2016 Feb 1;204(2):73.
Levy MM, Evans LE, Rhodes A. The Surviving Sepsis Campaign Bundle: 2018 Update. Crit Care Med. 2018 Jun;46(6):997–1000.
Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020 Mar;46(3):383–400.
Nguyen J, Davis K, Guglielmello G, Stawicki SP. Combating Alarm Fatigue: The Quest for More Accurate and Safer Clinical Monitoring Equipment [Internet]. Vignettes in Patient Safety - Volume 4. IntechOpen; 2019 [cited 2022 Jun 8]. Available from: https://www.intechopen.com/chapters/undefined/state.item.id
Ruskin KJ, Hueske-Kraus D. Alarm fatigue: impacts on patient safety. Curr Opin Anaesthesiol. 2015 Dec;28(6):685–90.
Cvach M. Monitor alarm fatigue: an integrative review. Biomed Instrum Technol. 2012 Aug;46(4):268–77.
Olakotan OO, Yusof MMohd. Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow. J Biomed Inform. 2020 Jun 1;106:103453.
Shashikumar SP, Wardi G, Malhotra A, Nemati S. Artificial intelligence sepsis prediction algorithm learns to say “I don’t know.” Npj Digit Med. 2021 Sep 9;4(1):1–9.
Scanlon LA, O’Hara C, Garbett A, Barker-Hewitt M, Barriuso J. Developing an Agnostic Risk Prediction Model for Early AKI Detection in Cancer Patients. Cancers. 2021 Aug 20;13(16):4182.
He J, Lin J, Duan M. Application of Machine Learning to Predict Acute Kidney Disease in Patients With Sepsis Associated Acute Kidney Injury. Front Med [Internet]. 2021 [cited 2022 May 3];8. Available from: https://www.frontiersin.org/article/10.3389/fmed.2021.792974
Persson I, Östling A, Arlbrandt M, Söderberg J, Becedas D. A Machine Learning Sepsis Prediction Algorithm for Intended Intensive Care Unit Use (NAVOY Sepsis): Proof-of-Concept Study. JMIR Form Res. 2021 Sep 30;5(9):e28000.
Goh KH, Wang L, Yeow AYK, Poh H, Li K, Yeow JJL, et al. Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare. Nat Commun. 2021 Jan 29;12(1):711.
Elbattah M, Molloy O. Using Machine Learning to Predict Length of Stay and Discharge Destination for Hip-Fracture Patients. In: Bi Y, Kapoor S, Bhatia R, editors. Proceedings of SAI Intelligent Systems Conference (IntelliSys) 2016. Cham: Springer International Publishing; 2018. p. 207–17. (Lecture Notes in Networks and Systems).
McCoy A, Das R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual. 2017;6(2):e000158.
Ho JC, Lee CH, Ghosh J. Septic Shock Prediction for Patients with Missing Data. ACM Trans Manag Inf Syst. 2014 Apr 1;5(1):1:1–1:15.
Tomašev N, Glorot X, Rae JW, Zielinski M, Askham H, Saraiva A, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019 Aug 1;572(7767):116–9.
Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU. Crit Care Med. 2018 Apr;46(4):547–53.
Probst L, Schalk E, Liebregts T, Zeremski V, Tzalavras A, von Bergwelt-Baildon M, et al. Prognostic accuracy of SOFA, qSOFA and SIRS criteria in hematological cancer patients: a retrospective multicenter study. J Intensive Care. 2019 Aug 7;7:41.
Tian H, Zhou J, Weng L, Hu X, Peng J, Wang C, et al. Accuracy of qSOFA for the diagnosis of sepsis-3: a secondary analysis of a population-based cohort study. J Thorac Dis [Internet]. 2019 May [cited 2022 Jun 10];11(5). Available from: https://jtd.amegroups.com/article/view/28675
Ahsan MM, Mahmud MAP, Saha PK, Gupta KD, Siddique Z. Effect of Data Scaling Methods on Machine Learning Algorithms and Model Performance. Technologies. 2021 Sep;9(3):52.

FPASupplementaryMaterial.pdf

Download PDF

Version 1

posted

You are reading this latest preprint version

False positive analysis of machine-learning based sepsis prediction

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Introduction

Methods

Results

Discussion

Limitations

Future perspectives

Conclusion

Declarations

References

Supplementary Files

Status:

Version 1