Two models were developed in accordance with TRIPOD methodology to predict death during hospital admission among SARS-CoV-2 patients. Both models demonstrate acceptable sensitivity and good specificity. Although both have good accuracy, the ANN has significantly greater discriminatory ability. Both models demonstrate acceptable calibration. Developing robust prognostic models for SARS-CoV-2 has benefits for the patient, medical departments, and hospital organisations.
Previous literature reports mixed performance of machine learning and deep learning techniques when compared to regression analysis.4 Whilst machine learning does not obviate the need for classical methods,20 machine learning techniques have been shown to perform significantly better than classical regression models in high-dimensionality datasets.21 Furthermore, ANNs have been shown to perform well on datasets of varying size.22–24 Our results support the use of an ANN in a moderate sized, high-dimensional dataset, whilst having a non-inferior performance profile to a Cox regression model.
The Cox regression model used 11 predictors to calculate survival function, whilst the ANN uses all 21 input features, and attributes different weightings to each feature. Both models identify confusion, collapse, dyspnoea, cough, chronic kidney disease, heart failure, cerebrovascular event history, fever, and sex as more significant predictors of mortality. The ANN additionally identifies ischaemic heart disease and hypertension as important features. Abdominal pain is considered to have little effect on model output by the ANN, which is a significant ‘protective factor’ in the Cox model. In this context, abdominal pain may represent a milder form of SARS-CoV-2. These variables, in particular the comorbidities, have been shown to be associated with mortality in the current literature, such as the ISARIC protocol, which analysed 20133 SARS-CoV-2 positive patients.25
The Cox regression model accounts for censored patients in the study and therefore no patients were excluded on account of not having a recorded outcome at the end of the follow-up period. This avoids the introduction of sampling bias. The predictors chosen for inclusion in both models can be accrued from an initial encounter with a healthcare worker and relate to the underlying clinical condition of each patient. This has a dual benefit. Firstly, this standardises the data-collection process and ensures both models are compared on a congruent dataset. Secondly, the nature of the predictors means that the intended use of the models is clear in that they both produce a point-of-admission mortality prediction, which is particularly applicable to the development of medical calculators. The models analyse the outcomes for laboratory confirmed SARS-CoV-2 patients, eliminating potential bias introduced by including suspected cases who are subsequently diagnosed with other conditions.
The predictive models here do have several limitations, however. There are a variety of haematological and radiological predictors which have been associated with SARS-CoV-2 outcomes which are not included in our models.26,27 Whilst our current models can produce point-of-admission outcome predictions due to the relative ease of collecting demographic, comorbidity and symptom data, additional clinical parameters could be introduced in future to improve the predictive accuracy of the models. We could not account for patients who were admitted for, and diagnosed with SARS-CoV-2, but may have died due to another comorbidity. However, this likely represents a minority of patient deaths. The Cox regression model predicts survival function at day 13; whilst this accounts for the majority of hospital admission lengths, predicting survival in this way may overestimate survival chance for outliers who died at a later date. In contrast, the ANN model produces an overall risk prediction irrespective of length of admission. However, given the median length of stay of 7 days with an upper quartile of 13 days, predictions from the ANN should be used cautiously for longer lengths of hospital stay. There may be a delay between patient presentation and obtaining a laboratory diagnosis. Therefore, whilst it is possible to use either model at the point of admission, the prediction should only be applied to patients who have a confirmed diagnosis of SARS-CoV-2. Finally, data was collected at a single site during a period of high prevalence, and therefore results should be generalised with caution to other populations and those with a different SARS-CoV-2 prevalence.
A prospective, multi-centre analysis is required to further validate the model and improve generalisability of results. Machine learning techniques are ideal for fluctuating environments as they can adapt to new data. For example, using online learning, an ANN can train incrementally by being fed data instances sequentially. Each step is relatively fast and cheap, meaning the system can continuously learn as more data is available. This represents a major advantage relative to static statistical models.28 Future research should focus on implementing adaptive workflows to allow for multi-site data collection and flexible systems which learn incrementally. Furthermore, deep learning techniques such as recurrent neural networks can be used for time-series analysis, and therefore account for important events such as ICU admission as they occur. This may represent an additional avenue for further research. Finally, other decision points in SARS-CoV-2 patient journeys need to be predicted, and adapting the models to predict need for antibacterial agents for secondary infection29, or for steroids where indicated30, are clear avenues for exploration.