Multivariable prediction model for predicting deaths in severe dengue cases

Background. Many predictive models have been developed to predict an outbreak, identify and stratify dengue but none has predicted death in severe dengue cases. To build a predictive model for deaths in severe dengue, a multicentre retrospective cohort study was conducted. Methods. Patients with severe dengue based on the World Health Organisation (WHO) 2009 classi�cation were studied. Demographic, clinical and laboratory data were collected at diagnosis of severe dengue. Penalised regression was used for variable selection and model-building. Ten-fold cross-validation with 1000 repeats were performed for internal validation. Results. A cohort of 786 severe dengue cases, including 35 deaths, was analysed. Our model that predicts death in severe dengue cases comprises eight independent predictors: persistent diarrhoea, body mass index, respiratory rate, platelet count, aspartate transaminase, serum bicarbonate, serum lactate and serum albumin. The area under the receiver operating characteristic curve is 89·6% with a sensitivity of 99·6%, speci�city of 23·6%, positive predictive value of 96·6%, negative predictive value of 71·1%, positive likelihood ratio 1·45 and negative likelihood ratio 0·01. We also found that the proportion of patients that were in the febrile phase at diagnosis of severe dengue for the overall cohort, decompensated and compensated shock were 74·3%, 73% and 75·4%, respectively. Conclusions. We developed a high-performance dengue death prediction model comprising clinical and laboratory data, and deployed an open-access web-based tool (www.saifulsafuan.com/REPROSED2017E2) for any centre to utilise for local validation. We additionally found that a large majority of patients developed severe dengue during the febrile phase.


Introduction
The world is burdened with an estimated 96 million dengue infection every year. 1 The World Health Organisation estimated that there are 500,000 severe dengue cases per year. 2 While the feared Ebola has killed 12,962 people from 1976 to 2015 3 , dengue is estimated to kill approximately 12,500 people each year. 2 In a recent estimate of 12 countries in South East Asia, between the years 2001-2010, annual dengue cases amounted to 2•9 million cases with 5906 deaths per year. 4Dengue is a common and endemic in Malaysia.The number of recorded dengue cases in Malaysia has risen from 6543 to 120,836 cases between 1995 to 2015. 5 The incidence rate had risen from 31•6/100,000 population in the year 2000 to 396•4/100,000 in 2015. 5Correspondingly, the number of deaths has increased from 28 in 1995 to 336 in 2015. 5The case fatality rate, however, remained at around 0•2% per year, for the exception of years 2011-2012. 5Dengue in Malaysia predominantly involves ages 15 and above. 6 South-East Asia, the estimated economic burden amounted to USD950 million [95% CI: USD610-1384 million] or USD1•65 [95%CI: USD1•06-2•41] per capita annually. 4In 2010, USD73•5 million was spent in Malaysia for its National Dengue Vector Control Program which is equivalent to USD2•68 per capita.This represents 0•03% of Malaysia's gross domestic product. 7[9] In order to prevent death due to dengue, strategies employing prediction models at various aspects of the infection have been made.1][12][13][14][15][16] Several studies have looked at two clinical aspects of dengue management: predicting the identi cation of dengue in unselected cases of febrile illnesses, 17,18 and predicting the severity of the disease. 17,19,20,21Early recognition of dengue infection and severity strati cation would improve clinical management amnd lead to a better outcome.
Predicting and estimating the probability of death is one aspect of clinical dengue management that has never been addressed.The ability to estimate the probability of death in illness will assist in therapeutic decision-making.It also allows assessment of an interventional treatment against a comparator in clinical research.An example is the TIMI score, which has been used in clinical research and decisionmaking in the management of acute coronary syndromes. 22[25] If the prediction of an explosion in the number of dengue cases due to impending climate changes materialises, 26 the global community of clinicians caring for patients with severe dengue must be intelligently equipped.Identi cation of risk factors of death should culminate in the building of prediction models that can be deployed as software applications for clinical use at the bedside.Thus, we conducted a multicentre retrospective cohort study to build a predictive model that will predict death and estimate the probability of death in severe dengue cases.This model is an open-access, web-based 'bedside' prediction tool.

Methods
We conducted a multicentre retrospective cohort study with a total population sampling of patients with severe dengue who were admitted during the period 1st January 2017 until 31st December 2017.Our report is based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) 2015 guideline. 27e ve participating centres were Hospital Raja Perempuan Zainab II, Hospital Sungai Buloh, Hospital Kuala Lumpur, Hospital Tuanku Ampuan Rahimah, and Hospital Sultanah Aminah.Patients were selected for inclusion into the cohort if, 1) they were ≥15 years old 6 , 2) the case ful lled the study de nition of severe dengue, and 3) the presence of dengue infection was con rmed via laboratory con rmation by the presence of at least one of the following: positive non-structural protein 1 (NS1) antigen, or dengue RNA by reverse transcription-polymerase chain reaction (RT-PCR), or high-titre level of immunoglobulin G (IgG), or positive immunoglobulin M (IgM) from an admission serum sample.Patients with positive IgM-only immunology were also excluded if their clinical and laboratory blood results patterns were incongruous with dengue disease course or had an alternative nal diagnosis documented in their case notes.High-titre IgG detects acute dengue infection at titres >1:2560, which therefore is highly speci c for dengue infection and also identi es such infection as acute secondary dengue infection.Excluded patients were those who were pregnant and those who were admitted for transfer of care from non-participating centres.
The de nition of severe dengue in our study was based on World Health Organisation (WHO) 2009 de nition but with adaptation for this study. 28We de ned severe dengue by presence of any one of the following: 1) decompensated shock due to severe plasma leakage, 2) compensated shock due to severe plasma leakage, 3) respiratory compromise due to plasma leakage, 4) severe bleeding that required intervention, or 5) severe organ involvement such as acute kidney injury de ned by elevated serum creatinine above upper limit normal (according to gender-speci c levels), severe hepatitis, myocarditis, or encephalopathy.Decompensated shock was de ned by the presence of systolic blood pressure (SBP) less than 90 mmHg, mean arterial pressure (MAP) of less than 65 mmHg, or a drop in systolic blood pressure of more than 40 mmHg from a patient's known usual baseline readings.Compensated shock required signs of impaired peripheral perfusion, occurring in combination rather than singly, in the presence of systolic blood pressure of ≥90 mmHg.Severe hepatitis was de ned as aspartate transaminase (AST) level >1,000 IU/L or alanine transaminase (ALT) level >1,000 IU/L.Data extracted from records were demographic, co-morbidity, clinical parameters at the time of diagnosis of severe dengue, serial laboratory investigations at the time of diagnosis of severe dengue and at 24 hours later (with their timings), nadir and peak values of selected laboratory investigations, serial treatment, time data and outcome data.A data collection proforma was used to ensure the integrity of data.Time data, in the format of dates and times, were: fever onset, admission, the time of diagnosis of severe dengue, the time of defervescence (exact start of temperature persistently <38°C), the time of occurrence of the outcome and times of all laboratory investigations.Fever onset was determined by careful history-taking, speci ed to the best estimated time to the nearest hour.If missing it was imputed as midday (12:00 pm).As such, the day of illness may be determined for all variables.The time of onset of severe dengue was taken as the time of diagnosis of severe dengue.The temperature was monitored 4-hourly, and the temperature at diagnosis of severe dengue was determined from the temperature graph, extrapolated by the time of diagnosis of severe dengue.Clinical parameters included warning signs, which are clinical and/or laboratory features that predict the development of severe dengue. 28 calculated the required sample size on the basis of comparison of the area under a receiver-operating characteristic (AUROC) curve with a null hypothesis value of 0•5, i.e. no discriminatory power.A sample size of 21 deaths was needed for this study for an AUROC of 80%, at the con dence level of 95% and a power of 95%.

Statistical Methods
Our analyses were two-pronged, the rst involved analyses to describe our cohort, and the other, predictive model building.In order to describe the characteristics of our cohort clearly, we included variables which are related to laboratory investigations at 24 hours after the diagnosis of severe dengue, nadir and peak values of selected laboratory investigations, and variables related to management at 24-hourly time intervals.We described our cohort with descriptive and inferential analyses.Continuous variables were tested for normality with the Shapiro-Wilk test.As our data mostly had non-gaussian distribution, we used non-parametric analyses for data interrogation.Categorical variables were expressed as frequencies and percentages.Continuous variables were summarised as median and inter-quartile range (IQR).
Before proceeding to model-building, we performed univariable and standard multivariable logistic regression on all variables.The outcome or event of interest was any death during the hospital stay.As the study outcome uncommonly occurs with dengue and was expected to be similarly uncommon in our cohort, in the standard multivariable logistic regression, each variable was adjusted for age, gender, centre and time of onset of severe dengue.We then report the adjusted odds ratios in comparison to corresponding unadjusted values.This analysis was made on complete case basis.
Wherever applicable, we reported the statistical parameter with its 95% con dence interval.All tests of signi cance were 2-sided, and we took p-value <0•05 indicating statistical signi cance.All analyses were made using R, R Core Team (2016), R Foundation for Statistical Computing, Vienna, Austria. 29

Predictive model building
Variables used for predictive model building are called candidate predictors.As our study was focused on building a predictive model that would predict death in a severe dengue cohort, candidate predictors were collected at the onset of severe dengue.There were 28 candidate predictors which can be grouped into variables related to patient characteristics, warning signs in dengue, vital signs and laboratory (see Supplementary Table 1.Candidate Predictors for Model-building Grouped into Types).These candidate predictors were: age, body mass index (BMI), systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, white blood cell count, platelet count, haematocrit, aspartate transaminase, alanine transaminase, serum albumin, serum bicarbonate, serum lactate, serum creatinine, gender, presence of any comorbidity, presence of non-structural protein 1 antigen, presence of immunoglobulin M, presence of immunoglobulin G, persistent vomiting, persistent diarrhoea, presence of abdominal pain, presence of abdominal tenderness, presence of clinical uid accumulation, occurrence of non-severe bleeding, day of onset of severe dengue, severe dengue presentation (presented as severe dengue or not).Persistent vomiting and persistent diarrhoea were quanti ed as three or more episodes within the 24-hour interval before admission.Clinical uid accumulation refers to the presence of any ascites or pleural effusion by clinical or radiological detection.In predictive model-building, the outcome of interest was also any death during the hospital stay.
In building our predictive model, we performed variable selection, model search, and internal validation.In order to select truly independent predictors of death, all 28 candidate predictors were simultaneously analysed in a single model using a technique called penalised regression. 30The uncommon outcome (death) led to low events per variable, which prevented all 28 candidate predictors from being simultaneously analysed in a model using standard multivariable logistic regression approach -the number of events restricts the number of variables to be analysed within a model.
Our prediction model was built using multivariable penalised logistic regression using the glmnet and caret packages in R. 31,32 Penalised or regularised regression was chosen as the method is known to be able to perform variable selection in the condition of low events per variable very well. 30,33In penalised logistic regression, a penalty term is introduced such that non-signi cant regression coe cients undergo shrinkage towards zero while coe cients of signi cant or in uential variables would be retained as nonzero. 30This is in contrast to the usual method of stepwise logistic regression to select independent predictors whereby variables are chosen based on the magnitude of the p-value.Penalised regression also reduces over tting, which is usual with logistic regression. 30,33Over tting leads to the overoptimistic performance of a model.Another advantage of penalised regression is that it is one method that can be used to address multicollinearity. 30,34It encourages a grouping effect, where strongly correlated predictors tend to be in or out of the model together.
We performed a model search for the best performing model by tuning different values of model hyperparameters α and λ of penalised regression.The best performing model was chosen by the highest AUROC.We then extracted the corresponding values of α and λ to build the nal model.Model search and the nal model underwent repeated k-fold cross-validation, using 1000 repeats and ten-folds that generated 10,000 sample models.Repeated k-fold cross-validation is one form of internal validation method of a predictive model.In this technique, resampling with replacement was made: 100,000 datasets were generated.Then, a total of 10,000 models were trained or built on 90,000 datasets and validated on the remaining 10,000 datasets.Cross-validation further minimised over-optimistic performance and served as our internal validation method.Performance measures we reported were the area under the curve of the receiver-operating curve, sensitivity, speci city, accuracy, positive predictive value, negative predictive value, positive likelihood ratio, and negative likelihood ratio.We also reported the calibration curve of the model.We illustrated the overall study pipeline in Figure 1 and the internal validation step in Supplementary Figure 1.In prediction model building, missing data within the candidate predictors were imputed using k-nearest neighbours imputation during model-building via the caret package in R. 32 Apart from serum lactate and serum bicarbonate, which had 16% and 11•6% missing data respectively, all other candidate predictors had less than 10% missing data.Since the proportion of missing data were less than 20%, we kept all the candidate predictors.Continuous variables among candidate predictors were retained in their original scales.

Results
The study cohort was composed of 786 laboratory-con rmed severe dengue cases of whom 35  The median age was 30•3 years old, 57•6% were males, 31•3% had co-morbidity, and 13•4% had no warning signs.Among those without warning signs, 28•6% presented as severe dengue.Severe dengue was diagnosed upon presentation in 23•8% of patients, and patients who were in the febrile phase at admission made up 82•7% of the cohort.The proportion who were in the febrile phase at diagnosis of severe dengue were 74•3%; by subset according to type of severe dengue the proportions were: for decompensated shock 73%, compensated shock 75•4%, respiratory compromise due to uid accumulation 55%, severe bleeding 45•9%, severe hepatitis 58•8%, acute kidney impairment 77•5%, encephalopathy 68% and carditis 62•7%.The median duration of febrile phase was 4•8 days, the median onset of severe dengue was day 4•0, and median day of admission was on day 3•6.The median length of stay was 4 days, and the median length of illness was 7•6 days.Inotropes were administered in 16•2%, invasive and assisted ventilation in 17•7%, renal replacement therapy in 2•7%, and blood product transfusion in 15•3%.Demographic and clinical characteristics are given in Table 1, and laboratory variables are given in Table 2. Univariate and multivariable analysis of selected variables are given in Table 3 (full univariate and multivariable analysis of all study variables, including management-related variables at different time intervals are provided in Supplementary Table 2).The number of missing observations for each variable are provided in Supplementary Table 3. Table 1.Demographic and clinical variables.Table 2. Laboratory variables.Table 3. Univariate and multivariable analysis of selected variables for risk of death in severe dengue.
Our nal model was a penalised logistic regression with elastic net penalty model or an elastic net regression model, de ned by model hyperparameters α = 0•1 and λ = 0•01613622.The nal model comprised eight independent predictors of death among severe dengue cases: the presence of persistent diarrhoea, BMI, respiratory rate, levels of platelet count, serum bicarbonate, serum lactate, serum albumin, and aspartate transaminase.Since the predictors were timed at diagnosis of severe dengue, the model, therefore, is to be used from this time point onwards.The model regression coe cients and intercept are given in Figure 2. The model coe cients shown in Figure 2 are the regression coe cients of statistically signi cant variables that make up the nal predictive model, i.e. their coe cients remained non-zero after shrinkage.
We reported the performance of the model in terms of discrimination measures (Table 4) and calibration.The nal model has a high discrimination performance as represented by its AUROC of 89•6% [95% CI: 89•5 -89 •7].In comparison, the raw standard non-cross-validated logistic regression model has an overoptimistic AUROC of 91•9% (Figure 3).The nal penalised regression model has a very high sensitivity of 99•62% [95% CI: 99•61 -99•63] but a modest speci city of 23•6% [95% CI: 23•2 -24•0].Nonetheless, the model has a very high accuracy of 96•23% [95% CI: 96•21-96•26].Calibration refers to how closely the predicted probabilities of death agrees with the observed probabilities.Calibration of the probabilities was assessed by plotting observed probabilities versus predicted probabilities: a calibration curve.The calibration curve of the model showed excellent agreement between prediction and actual deaths when the probability of death is 60% and beyond, whilst below 60%, the agreement was less strong (Figure 4).
During variable selection, centre and day of onset of severe dengue were found to be non-signi cant (coe cients underwent shrinkage to zero).The prediction model may be accessed at the following website: www.saifulsafuan.com/REPROSED2017E2.Table 4. Performance measures of the nal model, penalised logistic regression with elastic net penalty model.

Discussion
We have developed a predictive model whichis available as an open-access web-based tool that has performed well with high certainty with an AUROC 89•6% [95% CI: 89•5-89•7] and a high accuracy of 96.23% [95% CI: 96.21-96.26].The model comprises 8 independent predictors, incorporating demographic, clinical, and laboratory variables.The model was developed through regression search for in uential variables of the disease.We believe the model re ects all important underlying pathophysiological aspects of the disease represented by these variables and therefore led to its high performance.The model has a very high sensitivity of 99•6% but a modest speci city of 23•6%.In view that such a model is intended to capture and prevent potential deaths, and the overall high AUROC, the model would be very useful in clinical practice.Misidenti cation of a non-fatal case is an acceptable compromise compared to missing a fatal case altogether.A possible explanation for the modest speci city is the fact that the real underlying outcome-determining pathophysiological mechanisms in dengue has not been clearly elucidated and only biomarkers involved in these mechanisms could improve the speci city of any model.
We developed this model to be employed at the time of severe dengue diagnosis, which may reasonably be assumed as the time of onset of severe dengue.This is likely the time when outcome-determining pathophysiological processes become critical.This is the earliest and most appropriate choice of time to prognosticate a patient in terms of death in severe dengue.We believe the selection of the point of time of prediction is crucial in the development of a dengue death prediction model.An earlier time point would be too early where pathophysiological processes may not have reached outcome-determining signi cance.Selection of a later time point would be too late for the prediction model to be bene cial.With that in mind and the fact that the model performed well, we postulate that investigating processes represented by these variables could elucidate the pathophysiology of dengue with better clarity.We also believe that the model may be used to track the progression of a patient with severe dengue through the course of illness, assists in guiding prognostication and decision-making.
Making predictions in dengue research has gained momentum.[25] Huang et al. studied all patients with dengue that included 34 deaths and identi ed ve independent predictors of death: age >64 years old, diabetes mellitus, systolic blood pressure <90 mmHg, chronically bedridden, and haemoptysis. 23However, this study used a scoring method.Risk scoring is not without weaknesses and has been deemed to have serious problems. 35Risk scoring involves converting continuous predictors into categorical predictors.The conversion results in loss of granularity of information contained within continuous predictors. 36Moreover, the presence of any of the rst three predictors only occurred in 162 (20•6%) cases of our cohort and the combination of all three in only 2 cases.It is imperative in clinical prediction that a predictor is su ciently prevalent for the achievement of reasonable accuracy. 37-Sani et al. examined severe dengue cases and predicted death at the onset of severe dengue similar to the current study but at a single centre. 24That study however, had only 20 events (deaths). 24Building a predictive model to predict death among severe dengue cases is challenging as dengue death, the event or outcome of interest, is actually uncommon -in Malaysia it is just above 0.2% of all dengue cases. 6his poses a huge problem in that, in the usual approach of statistical modelling using multivariable logistic regression, 5-10 events per candidate predictor variable (EPV) are required. 38This ratio dictates the number of candidate predictors that may be simultaneously analysed in the multivariable model.When the EPV is less than this, the number of candidate predictors that may be simultaneously analysed to identify which among them are truly independent predictors is limited.Thus, Md-Sani et al. employed the approach of using adjustment or controlling variables instead of assessing all candidate predictors simultaneously in one model.The present study is larger, multicentred and used a different analysis technique which addressed over tting, multicollinearity and the low number of events per variable.
Pinto et al. which had a large cohort with 61 severe dengue deaths, built a simple predictive model comprising only four categorical predictors: age (binary, cutoff age 55), haematuria, gastrointestinal bleeding and thrombocytopaenia (binary, cutoff platelet count 20,000 cells/mm3). 25In our study, we did not speci cally identify the source of bleeding.However, assuming the variable warning sign of spontaneous bleeding tendencies and the variable severe bleeding represent haematuria and gastrointestinal bleeding, respectively, the presence of any of Pinto et al.'s predictors occurred only in 259 (33%) cases in our cohort.The presence of all four predictors occurred only in 3 cases.Therefore, these models may not be adequate to prognosticate death in severe dengue.Our model kept variables in their original continuous attribute, and persistent diarrhoea was the only categorical variable.Persistent diarrhoea occurred in 38•5% of our cohort, which is more prevalent than any of the predictors of Huang et al. and Pinto et al. if applied to our cohort.Thus, because of this and the predominantly continuous attribute of predictors in our model, we obtained a higher performance accuracy.Additionally, Pinto et al. used the WHO 1997 dengue classi cation instead of the WHO 2009 schema which Malaysia has adopted in clinical practice. 6,280][41][42][43][44][45][46] Our study is the rst modelling study, based on the latest and widely adopted WHO 2009 classi cation scheme, to model prediction of death in severe dengue cases.As mentioned above, there have been many studies that built models for identi cation of dengue and strati cation of severity in dengue.Our model complements this and completes the prediction aspect of clinical management.
The study additionally documented another interesting nding.We found that almost three-quarters of the cases were still in the febrile phase (study de nition temperature >38°C) at diagnosis of severe dengue.This nding supports similar documentation in a previous study. 24A similar proportion was found for those who had shock.This nding is different from what majority of guidelines have stated where shock only occurs during the critical phase, i.e. upon defervescence and later. 6,28Current clinical practice only recognises that severe dengue would occur only during the critical phase, thereby missing severe dengue which could occur earlier in the febrile phase.Our study provided quanti ed evidence that severe dengue could occur early in the febrile phase and clinicians should be vigilant of this fact in order to prevent deaths.
The limitation of our study is its retrospective design in which missing data was inevitable.However, only variables of utmost importance were included as candidate predictors for variable selection.We included serum bicarbonate and serum lactate though these had higher missing proportions (11•6 and 16%, respectively) as we believe they play essential roles in determining not only progression to severe disease but also death. 24,47,48Even though external validation of the model was not performed, we believe that the repeated k-fold cross-validation algorithm we employed had ensured the robustness of the model for unseen data.A nal limitation is that though our model may save costs due to its accessibility, it will require additional laboratory-related resources.Nevertheless, any severe dengue cases should be treated in a setting with adequate resources to implement evidence-based clinical practice, and the laboratory predictors in our model are commercially available.Zakaria et al. demonstrated that the WHO 2009 dengue severity strati cation scheme classi es more patients (4•6%) into the most severe form as compared to the previous WHO 1997 scheme (0•7%). 49They highlighted that this might pose a signi cant impact on hospital resources.Our model can potentially prioritise patients to local resources based on their probability of death.
In conclusion, we have developed a dengue death prediction model comprising clinical and laboratory data and deployed an open-access web-based tool for any centre to utilise for local validation.The ndings from this study would be valuable to the global community of clinicians who treat dengue, hopefully paving better and tailored clinical decision-making and resource management.In terms of research, the tool may be a useful yardstick, similar to how TIMI and APACHE are useful to cardiovascular and critical care medicine, respectively. 22,50,51

Figure 2 .
Figure 2. Regression coe cients of the nal model, penalised logistic regression with elastic net penalty model.

Figure 3 .
Figure 3. Receiver-operating characteristic curves of the raw model (standard non-cross-validated logistic regression model) vs the nal model (elastic net regression model with repeated K-fold cross-validation).

Figure 4 .
Figure 4. Calibration curve of the nal model, penalised logistic regression with elastic net penalty model.

Table 1 .
Demographic and clinical variables a .
a Categorical variables are summarised as n(%).Continuous variables are represented as median (IQR).bUnivariate logistic regression was used to compare differences between groups.Analysis was made on complete case basis.

Table 2 .
Laboratory variables a .

Table 3 .
Univariate and multivariable analysis of selected variables for risk of death in severe dengue.

Table 3 .
Univariate and multivariable analysis of selected variables for risk of death in severe dengue.

Table 4 .
Performance measures of the nal model, penalised logistic regression with elastic net penalty model.Abbreviations: CI, con dence interval; AUROC, area under receiver operating characteristic curve.