Comparison of the Discrimination Capacity of the ROX and HACOR Scales to Predict High-ow Nasal Cannula Failure in Patients With SARS-CoV-2 Pneumonia

The pandemic of SARSCov2 infection has created a challenge in health services worldwide. Some scales have been applied to evaluate the risk of intubation, such as the ROX and HACOR. The objective of this study is to compare the predictive capacity of the HACOR scale and the ROX index and dene the optimal cut-off points. Methods Study of diagnostic tests based on a retrospective cohort. Composite outcome was the proportion of patients that needed orotracheal intubation or died of COVID19 pneumonia. Discrimination capacity was compared by the area under the curve of each of the two scales and the optimal cut-off point was determined using the Liu method. Results composite end result The discrimination capacity was similar the two with an AUROC of 0.71 and 0.72 for the HACOR scale for the ROX index, respectively. The optimal cut-off point for the ROX index was 5.62 (sensitivity 62% specicity 65%), while the optimal cut-off point for the HACOR scale was 5.5 (sensitivity 66% specicity 65%).


Introduction
It is well known that about 16% of patients with severe COVID-19 pneumonia required intensive care (ICU) (1). Among them the rate of invasive mechanical ventilation (IMV) is between 71% and 90% (2), with mortality close to 50% (3)(4). There are different ways to deliver oxygen in patients with severe pneumonia such as high-ow nasal cannula (HFNC), standard oxygen therapy delivered through a face mask, or noninvasive positive-pressure ventilation (NIPPV). Although there was no signi cant difference between the requirement for orotracheal intubation in the different interventions, the number of ventilatorfree days at day 28 was signi cantly higher in the high-ow-oxygen group (5). Therefore, the use of HFNC associated with the prone position (6)(7) is currently recommended for severe COVID-19 pneumonia as rst-line treatment, since some patients can improve oxygenation and fatigue and avoid invasive mechanical ventilation with these strategies (8).
However, it is also known that the delay initiation of invasive mechanical ventilation (IMV) in patients with severe pneumonia leads to an increase in mortality (9), that is why it is so important to identify in a timely manner the patients in whom this therapy may fail. To do this, scales have been created to predict failure to non-invasive mechanical ventilation strategies (NIMV), including the ROX index, which consists of a relationship between arterial oxygen saturation / fraction of inspired oxygen ratio (SpO2/FIO2) and respiratory frequency. It was validated in 2016 (10), as an index for predicting HFNC therapy failure in patients with severe pneumonia. Roca et al. (5) found that a score of 4.88 after 2 hours of HFNC therapy predicts failure to HFNC (AUC of 0.74). In addition, studies have been done that validate its usefulness in patients with COVID19 pneumonia (14,15,16,17). However, the main problem with the ROX scale lies in the delay in the initiation of IMV in people who require it.
In 2017, Jun Duan et al (8) created a tool called HACOR, an acronym for heart rate, acidosis, state of consciousness, oxygenation, and respiratory rate, to predict NIMV failure. In 2020, Carrilo et al (10) validated this tool with 2711 patients, showing that the HACOR scale accurately predicted NIV failure in the rst hour. A cut-off value of 8 points at 1 h of HFNC beginning had a AUC greater than 0.9 for predicting NIV failure in patients with pneumonia and adult respiratory distress syndrome (ARDS). To our knowledge, the HACOR score has not been validated in patients with COVID19.
Although these scales have been validated in patients with severe pneumonia, we do not know if the HACOR scale is useful in patients with COVID19 and if its discrimination capacity is better than the ROX scale. The objective of this study is to compare the discrimination capacity between the HACOR and ROX scales to predict the risk of invasive mechanical ventilation in patients with severe SARSCOV2 pneumonia who are with HFNC strategy in the resuscitation unit of a hospital in Colombia.

Trial oversight
Study of diagnostic tests based on a retrospective cohort. We conducted the study in the emergency unit of a high complexity University Hospital in Bogota, Colombia. The study protocol was approved by the "Research and Institutional Ethics Committee of the Faculty of Medicine of the Ponti cia Universidad Javeriana, and the Hospital Universitario San Ignacio". Since the study was based on data collection, patient consent was not required. This waiver is given in agreement with the Helsinki declaration and resolution number 008430 of 1993 issued by the Ministry of Health from the Republic of Colombia.
The rst investigator was responsible for searching the hospital's database, while the second investigator ensured adherence to the protocol. Tests were conducted in accordance with principles of the Helsinki Declaration. Authors assume responsibility for the integrity of the data and its analysis.

Patients
Total adult population identi ed with viral pneumonia by SARS CoV2 that required HFNC in the emergency department of the San Ignacio Hospital, between August and December of 2020 were eligible for enrollment. Data were taken from patients older than 18 years of age who had the diagnosis of SARS CoV2, con rmed by polymerase chain reaction (PCR). Patients with SARS CoV2 pneumonia who had required immediate ventilatory support with invasive mechanical ventilation and patients with severe or refractory agitation to sedation, massive bronchial aspiration, inability to manage bronchial secretions were excluded.

Trial Procedures
The decision to initiate a HFNC was made by the emergency physician or internist attending the patient, based on the following criteria: The clinical presentation of respiratory failure (such as the use of accessory muscles, breathing rate greater than 20 per minute, paradoxical abdominal movement, desaturation despite nasal cannula or venturi therapy), partial pressure of arterial oxygen (PaO2) less than < 60 mmHg or the PaO2/inspired fraction oxygen ratio < 300 with supplemental oxygen.
HFNC was managed by respiratory therapist professionals and patient's emergency physicians and internists, depending on the case. Masks, heated humidi ers, air-oxygen blenders, and cannulas (i.e., consumable materials) were owned by the Hospital. The device used as high-ow therapy was the HFNC AIRVO™ 2 Opti ow™ and the parameters at the beginning of the therapy were established according to the patient's last partial PaO2/FiO2 to achieve a SaO2 greater than 90%. Once respiratory failure improved, progressive decrease in the parameters of the HFNC were initiated. This was done when the patient had a breathing rate of fewer than 25 breaths per minute, without signs of muscle fatigue, had a SaO2 greater than 90%, and PaO2/FiO2 in ascent concerning that taken before the beginning of the HFNC. Oxygen Flow was always lowered initially and, according to the patient's tolerance, the inspired fraction of oxygen parameter was subsequently lowered. All patients underwent gasometric monitoring before the onset of HFNC. After 2 hours of HFNC therapy onset, vital signs, arterial blood gases and the HACOR and ROX scales were performed.

Outcomes
Primary outcome was HFNC therapy failure, including need for mechanical ventilation onset and death associated with COVID-19 pneumonia.

Statistical analysis
Demographic and epidemiological characteristics of patients diagnosed with COVID-19 pneumonia who required HFNC therapy were described.
According to the recommendations of the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis statement (TRIPOD) (11), the minimum sample size was 100 events and 100 not events, so a minimum of 100 failures to the HFNC was sought. The baseline characteristics were compared between the treatment groups with the use of chi-square and Wilcoxon tests.
The discrimination capacity of the HACOR and ROX scales was evaluated using the area under the receiver operating characteristic curve (ROC curve). The comparison of both scales was carried out by comparing the areas under the curve of the two scales. A secondary analysis of the ROX and HACOR scales was nally performed to predict failure of HFNC.
Finally, the Liu method was performed to identify the cut-off point with the best performance of the HACOR and ROX scales (12).

Patient Characteristics
A total of 245 patients were included in the study. Most of them were men (65.3%). Mean (± SD) age of the patients was 62.53 (± 13.83) years. A total of 93 patients (37.96%) were 18 to 60 years of age and 152 (62.04%) were 60 years of age or older. 152 (62.04%) patients had the nal composite outcome of HFNC failure (requirement of orotracheal intubation or death by COVID19). Death associated with SARSCov2 infection occurred in 72 patients (29.39%). Table 1 shows key characteristics of the patients, strati ed into two groups according to whether the patient had the nal composite outcome. Interestingly, there were clinically signi cant imbalances in baseline characteristics with respect to prior history of hypertension, atrial brillation, and chronic kidney disease (CKD) between both groups.  Table 2). The 2-hour ROX index was 6.63 and 5.61 for patients who did not fail and failed to HFNC, respectively (P < 0.0001). The HACOR score was 4.45 and 7.14 for patients who did not fail and failed to HFNC at 2 hours, respectively (P < 0.0001). The mean time to failure of the HFNC was 1.4 days (± 3.8).  * There were no signi cant differences (p < 0.05) between the study groups. BMI denotes Body mass index, CKD chronic kidney disease, COPD chronic obstructive pulmonary disease. HFNC High-ow nasal cannula, SaO2 blood-oxygen saturation, PaO2 Arterial partial pressure of oxygen, PaCO2 Partial pressure of carbon dioxide, Fio2 fraction of inspired oxygen. † the body-mass index is the weight in kilograms divided by the square of the height in meters.
In the univariate and multivariate analysis, the HACOR scale was the only one that maintained statistical signi cance. However, when the discrimination capacity of the HACOR and ROX scales was evaluated (Fig. 1), it was found that the AUC was similar for the two scales without signi cant differences (ROX AUROC 0.72 vs HACOR AUROC 0.71).
Finally, an estimate of the best cut-off point was made using the Liu method to predict the failure of the HNFC, nding that the value for HACOR scale was 5.5 with a sensitivity of 66% and a speci city of 65% (AUC for the cut-off point of 0.66) while the cut-off point for ROX scale was 5.625 with a sensitivity of 62% and a speci city of 65% (AUC for the cut-off point of 0.64).

Discussion
In this study, we investigated the discrimination capacity of the ROX index and the HACOR scale to predict high-ow nasal cannula failure in patients with COVID19 pneumonia. Our data suggest that the discrimination capacity of the 2 scales is similar despite the scales evaluating different variables.
In our study we found that 140 (57,14%) patients required OTI and 152 (62.04%) had the nal composite outcome of HFNC failure (requirement of orotracheal intubation or death by COVID19). We believe that the differences in study populations may lead to variation in ndings. Compared to the report by Roca et al., our sample analyzed had a higher median age (64 vs 53 years). Also, a higher proportion of our patients on HFNC failure had CKD (13.8% vs 4.5%). Secondly, our study found that patients with atrial brillation had a worse outcome. This variable was not evaluated by Roca et al. Furthermore, a later study did nd that a heart rate greater than 90 beats per minute was associated with a higher risk of failure of HFNC (19).
The discriminative capacity of both is similar (ROX AUROC 0.72 vs HACOR AUROC 0.71). The discriminative capacity of the ROX index is similar to those observed in other studies (16,17,18). As for the HACOR scale, so far, we do not know that it has been evaluated in patients with COVID19 infection. In our study, the discriminative capacity of the HACOR scale was similar to that reported by Innocenti et al (AUROC of 0.68). (20) but it was lower than that found by Duan et al, (AUROC of 0.89) (9) and Carrillo et al (AUROC of 0.88) (11). We believe that these differences occurred because in the studies by Duan et al, and Carrillo et al. Patients with respiratory failure secondary to causes other than COVID19 infection, as well as patients with hypercapnic respiratory failure, were taken. These ndings may also be associated because in our study the discriminative capacity of the HACOR scale was taken based on a composite result of the need for orotracheal intubation and death, similar to the study by Innocenti et al. but different from the Duan study where only the need for orotracheal intubation was taken as a result. However, more studies are needed to carefully evaluate these hypotheses.
In our study, the best cut-off point for the ROX index to determine the success of HFNC was 5.62 (sensitivity 62% speci city 65%) and not 4.88, as previously established by Roca et al. (10). On the other hand, the highest cut-off point of the HACOR scale to de ne the success of the HFNC was 5.5 (sensitivity 66%, speci city 65%), similar to that previously reported in the literature (16,17,18,20).
Interestingly, the HACOR scale was statistically signi cant in the multivariate analysis when comparing variables that are not included in the calculation of this scale, unlike the ROX index, which did not achieve statistical signi cance. We believe that this nding is due to the fact that the HACOR scale uses a greater number of variables that reached statistical signi cance such as the Glasgow scale and that are not part of the ROX index. However, the discrimination capacity of the HACOR and ROX scales is similar. Given that the outcome we seek to avoid is mortality, we suggest using both scales to de ne the failure of the HFNC, which would provide greater safety for the patient, although this would imply a closer monitoring of the person.
Although the predictive capacity of both scales is adequate, we believe that it was not so high that it could be used as the sole criterion to predict failure of the HFNC. Therefore, we consider it necessary to add to these tools clinical variables that were also associated with the outcome of interest, such as a history of atrial brillation, hypertension, and chronic kidney disease. These variables could lead to the development of new models that allow a better prediction of the failure of the HFNC. New studies will be necessary to propose and evaluate these models.
Within the strengths of the study, this is the rst study to compare the ability to predict cannula failure of the ROX and HACOR scales. Furthermore, it is one of the individual studies with the largest number of patients evaluated with HFNC and is the rst to describe the discriminative capacity of the HACOR scale in patients with COVID19 pneumonia. Finally, we did not have data loss and we were able to count the outcomes of all the patients. Among the limitations of the study is the fact that the criteria to de ne the OTI requirement depended directly on the treating physician. However, the criteria for therapeutic failure to HFNC are clearly de ned in our institution and these are part of a management protocol, which is why we believe that the decision for mechanical ventilation is similar among medical personnel.

Conclusion
The HACOR scale and the ROX index have a moderate predictive capacity to predict failures to the HFNC strategy. They can be used in conjunction with other clinical variables to de ne which patients may require invasive mechanical ventilation. Figure 1 AUROC for HACOR and ROX scores in predicting the nal composite outcome (requirement of orotracheal intubation or death by COVID19) after 2 hours of High ow nasal cannula.