Interrater Agreement of The Copenhagen Triage Algorithm

Systematic triage is performed in the Emergency Department (ED) to assess the urgency of care for each patient. The Copenhagen Triage Algorithm (CTA) is a newly developed, evidence-based triage system, however the interrater agreement remains unknown. Method This was a prospective cohort study. The collection of data was conducted in the three sections (Acute/Cardiology, Medicine and Surgery) of the ED of Herlev Hospital. Patients were assessed independently by two different nurses using CTA. The interrater variability of CTA was calculated using Fleiss kappa. The analysis was stratified according to less or more than 2 years of ED experience. Results A total of 110 patients were included of which 10 were excluded due to incomplete data. The raters agreed on triage category 80 % of the time corresponding to a kappa value of 0.70 (95% confidence interval 0.57-0.83). Stratified on ED sections, the agreement was 83 % in the Acute/Cardiology section corresponding to a kappa value of 0.73 (0.55-0.91), 79 % in the Medicine section corresponding to a kappa value of 0.64 (0.39-0.89) and 0.56 % in the Surgery section corresponding to a kappa value of 0.56 (0.21-0.90). The experienced raters had an interrater agreement of 0.73 (0.56-0.90), while the less experienced raters had an agreement of 0.76, (0.28-1.24). A substantial interrater agreement was found the only internationally method, a clinical assessment as a central part of the system. in ESI the clinical assessment precedes a series of questions from flowcharts. (18) The interrater agreement on live patient triage using ESI has shown results close to what we show in this study with substantial agreement ( k =0.78). (15) These similar results were in spite of the difference in experience between the raters in this study and the ESI study in which most of the triage personnel had prior experience using ESI and received a three-hour intensive training course before data collection began. (15) Considering the limited experience with CTA prior to the study, the findings of an overall substantial agreement in our study suggests that CTA is relatively easy to use and will generate consistent results even when the user lacks experience with the method. Higher interrater levels of agreement would be expected if the study was repeated after the CTA had been applied as the standard triage method in the ED for a longer period.


Introduction
Triage is used to prioritize patients in the Emergency Department (ED). The triage tool is based on perceived urgency in an effort to give the best care and lessen the effects of crowding. (1,2) There is no consensus regarding the optimal method of triage, but most of the applied models are built on the patient's chief complaints and vital signs. (3) As triage requires time and resources it is problematic that most triage methods are poorly validated. (1,3) Most of the systems have been developed based on expert opinion rather than data from large prospective cohorts. (1,3) An important feature of a triage system is reliability and different nurses should obtain similar triage categories. The reliability of triage has traditionally been measured using interrater agreement, and studies on contemporary triage systems have shown interrater agreement rates with kappa values varying between 0.20 and 0.87. (1,3) Recently, The Copenhagen Triage Algorithm (CTA) has been developed with the intent of creating an evidence-based triage model. (4,5) An example of the chart used for the CTA triage in this study and a brief description of its use are shown in figure 1. CTA classifies patients based on vital signs and a clinical assessment by the ED nurse and has been shown to be a stronger predictor of mortality than a well-known triage system, Adaptive Process Triage (ADAPT), in a large randomized trial. (4,5) However, the interrater agreement for this model is unknown.
The objective of this study was to examine the interrater agreement of the CTA in the ED.

Study design and setting
This was a prospective observational cohort study. The collection of data was conducted in the ED at Herlev Hospital on arbitrarily selected dates from June 2018 through March 2019.
This allowed for inclusion throughout the year in order to obtain a representative patient cohort. Herlev Hospital is a 24-hour secondary care unit offering emergency, level-2 trauma, medical, neurological and surgical care. The hospital provides medical care for 425.000 citizens living in the coverage area and has about 150.000 annual admissions to the ED. (6,7). Patients admitted directly to specific departments in the hospital do not pass through the ED, and this includes paediatric patients, gynaecological and obstetric patients, and trauma patients admitted to tertiary centres of the region.
Patients are admitted to one of three sections (Acute/Cardiology, Medicine, and Surgery,) in the ED according to their primary symptoms upon arrival, and data collection was alternating between sections. The Acute/Cardiology section receives trauma patients, medical and surgical patients triaged as 'Red' (level 1, resuscitation) as well as patients with orthopaedic or cardiac complaints. The Medicine section treats patients within the field of internal medicine, apart from cardiology, gastroenterology, oncology, and haematology. (8) The Surgery section receives patients with gastric or urological complaints. (8)

Selection of participants
All patients above 16 years of age admitted to the ED on the days selected for data collection were included. Patients with minor injuries triaged as blue by both CTA and ADAPT were excluded from the study, since their vital signs were not measured. Patients who were not triaged by two different ED personnel using CTA were also excluded from the study.
On study days patients admitted to the specific section were assessed once using the conventional ADAPT method and twice by different members of the ED triage personnel using CTA. The triage categories of CTA as well as the maximum waiting time to an assessment by a doctor are identical to those of ADAPT. The red category was the most urgent, while the green category was the least urgent, see figure 1. To ensure the clinical presentation and vital signs of the patient did not change between the assessments, the ED personnel were performing the CTA assessment at the same time blinded to the other's triage.
At the beginning of each shift in the study period, the ED personnel received a brief instruction in CTA and how the triage level should be determined, as well as a general presentation of the study. Apart from this, the nurses did not receive any further training in the use of CTA prior to data collection.

Triage staff
The ED personnel performing the triage were nurses, social-and health service assistants (SOSU) and student nurses, in accordance with the usual clinical practice. We classified the raters as either 'experienced' or 'not experienced', where an "Experienced rater" had at least 2 years of work experience in the ED.

Ethics approval
The study was performed without the need for a formal ethics approval in accordance with Danish law. The triage nurses gave consent to participate, as did patients.

Statistics
We hypothesized that the kappa-value of the interrater agreement would be equal to or higher than 0.7. Prior to the study, we performed a power calculation using the "Power4Cates" function in R. With a power of 80 % and a significance level of 0.05, a population of at least 99 patients was required. We assumed that the distribution of patients on the four triage levels from the most to least urgent category would be 5 %, 15 %, 40 %, 40 %, in accordance with previous studies of the CTA. (5) The interrater agreement was calculated using Fleiss' kappa. We calculated a global kappa and kappa values for each ED section. We also calculated kappa values according to the level of experience of the raters. Kappa levels were interpreted according to Altman's definitions, shown in Supplementary Figure 1. (9,10) The data from this study were analyzed using R version 3.6.1.

Results
During the study period 110 patients were included. Of these, 10 had incomplete data and were excluded, thus a total of 100 patients were included in the study (table 1). A team of 51 ED nurses, SOSU's and student nurses performed the assessment of these, 31 (61%) were classified as "experienced raters" (table 2).
The most commonly used category was "green" with 41.5 % of the final triages, the "yellow", "orange" and "red" category were chosen in respectively 37.5 %, 16 % and 5 % of the triages.
The changes made from primary to final triage as a result of the clinical assessment are presented in Figure 2. The final triage level differed from the primary triage in 52 % of the cases. Using the clinical assessment, the triage level was changed to a more urgent category in 40 % of the triages and changed to a less urgent category in 12 %.
The raters agreed on the CTA triage level 80 % of the time. The interrater agreement in all ED teams was k=0.70 (95 % Confidence Interval (CI) 0.57-0.83). The highest agreement was found among the raters from the Acute/Cardiology section of the ED, who agreed 83 % of the time, equivalent to a kappa value of k=0.73 (95 % CI 0.55-0.91). In the Medicine section the agreement between raters was 79 % and the corresponding kappa was k=0.64 (95 % CI 0.39-0.89). Agreement between the raters was lowest in the triages performed in the Surgery section of the ED, who agreed 71 % of the time with a value of k=0.56 (95 % CI 0.21-0.90).
The rate of agreements and disagreements distributed at the different triage levels are shown in Table 3. Agreements on changes of the triage level from primary to final triage based on the clinical assessment are shown in table 4.
A sub analysis was performed to investigate whether the raters' experience affected the agreement. The experienced raters performed the triage of 60 cases and had an agreement of k=0.73 (95% CI 0.56-0.90), while the less experienced raters performed the triage of 12 cases with an agreement of k=0.76 (95% CI 0.28-1.24). In 28 of the cases where an experienced and a less experienced rater triaged the same patient agreements were lower k=0.54 (95% CI 0.28-0.81).

Discussion
This is the first study examining the interrater agreement of CTA. We found that the CTA has a good overall agreement. (9) There is no standardized procedure regarding the validation of triage methods. overtriage may stretch resources, while undertriage may increase morbidity and mortality due to longer waiting time to treatment. (16,26,27) In comparison with ADAPT, CTA has been shown to triage at a significantly lower urgency level than ADAPT without it having a negative effect on the examined patient outcomes. (5)

Strengths and limitations
The overall distribution of patients at the four urgency levels in this study is similar to the one found by a larger prospective trial comparing CTA to ADAPT, indicating a representative sample was collected in our study. The ED personnel had limited experience with the use of CTA and received only a brief instruction prior to using the triage system. Because the data was collected alternately in the three sections of the ED and sporadically during the study period, the personnel were not given an opportunity of increasing their experience using the system over time. Their lack of experience with the method may have reduced the level of agreement, and it is possible that the interrater agreement would increase over time, if the CTA was implemented as the standard triage method in the ED.

Conclusion
The Copenhagen Triage Algorithm is a valid triage system with substantial inter-rater agreement.

Ethics approval and consent to participate
The study was performed without the need for a formal ethics approval in accordance with Danish law. The triage nurses gave consent to participate, as did patients.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare, that they have no competing interests.

Not applicable
Author's contributions JIHB was the main author of the manuscript and contributed to the collection of and analyzation of data.
RBH was a major contributor in writing the manuscript and analysing the data as well as planning the study.
TK performed the calculations of kappa.
ML and LR contributed to the planning of the data collection.
LSR contributed substantially to the revision of the work.
MS was a major contributor in writing the manuscript as well as planning the study. MS also contributed to the collection of and analyzation of data.
KI had the main idea of the project and contributed to the planning of the study.
All the authors read and approved the final manuscript. Acute/Cardiology, Medicine and Surgery refer to the respective ED sections. The experienced raters were defined by at least 2 years of working in the ED. Some of the raters performed triages in several ED teams, and are therefore included in more than one ED team in the table. Orange (Rater A) 10 2

Red (Rater A) 4
This    Figure 1 -The Copenhagen Triage Algorithm. The patients are initially classified using a vital sign scoring system, but the assigned category suggested by the score can be altered based on a clinical assessment. CTA allows for a two-class upgrade or a one-class downgrade of the triage category, when the ED nurse assesses that the initial category is not in line with the clinical state of the patient.