Effect of Problem-based Learning on Agreement of Severity Classication Results of Triage Nurses in the Emergency Department

Background Problem-based learning is a learner-led learning method that helps improve critical thinking, problem solving skills, and knowledge. It is necessary to conrm whether it can help to agree the severity classication results among nurses through problem-based learning. Methods This study had a single-group time series design to investigate the effect of problem-based learning led by triage nurses on the agreement of Korean Triage and Acuity Scale classication results for patients who visited the emergency department. We extracted 300 patients each in May and August 2018 before problem-based learning began and 300 patients each in May and August 2019 after problem-based learning.


Abstract Background
Problem-based learning is a learner-led learning method that helps improve critical thinking, problem solving skills, and knowledge. It is necessary to con rm whether it can help to agree the severity classi cation results among nurses through problem-based learning.

Methods
This study had a single-group time series design to investigate the effect of problem-based learning led by triage nurses on the agreement of Korean Triage and Acuity Scale classi cation results for patients who visited the emergency department. We extracted 300 patients each in May and August 2018 before problem-based learning began and 300 patients each in May and August 2019 after problem-based learning.

Results
After problem-based learning, the length of emergency department stay decreased about 30 minutes, although the decrease was not statistically signi cant (p=.172). However, self-e cacy for the classi cation of emergency patients in triage nurses and weighted kappa coe cients were improved (p<.001).

Conclusion
In this study, problem-based learning led by triage nurses improved the inter-rater agreement of Korean Triage and Acuity Scale classi cation results and self-e cacy of triage nurses. Therefore, problem-based learning led by triage nurses can contribute to patient safety in the emergency department by enhancing the expertise of triage nurses and increasing the accuracy of triage classi cation.

Background
According to statistics released by the National Emergency Medical Center in 2019, more than 10 million patients visited the emergency department (ED) during 2018, which was more than 20% higher than that in 2008 as a result, ED overcrowding is increasing [1]. As the number of patients who visited the ED increases, ED overcrowding has been pointed out as one of the most important problems [2]. One of the most important ways to relieve ED overcrowding is by reducing the length of stay of emergency patients [3,4]. The ED stay time can be reduced by appropriately classifying the severity of emergency through accurate initial assessment of patients who visit the ED [5,6].
Classifying the severity appropriately through initial assessment of patients visiting the ED is a process of determining where to receive treatment [7]. Accurate patient classi cation is essential for identifying patients who do not require prior intervention and optimizing emergency medical resources for immediate treatment [8]. Therefore, triage nurses in charge of initial assessment in the ED should quickly and appropriately categorize the severity of patients visiting the ED. This is one of the most important tasks of ED triage nurses [9].
For initial assessment and classi cation tool of emergency patients in Korea, various classi cation tools were used in each hospital until recently [10]. In 2012, the Korean Triage and Acuity Scale (KTAS) Committee a liated with the Korean Society of Emergency Medicine developed KTAS as a Korean type emergency patient classi cation tool based on Canadian Triage and Acuity Scale (CTAS), Canada's vestage classi cation tool, by modifying and supplementing it to suit the domestic situation [11]. Since 2016, the law on emergency medical care has been changed. All EDs have been using the KTAS to classify patients visiting the ED [12]. This severity classi cation is performed by nurses in the United States [12]. In most EDs in Korea, nurses perform patient classi cation through initial evaluation.
However, there is a difference in the degree of agreement among triage nurses according to ED experience or clinical experience of nurses who perform the evaluation. Objective evaluation may be di cult in psychological and mental health areas, although respiratory diseases could be objectively evaluated [9]. In addition, the severity classi cation may be different because different factors determine the degree of urgency [13]. Even if there are clear criteria for classi cation, there might be differences in the severity classi cation results among evaluators. In one study determining the agreement between nurses and emergency doctors [14], the Kappa coe cient was 0.659. In addition, there was a signi cant difference in classi cation between male and female nurses (p = 0.003) [14]. Another study has assessed the degree of agreement among nurses [15] and found that the kappa coe cient is 0.79. The kappa coe cient was 0.721 in one study that determined the degree of agreement between nurses and medical students [16]. In one study that determined the agreement between triage nurses and an expert group consisting of KTAS instructors, the weighted kappa coe cient was 0.77 [17]. These results are interpreted by Landis & Koch, which shows a 'substantial agreement', but there are some differences among evaluators [18].
On the other hand, problem-based learning (PBL) is a small group of learning method that enables students to present problems and learn knowledge, skills, and attitudes to solve problems [19]. This is a learning method that can develop not only knowledge acquisition, but also reasoning ability and critical thinking ability [19]. In previous studies, after PBL was conducted, learning attitude [20,21], critical thinking skill [22,23], problem solving [21,24], and knowledge [25,26] of nursing students and medical students were improved.
As such, differences in severity classi cation results may occur among triage nurses. The consistency of classi cation results may be improved by education or algorithm application [13]. However, previous studies that suggested how to increase the agreement of the triage results through PBL were insu cient. Therefore, an interventional study is needed to improve the agreement among triage nurses through learner-led PBL. Therefore, the aim of this study was to investigate the effect of PBL led by triage nurses on the agreement of KTAS classi cation results for patients who visited the ED.

Study design
This study had a single-group time series design to investigate the effect of PBL led by triage nurses on the agreement of KTAS classi cation results for patients who visited the ED.

Study subjects
Patients Bujang and Baharum have presented a minimum sample number of 396 with a signi cant level of 0.05, power of 90%, with a goal of 0.7 to 0.8 kappa coe cient for tools of grade 5 [27]. In another study that presented a monogram for calculating sample size with Kappa statistics, the required number of samples was at least 283 at signi cant level of 0.05, power of 80%, and kappa coe cient of 0.9 [28]. Therefore, this study included approximately 1,200 patients (600 patients before PBL and 600 patients after PBL, considering daily visits to the ED. Patients aged 15 years or older who had visited an ED during May and August 2018 for preliminary surveys and May and August 2019 for postmortem surveys respectively were included. Those who had insu cient initial evaluation nursing record in the electronic medical record were excluded.

Triage nurses
Subjects were nurses with KTAS certi cation in the ED who were in charge of initial patient classi cation during the study. A total of 51 nurses are working in the ED of the hospital in which this study was conducted, and one triage room is operated normally. Of a total of 51 nurses, about 15 nurses who have received the quali cations granted by the KTAS committee take turns classifying patients in the ED. We retrospectively investigated the results of triage nurses classifying patients who visited the ED. In May and August 2018, the preliminary survey period, 11 and 12 triage nurses were included, respectively, and in May and August 2019, the post-test period, 12 and 13 triage nurses were included, respectively.

Study procedure
Problem based learning (PBL) PBL was led by triage nurses in charge of patient classi cation in the ED. Triage nurses in charge of patient classi cation in the emergency room were asked to record ambiguous cases during the initial patient classi cation task in a shared notebook placed in the triage room. In this learning meeting, triage nurses and emergency physicians gathered to present their opinions on the KTAS classi cation results of cases recorded in this shared note, and through discussions, a classi cation policy was set so that patients with similar symptoms can later visit the ED (Fig. 1). The date and time of PBL was announced one month in advance, and triage nurses were encouraged to participate voluntarily. When additional learning is needed for a speci c case, one or two triage nurses reviewed the literature and made a presentation within 10 minutes at the next meeting, and the classi cation policy for the case was decided through further discussion. Social media, groupware e-mails, and shared notes in the triage room were used to share the decisions made in PBL with other ED nurses. This learning meeting lasted about 60 minutes each time, and about 10-15 people participated, and it was held twice a month.

Data collection procedure
For pre-investigation, we extracted 300 patients using Microsoft's Excel random function from EDs in May and August of 2018 before the PBL meeting began. We retrospectively reviewed emergency nursing information surveys stored in the electronic medical record (EMR). For post-investigation, we reviewed patient's emergency nursing information surveys stored in the EMR for patients visiting the ED six months after the problem-based study meeting in May 2019 and August 2019 in the same way as the preinvestigation. Emergency nursing information surveys for patients who visited the ED during pre-and post-investigation periods were retrospectively reevaluated without knowing KTAS classi cation results evaluated by triage nurses by two Gold Standard nurses; one nurse was a KTAS instructor and a nurse with over 15 years of ED work experience, and the other nurse had over 15 years of emergency room work experience and over 10 years of triage work experience. We compared KTAS classi cation results classi ed by these Gold Standard nurses with those evaluated by triage nurses. We measured and compared the weighted Kappa coe cient, the coe cient of agreement.

Study endpoints
Inter-rater agreement of KTAS Gold Standard evaluators re-evaluated the KTAS classi cation results classi ed by triage nurses, and the weighted KAPPA coe cient was measured to con rm the degree of agreement between these two groups. Cohen suggested the Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01-0.20 as none to slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1.00 as almost perfect agreement.

KTAS self-e cacy
The self-e cacy of the severity classi cation was investigated with the self-e cacy tool developed by Bandura [29]. This tool is a numeric evaluation scale, with 0 points for the leftmost and 100 points for the rightmost for self-e cacy for triage, and the subject selects one point on a continuous line. And scored.
The higher the score, the higher the level of self-e cacy.

length of ED stay
In the electronic medical record, length of ED stay (ED LoS) was de ned as the time of discharge in the ED such as discharge or hospitalization in minutes from the time of receipt in the ED.

Statistical analysis
Collected data were analyzed using SPSS for Windows 23.0 (IBM, Armonk, NY, USA). General characteristics of triage nurses and patients were subjected to descriptive statistics using nominal variables, frequency, and percentage. Continuous variables are presented as mean and standard deviation. One-way ANOVA was used to test the homogeneity of general characteristics of patients and triage nurses before and after PBL meeting. One-way ANOVA was used to compare ED stay time and selfe cacy of nurses before and after PBL. Weighted Kappa coe cient was measured to analyze the consistency of KTAS classi cation results between triage nurses and Gold standard evaluators.

Ethical considerations
This study was conducted after receiving approval from the B Hospital Medical Research Ethics Review Committee, where researchers of this study were a liated (IRB approval number: 20190426/30-2019-36/053).

Results
General characteristics of study patients and triage nurses according to investigation time Homogeneity test was performed for general characteristics of study patients and triage nurses at the time of the investigation. As a result, the homogeneity of general characteristics according to the time of investigation was secured for both patients and triage nurses groups.   Landis and Koch (1977). The inter-rater agreement of KTAS classi cation results between triage nurses and gold standard nurses was signi cantly (p < .001) improved after conducting PBL meeting (Fig. 2).

Discussion
As Emergency Medical Service Act was amended, all EDs in Korea have been using KTAS to classify emergency patients since January 2016 [10]. This task is handled by nurses in most EDs. Many previous studies have reported that KTAS classi cation results might be different among evaluators. However, it is di cult to nd studies suggesting a way to improve the inter-rater agreement among triage nurses.
Therefore, this study attempted to improve the agreement among triage nurses' self-e cacy for severity classi cation and inter-rater agreement among triage nurses through PBL meeting. Whether the improved agreement might affect the length of ED was also examined. Based on the results of this study, we would like to discuss ways to improve the professionalism of ED triage nurses.
In this study, self-e cacy for emergency patient severity classi cation was improved after PBL led by ED triage nurses. This result cannot be directly compared to previous results because there is no previous research applying PBL. However, results of the study [30] were similar to results of the present study. They compared self-e cacy by developing and applying a web-based KTAS learning program with a control group [30]. Self-e cacy has been found to be improved after education compared to that before and after education in a study comparing HIRAID (History, Identify, Red ags, Assessment, Interventions, Diagnostics, reassessment and communication) framework developed to provide ED nurses with a structured and systematic approach to emergency patient assessment as a simulation training program [31]. The present study con rmed that the self-e cacy was improved through PBL led by triage nurses. However, previous studies have evaluated self-e cacy by applying web-based self-learning program or simulation program. We speculate that self-education, repetitive learning, and meetings to discuss learner-led clinical cases might have raised the self-e cacy of triage nurses' KTAS classi cation.
CTAS and KTAS, unlike ESI which predicts and classi es emergency medical resources to be used, are tools that can classify patients based on their complaints [10]. In the rst stage, patients are classi ed by age. In the second stage, patients are classi ed according to symptoms that patients complain. In the third stage, detailed symptoms included in the major classi cation are selected. In step 4, the KTAS level is determined by checking the pain or symptom of detailed symptoms selected in step 3. Finally, the place of treatment of the patient is determined according to the possibility of infectious disease that can transmit the disease [10]. Worldwide, emergency patients are classi ed by nurses in most EDs [9,32]. Thus, KTAS classi cation results should be carefully classi ed because they can be directly linked to the calculation of emergency medical care fees. They are also important for the safety of patients. However, several previous studies have reported differences in classi cation among evaluators. The Kappa coe cient was .659 in a study of Rahmani et al. to determine the agreement of ESI classi cation between nurses and emergency physicians. In the study of Choi et al., classi cation was conducted by two research nurses for a total of 233 patients [15]. These patients were also reassessed by 10 triage nurses. They found that the weighted Kappa coe cient was 0.79. The weighted Kappa coe cient was 0.721 in a study of Kim et al., con rming the inter-rater agreement between nurses and medical students [16]. In addition, a weighted Kappa factor of 0.77 was reported in a study con rming the agreement between an expert group consisting of KTAS instructors and triage nurses for severity classi cation of ED pediatric patients [17]. The interpretation presented by Landis  Barrows has proposed PBL method that can be conducted in small groups of student-led learning for medical students [19]. It is a learning method that learners can present their problems and learn knowledge and skills to solve problems through discussions in small groups so that they can develop reasoning skills and critical thinking skills to cope with given similar situations [19]. Previous studies have reported that learning attitude [20,21], critical thinking skill [22,23], problem solving ability [21,24], and knowledge [25,26] of nursing students and medical students are improved after the implementation of PBL. In the present study, learner-led PBL meeting was held with triage nurses working on the KTAS classi cation task to discuss cases that needed further discussion. In doing so, a consensus was reached and shared among ED nurses. These shared results might have helped their decision making for similar situations later and inter-rater agreement among triage nurses is likely to improve.
However, there was a slight decrease in the ED LoS before and after PBL, although the decrease was not statistically signi cant. This was different from previous studies [5,6] showing that the ED LoS was reduced due to accurate patient classi cation. This study has some limitations. First, the inter-rater agreement of KTAS classi cation might have improved as nurses' experiences in patient classi cation work increased after PBL meeting compared to those before PBL meeting. On the other hand, some nurses may start a new work of classifying emergency patients before or after the investigation. Thus, the agreement might have declined. Second, the participation rate was very high, although PBL meeting for nurses was voluntary. However, there were also a small number of nurses who showed low participation frequency. For nurses who did not participate in the problem based-learning meeting, results of the meeting were shared using SNS 'Band' application, 'Groupware email', and 'Shared Note'. Third, in this study, the inter-rater agreement between triage nurses and Gold Standard nurses was evaluated. However, we did not determine agreements among the three evaluators. Such information might provide more accurate classi cation results. Lastly, this study was a single-center study conducted at a general hospital in Seoul. Thus, it is di cult to generalize results of this study to other settings.

Conclusion
In this study, PBL meeting for triage nurses improved the inter-rater agreement of KTAS classi cation results and the self-e cacy of triage nurses for emergency patient severity classi cation. Therefore, PBL meeting led by triage nurses can contribute to patient safety in hospitals by enhancing the expertise of triage nurses and increasing the accuracy of triage classi cation. Further studies should be conducted to determine whether the agreement of KTAS classi cation results among triage nurses affects the ED LoS. Educational programs need to be developed to enhance the capacity and professionalism of triage nurses.

Consent for publication
We obtained written consent from all participants for the use of personally identi able images in one scene of the PBL meeting of triage nurses and emergency physician in " Figure 1".

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request. Figure 1 A scene from a PBL meeting on emergency patients triage with triage nurses and emergency physicians.

Figure 2
Inter-rater agreement by time of investigation.