Questionnaire for determining physicians’ attitude change after Triage & Action minor emergency course

Background: The Triage & Action (T&A) minor emergency course was developed to improve clinical skills for minor emergent problems of Japanese non-specialists. Currently, the course quality is evaluated only by self-reported degree of satisfaction. This study aimed to describe a newly developed questionnaire to evaluate the change in attitude toward clinical practice after completing a T&A minor emergency course and to evaluate its validity and reliability. Methods: The questionnaire was prepared, and the content and contextual validity of the aforementioned course were validated by a clinical sensibility test. Reliability was assessed by the interclass correlation coecient after 2 weeks via a follow-up questionnaire. McNemar’s test for categorical variables or the Wilcoxon signed-rank test for continuous variables was used for sensitivity analysis, with a statistical signicance level of p < 0.05. Results: Among 103 evaluators receiving the questionnaires, 44 (42.7%) responded to the questionnaire, 36 (40.8%) to the follow-up questionnaire, and 33 (32.0%) to both questionnaires; 28 (27.2%) participants took the clinical sensibility test. Seven questions on the reliability test showed fair agreement, and there were no signicant differences in the answers to the sensibility test questions. In the clinical sensibility test, 82.1% of the physicians answered “Fair to large extent” to the question about perspicuity, whereas 92.8% answered “Normal to very likely” to the question “To elicit the participants’ attitude.” Conclusion: The new questionnaire was shown to be contextually well validated and reliable.


Background
The concept of emergency medicine (EM) in Japan is different from that in western countries. In the 1960s, a legislation was passed in Japan that implemented the use if emergency medical services, and emergency hospitals were designated by the government. However, there were no EM specialists; therefore, surgeons/physicians without specialist training in EM provided care to emergency patients (1).
The Japanese Association for Acute Medicine (JAAM) established a committee to promote the westerntype model of EM in 2003. By 2007, more than 150 JAAM-a liated hospitals had implemented this type of EM. However, because adequate emergency physicians are not available, non-emergency physicians are required to see patients with minor emergent problems in the emergency room. Furthermore, there have not been enough courses aimed at improving the skills of physicians who can treat minor emergent problems. This de cit has led to ambulances not being able to nd hospitals with the appropriate resources to treat patients. Furthermore, hospital refusals to accept patients have been a major problem in Japan. One of the most frequent reason given for refusal is "without a specialist," often in the case of minor emergencies (2).
We developed a Triage & Action (T&A) minor emergency course (http://minoremergency.club/) in Japan and began offering it in 2015 because simulation training can improve clinical knowledge and skills and has been demonstrated to have clinical context validity in a growing number of studies (3− 5). The courses have been held 21 times, and a total of 461 physicians have completed the course as of December 31, 2018. The course aims to improve the clinical knowledge and skills for treating minor emergencies. The training sessions comprises ve minor emergencies (Epistaxis, Ear and nose foreign body, Sprain or fracture, Ocular surface foreign bodies, and Burns) as well as lecture-and simulationbased training that is conducted by dedicated multidisciplinary instructors. The simulation-based training uses real-case scenarios, and the participants decide how to treat each training patient with a minor emergency. The instructors describe patients with various disease and injury scenarios. Mannequins are used for "Epistaxis" and "Ear and nose foreign body". For "Sprain or fracture," an ankle sprain scenario described by the instructor is used, and the participants treat the patient with a splint. There are pictures of items (e.g., gauze, Vaseline®, and normal saline) in "Burns" and "Ocular surface foreign bodies." The participants choose which items they will use for treatment and decide if the patients should consult a specialist. The instructors assess the participants' decisions and skills. This course is a 1-day course (7 h), and two T&A minor emergency instructors supervise ve attendants. The instructors for T&A minor emergency are selected by a T&A principal member after taking the course. More than 90% of previous participants have answered "Excellent" or "Good" concerning their level of satisfaction with the course.
However, information about the physicians' clinical practice in treating minor emergencies after taking the course has been limited.
We thought that it was important to improve the physician's con dence and attitudes in treating minor emergencies. Previously, improvements in clinical skills have been usually evaluated by performing the same simulation task or in a real clinical situation > 3 weeks after the simulation (3,6). However, T&A minor emergency course trainees from all over Japan have participated in this course, making it di cult to evaluate the change in a particular physician's skill level. Therefore, we considered that information about a physician's con dence level and total number of minor emergencies treated could be used in place of real clinical assessments because they could change the physician's practice and reduce their refusals to treat minor emergencies.
Questionnaires are widely used in medical research to collect information from both patients and healthcare professionals (7,8). Some simulation courses have used questionnaires to evaluate knowledge before and after the simulation course (9 − 11). To ensure that the questionnaire closely matched realworld experience and ability, we thought that the post-course questionnaire should be completed within a few months and should contain information about self-con dence and real clinical practice experience (e.g., the change in the total number of treatments without referrals to a specialist within the speci c period). We hypothesized that checking the changes in con dence will provide us information regarding what should be done in T&A minor emergency course in the future.
To the best of our knowledge, no study has reported the use of questionnaires to assess improvement in a physicians' attitude and self-con dence after participating in a simulation course after a few months.
The present study aimed to describe a newly developed questionnaire to evaluate how participants' attitudes toward clinical practice had changed after completing a T&A minor emergency course and to evaluate the validity and reliability of the new questionnaire.

Evaluation strategy
We conducted this study in accordance with the method proposed by Burns et al. (12). The survey used a questionnaire and a clinical sensibility test. We previously developed the questionnaire to evaluate the change in attitude toward clinical practice after taking a T&A minor emergency course. In the present study, we tested the reliability of the questionnaire and used a clinical sensibility test to validate the content.

Methods
The participants responded to the rst questionnaire to validate the content by a clinical sensibility test, followed by responding to the follow-up questionnaire within 2 weeks after the test to evaluate reliability (test-retest exam). This survey was conducted in November and December 2018. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results. Furthermore, they were not invited to contribute to the writing or editing of this document for overall readability or accuracy.

Participants
The web-based questionnaire was evaluated by 103 physicians identi ed from a mailing list as having taken the T&A minor emergency course. There were two inclusion criteria: (1) To evaluate reliability, we included the responders who answered both the rst and follow-up questionnaires. (2) To evaluate validity, we included physicians who answered the rst questionnaire and took the clinical sensibility test.

Questionnaire
Two principal developers of the T&A minor emergency course and one physician-researcher created a new questionnaire to evaluate self-con dence in treating minor emergencies and real practice experience.
The questionnaire for this study comprised 32 questions related to two major factors: the physicians' background and self-con dence in treating minor emergencies (Additional le 1). The responses were provided by checking boxes or by ranking responses.
The second part of the questionnaire had questions about the physicians' experience and con dence in treating each type of minor emergency. We chose ve minor emergencies (Epistaxis, Ear and nose foreign body, Sprain or fracture, Burn, and Ocular surface foreign bodies) that were presented via simulationbased training in the T&A minor emergency course. To evaluate the respondent's hospital for their ability to provide minor emergency professional treatment, we required that the hospital has a specialist for the disease; the specialists for Epistaxis and Ear and nose foreign body were otolaryngologists (14,15), those for Sprain or fracture were orthopedic physicians (16, 17), those for Burn were dermatologists and plastic surgeons (18, 19), and those for Ocular surface foreign bodies were ophthalmologists (20). We created the following questions: 1. "What was the total number of patients you treated for each specialty disease in a month?," 2. "Was there a specialty department for each disease in the respondent's hospital?," 3.
"What was the total number of each minor emergency treated within 6 months?," 4. "How con dent are you in your clinical skill for treating each minor emergency?," 5. Have you treated patients with each minor emergency without a specialist's support within the last 6 months?," and 6. "What is the total number of patients treated for each minor emergency without a specialist's support within the last 6 months?" The participating physicians were asked "Yes" or "No" in question 1. Question 5 had three categorical answers: "Yes," "No," and "I am a specialist." If the participant answered, "I am a specialist," they did not need to answer question 6. The answer was an integer number for questions 2, 3, and 6. The participants were asked to rate their con dence level from 0 to 5 (0 = No con dence and 5 = Fully con dent) for question 4.
We removed the physician's background to reduce the effort needed to answer the follow-up questionnaire. The second part of the follow-up questionnaire was almost the same as that used in the rst questionnaire.
To evaluate the reproducibility of the questionnaire, the participant's name was used to match the rst and follow-up questionnaires' responses.

Clinical sensibility test
The goals of the clinical sensibility test were to assess the comprehensiveness, clarity, and contextual validity of the questionnaire's content (Additional le 2). We used the clinical sensibility test, which contains seven questions (13). The following questions were used: 1. "To what extent are the questions directed at important issues pertaining to T&A minor emergency course's participants?," 2. "Were there any missing items?," 3. "Was the questionnaire simple and easily understandable?," 4. "Did the questionnaire provide information pertaining to the physician's knowledge and experience?," 5. Were any of the questions inappropriate or redundant?," 6. "How likely is the questionnaire to elicit important issues in physician's attitude toward minor emergencies in T&A minor emergency course's participants?," 7. How long did it take you to complete the questionnaire (min)?" The participants answered questions 1-4 and 6 by selecting a response from the Likert scale (e.g., very unlikely, unlikely, neutral, likely, and very likely).
Questions 2, 4, 5, and 6 had a free-entry column about each question. We also made a free-entry column for the participants to provide ideas on how to improve the questionnaire.

Primary data analysis
We used STATA/MP 15.1 software (StataCorp LLC) for the data analyses and the interclass correlation coe cient (ICC) to assess reliability. We decided that the reliability coe cient could be qualitatively categorized as follows: ICC < 0.4 is poor, 0.4 ≤ ICC < 0.6 is fair, 0.6 ≤ ICC < 0.75 is good, and 0.75 ≤ ICC ≤ 1 is excellent (21). McNemar's test for categorical variables or the Wilcoxon signed-rank test for continuous variables was used for sensibility analysis, with a statistical signi cance level of p < 0.05.

Characteristics of the study participants
During the study period, 44 (42.7%) participants responded to the rst questionnaire and 36 (40.8%) responded to the follow-up questionnaire. Thirty-three (32.0%) participants answered both questionnaires.
Twenty-eight (27.2%) participants took the clinical sensibility test. No responders replied only to the clinical sensibility test. Table 1 presents the backgrounds of the physicians who answered both questionnaires.   Test-retest exam Table 2 presents the results of the test-retest exam. The mean period in which the two tests were completed was 35.4 days (SD = 12.1). We found that six questions regarding "The total number of epistaxis patients treated without an otolaryngologist within 6 months," "The total number of ear and nose foreign body patients treated without an otolaryngologist within 6 months," "The total number of dermatological disease patients treated within a month," "The total number of burn patients treated within 6 months," "The total number of burn patients treated without a dermatologist or plastic surgeon within 6 months," and "The total number of ocular surface foreign bodies patients treated without an ophthalmologist within 6 months" gave an ICC that was fair (0.4 ≤ ICC < 0.6).  Clinical sensibility test  7. How long did it take you to complete the questionnaire (min)? 5 (5-10) Abbreviation: T&A, triage and action

Discussion
The study results showed that the questionnaire about the T&A minor emergency course was valid and reliable.

Test-retest exam
In the test-retest examination, six questions about the total number of patients treated for various minor emergencies alone or with a specialist gave an ICC value of fair (0.4 ≤ ICC < 0.6). Morita et al. reported that it was di cult to show reliable results involving human emotions or knowledge because of changes in the subjects' subjective conditions (22). To minimize this difference, the participants are usually assessed by having them answer the same questions typically with a 2-4-week period (13). Therefore, we chose an answering interval of 2 weeks. It is possible that this fair ICC may have been caused by random error or because the respondents saw different number of patients during the study periods.
However, we did not observe a signi cant difference in the answers to these questions in the sensitivity analysis between test and retest. Therefore, we considered that the results of the sensibility test could compensate for the low reliability of the ICC results. Other questions showed moderate to excellent ICC. A signi cant difference was observed in the test-retest answers to question 5 "Do you have experience treating patients with an ear and nose foreign body without an otolaryngologist's support within 6 months?" in the sensibility test. However, the question's ICC was good (0.6 ≤ ICC < 0.75). Consequently, we considered this questionnaire to have high reliability.

Clinical sensibility test
All of the answers to the questions in the clinical sensibility test were favorable, which we interpreted as a strong indication of the validity of the questionnaire's content and clinical context (13,23). Regarding question 2, some responders mentioned that this survey did not include a question concerning the "Satisfaction rate." The lack of this question may cause less con dence in the questionnaire for some physicians. However, our course already collected information about the "Satisfaction rate" on a paperbased questionnaire without the physicians' names before we started our study. Because we thought that the participants' answers may be affected if the questionnaire was not anonymous (24), we thought that no name should be associated with the "Satisfaction rate" question. Therefore, we did not include a question about the "Satisfaction rate" in our web-based questionnaire.
Mirta et al. (25) reported in their web-based study that their online questionnaire's stated length (10, 20, and 30 min) was longer than the length and number of respondents in our study and that fewer respondents started and completed the questionnaire. Our study showed that the median time to answer the questionnaire was 5 min (IQR 5-10 min), which we considered to be very reasonable.
One of the participants commented in the free-entry column that "This questionnaire should contain questions about the availability of a specialist at night or when the primary one has a day off." Therefore, we added a question asking "Do you have a specialist's support at night or on holidays?" and the answers were "Any time as needed," "Sometimes," and "Not at all."

Limitations
This study had several limitations. The rst concerns the external validity because we chose T&A minor emergency mailing list members to serve as participants; thereofre, the risk of selection bias should be considered. Furthermore, the limited sample size should be considered. The second limitation was that we used two questionnaires over a 2-week period. It is possible that the actual practice pattern may have changed during the 2 weeks. The third limitation concerns potential confounders. The possibility of confounding factors not included in this study should be considered. Fourth, the sample size was small and the response rate was low. The low response rate can be attributed to the fact that the respondents were volunteers.

Conclusion
The object of this newly developed questionnaire was to evaluate the change in attitude toward clinical practice after completing a T&A minor emergency course. In the future, we plan to send the pre-webbased questionnaire before the course and the post-web-based questionnaire 6 months after the course.
The post-web-based questionnaire contains the same question as those in the pre-questionnaire except for the addtion of the question "Did you change your place of work after answering the pre-  Abbreviation: IQR, interquartile range OriginalJapanesequestionnaire.pdf OriginalJapaneseClinicalsensiiblitytest.pdf