Behavior Change and Skills Retention With an Action Record After Ultrasonography Training

In simulation training, behavior change (Kirkpatrick’s level 3) is more important than learning improvement (Kirkpatrick’s level 2). However, few studies have evaluated behavior change because it is dicult to assess objectively. Skills retention is another challenge. We evaluated whether keeping a record of the number of ultrasound (US) examinations performed after a simulation course led to positive behavior changes and improved skills retention.

diagnosis, and contributes to reducing complications. 1, 2 POCUS is also part of under-graduate and postgraduate medical education. [3][4][5] Numerous POCUS simulation courses with variable content and duration are available. 6,7 Generally, simulation courses such as the Advanced Cardiovascular Life Support course have been shown to increase knowledge or skills. 8 POCUS simulation courses for medical students and doctors have shown similar results. 1,5,7,9 Few studies have evaluated the educational effect of POCUS courses for nurse practitioners; however, some studies reported that this training improved skills. [10][11][12] In evaluating the educational effect of training, the four levels of learning evaluation advocated by Kirkpatrick and Kirkpatrick showed that "What did participants apply in practice?" (level 3; behavior) is more important than "What have participants learned?" (level 2; learning) ( Fig. 1). 13 Few studies have shown that simulation training can change both behavior and the learning level, especially regarding POCUS. 14 In addition, simulation courses can increase knowledge and skills immediately after the course; however, these gains tend to decline a few months after the training. 4,8,9,15,16 To resolve this problem, follow-up lectures or hands-on training after the initial course may be effective for maintaining knowledge. [17][18][19] POCUS simulation courses are similar; however it is unclear whether both knowledge and clinical skills (e.g., image acquisition) can be maintained. 4,5,9,15,16,20,21 In this study, we held a 2-day POCUS simulation course for Japanese nurse practitioners (JNP) and JNP trainees. These practitioners were instructed to record the number of ultrasound (US) examinations they performed before and after the course. This study had two aims. First, we aimed to determine whether keeping a record of the number of US examinations performed after the simulation course improved both knowledge and skills (level 2; learning) and led to behavior change (level 3; behavior). Second, we aimed to determine whether keeping a record of the number of US examinations helped to maintain US knowledge and skills. Behavior change was evaluated by comparing the number of US exams performed before and after the course. To evaluate maintenance of the learning level, we evaluated image interpretation skills, image acquisition skills, and con dence in performing POCUS before and immediately after the course, and again 4 months after the course.

Study design and setting
We held one POCUS training program in 2018 and one in 2019. The program involved four parts: 1) recording the number of US exams performed during the 3 months before participating in the POCUS course, 2) participating in the 2-day POCUS course, 3) recording the number of US exams performed during the 3 months after participating in the course, and 4) follow-up evaluation 4 months after the course. All participants recorded the number of cardiac US, lung US, deep vein thrombosis (DVT) US for the lower extremities, and abdominal US they performed in the 3 months before the course. We chose these four US examinations because the 2-day POCUS course focused on these examinations A standardized POCUS course with a proven educational effect was adopted for our 2-day POCUS course. 7 The educational effects of this course for medical students and doctors have been demonstrated; however, this was the rst such training for JNPs. 7 Participants' image interpretation skills, image acquisition skills, and con dence in performing POCUS were evaluated before and after the course. Image interpretation skills were evaluated by a written examination using POCUS case study videos and multiple-choice questions. 7 Image acquisition skills were evaluated in hands-on situations by the POCUS instructors using live models and evaluation sheets. Con dence was evaluated by a self-evaluation survey with a ve-point Likert scale using previously validated multiple-choice questions and a selfevaluation survey. 7 Participants then recorded the number of US examinations they performed for the 3 months after the course. Four months after completing the course, participants completed a follow-up test covering image interpretation skills, image acquisition skills, and con dence. This test was the same as that performed immediately after the course ( Figure 2). There were no interventions, including didactic lectures, between the end of the course and the 4-month follow-up test.
Nine instructors were involved in the course and evaluated participants. All instructors were certi ed POCUS instructors. 7 Before the course, the instructors received a lecture presenting the evaluation method and online discussions to standardize the evaluation method.
This study was approved by the Institutional Review Board of the Tokyo Bay Urayasu Ichikawa Medical Center. Before participation, participants were informed that the results of this study would not affect evaluation of their work or future training. Written informed consent was obtained from all participants.

Participants
Japan has an original nurse practitioner system (JNP system), which began in 2008 and was partially revised in 2015. 22 There are several certi ed JNP training programs in Japan. In the present study, JNPs and JNP trainees were recruited through the JNP training program delivered by the Japan Association for the Development of Community Medicine from 2018 to 2019. During the study period, JNPs worked in hospitals or clinics, and JNP trainees worked in hospitals and were engaged in on-the-job training under the supervision of attending doctors; therefore, all participants could access portable US machines and perform US examinations.

Data collection
All participants recorded the number of US exams they performed on their own for 3 months before participating in the 2-day POCUS course. This information was recorded on a Microsoft Excel spreadsheet distributed by the study secretariat. These sheets were collected before participants began the POCUS course. The number of US examinations performed during the 3 months was categorized: category 1: 0 cases, category 2: 1-9 cases, category 3: 10-29 cases, category 4: 30-49 cases, category 5: 50-99 cases, and category 6: ≥100 cases. Image interpretation skills, image acquisition skills, and con dence in performing POCUS were evaluated pre-and immediately post-course. Participants recorded the number of US examinations they performed for 3 months after the course, using the same system as for the 3-month period before the course. These records were collected by the study secretariat. Four months after the course, participants completed follow-up testing to evaluate their image interpretation skills, image acquisition skills, and con dence. This test was the same as that used for the pre-and immediate post-course testing.

Statistical analysis
Comparisons of the difference between the US examination categories before and after the course were analyzed using Wilcoxon's signed-rank test. Written examinations, evaluation sheets, and self-evaluation survey scores were analyzed with the Friedman test with Bonferroni adjustment. Data analyses were performed using EZR statistical software (version 1.52), which is a graphical user interface for R (The R Foundation for Statistical Computing, Vienna, Austria). 23

Participants
Thirty-ve participants completed the POCUS training program in 2018 or 2019. Two participants were excluded because they could not complete the program. Nine JNPs and 24 JNP trainees from 21 facilities completed the program. These facilities were geographically distributed across Japan from Hokkaido in the north to Nagasaki prefecture in the south. Some participants were from the same facilities, and most (94%) worked in community hospitals; no participants worked in university hospitals. The mean number of post-graduate years was 13.2 years (range: 6-22 years). All participants were novice POCUS trainees (Table 1).  Image interpretation skills, image acquisition skills, and con dence scores The mean scores for the image interpretation skills test pre-course, and immediately post-course, and at the 4-month follow-up evaluation were 37.1 (SD: 16.0), 72.6 (SD 11.1), and 71.8 (SD 9.9) (out of 100 points), respectively. Both the immediate post-course test and the 4-month follow-up test scores were statistically signi cantly higher than the pre-course scores (P < 0.001). However, the difference between the immediate post-course and the 4-month follow-up test scores was not statistically signi cant (P = 1.00). The mean scores for the image acquisition skills test pre-course, immediate post-course, and at the 4-month follow-up were 13.7 (SD: 10.7), 53.6 (SD: 8.9), and 52.9 (SD: 9.3) (out of 71 points), respectively.
Both the immediate post-course and 4-month follow-up test scores were statistically signi cantly higher than the pre-course test scores (P < 0.001). The difference between the immediate post-course and the 4month follow-up test scores was not statistically signi cant (P = 1.00). The mean scores for con dence pre-course, immediate post-course, and at the 4-month follow-up survey were 15.8 (SD: 3.6), 35.7 (SD: 10.5), and 33.0 (SD: 11.6) (out of 70 points), respectively. Both the immediate post-course survey and 4month follow-up test scores were statistically signi cantly higher than the pre-course survey scores (P < 0.001). The difference between the immediate post-course and the 4-month follow-up test scores was not statistically signi cant (P = 0.34) (Figure 4).

Discussion
This study aimed to determine whether recording the number of US examinations performed after taking a POCUS simulation course led to a behavior change, and whether keeping a record of the number of US examinations performed maintained US knowledge and skills. Our results showed that keeping a record signi cantly increased the number of US examinations performed. In addition, keeping a record after the simulation training led to a behavior change in the eld of simulation education. Keeping a record also contributed to maintaining POCUS knowledge, skills, and con dence. Our study suggests that keeping a record may be useful to improve skills retention in the eld of simulation education.
In educational methods, including in simulation training, it is important and most effective to cause both a reaction or learning improvement (Kirkpatrick's levels 1 and 2, respectively), and behavior change or improvement (Kirkpatrick's levels 3 and 4, respectively). 13 However, it is often di cult to evaluate levels 3 and 4 because this evaluation is time consuming and requires effort and cost to train evaluators and prepare tools and facilities. Therefore, few studies have evaluated behavior change, and effective methods to change behavior have not been established in the eld of simulation training. 14,24,25 In our study, recording the number of clinical US examinations increased the number of these examinations that participants performed after the simulation course. Our results also showed that keeping a record after the simulation course led to a behavior change.
This study showed that the quality of the examinations was maintained after the course. The 4-month follow-up test results showed that the image interpretation skills, image acquisition skills, and con dence scores were statistically signi cantly improved compared with the pre-course test scores, and that these scores did not decline compared with the immediate post-course test scores. The problem of skills retention is an most important problem in the eld of simulation training. 4,5,16,17 Knowledge, skills, and con dence decline in a few months to 1 year after a simulation course with no interventions. 4,5,9,16,19 Several methods have been proposed to help participants retain knowledge and skills; for example, providing didactic or online lectures, and holding hands-on training sessions or simulation training courses regularly or several months after the course. 15,[18][19][20][21] The method that we used in this study (keeping a record) was useful to maintain knowledge, skills, and con dence 4 months after the simulation course. This method involves less effort and cost than conventional methods and is feasible and can be implemented at most facilities.
This study had several limitations. First, we recruited only JNPs; however, participants were from 21 facilities in numerous regions across Japan. The number of post-graduate years also varied; repeating this research with attending doctors, residents, and medical students is needed to clarify the usefulness of this method for these cohorts. Second, this study involved a follow-up test 4 months after the course. Participants were aware of this follow-up test in advance, which might have in uenced their behavior.
However, all participants were informed that the results of this study would not affect their future work or training. Therefore, the impact of the follow-up test was not considered large. Third, the number of US examinations was self-reported. Additionally, nine instructors participated as evaluators. All instructors were certi ed and trained regarding how to evaluate participants before this study. However, it cannot be denied that there might have been measurement error. This study was not a crossover study, and we did not compare study participants with a group who did not keep a record. However, research has shown that skills and knowledge decrease after a simulation course without intervention. Therefore, our method appeared to be effective to improve the educational effectiveness of simulation courses.
Although there were several limitations, our study indicated that keeping a record after taking a simulation course can lead to behavior change. This method also effectively maintained knowledge, skills, and con dence and is inexpensive, with good feasibility. This method is therefore useful to induce behavior change (Kirkpatrick's level 3) and improve skills retention in the eld of simulation training.

Conclusion
Keeping a record after a POCUS simulation course increases the number of clinical US examinations performed after the course. Image interpretation skills, image acquisition skills, and con dence also improve and are maintained. This method not only improves the learning effect, but also leads to changes in behavior (Kirkpatrick's level 2 and 3, respectively) in the eld of simulation training. Skills retention is also improved. The method is inexpensive and feasible. Combining simulation training with keeping a record may improve the educational effect in the eld of simulation training.

Declarations
Ethics approval and consent to participate: The study protocol was approved by the Institutional Review Board of the Tokyo Bay Urayasu Ichikawa Medical Center. Written informed consent was obtained from all study subjects before participation. All methods were carried out in accordance with regulations of Institutional Review Board of the Tokyo Bay Urayasu Ichikawa Medical Center.
Consent for publication: Not applicable.
Availability of data and materials: The datasets generated and analysed during the current study are available from the corresponding author on reasonable request Competing interests: The authors declare that they have no competing interests.
Authors' contributions: All authors were involved in study design, data interpretation, and manuscript preparation. TY was the principal investigator and was responsible for regulatory compliance, participant recruitment, data collection, data analysis, and manuscript preparation. JE, HF, KE, YK, and YT contributed to the study coordination and data collection, entry, and analysis. All authors read and approved the nal manuscript.