Clinical Utilization of Point-of-care Ultrasound After Training: A Prospective Study

Background: This study aimed to investigate the clinical utilization of PoCUS of the rst post-graduate year residents after a PoCUS curriculum. Methods: It was conducted at the emergency department of the National Taiwan University Hospital between July 2015 and October 2017. Every resident had one-month ED training and a curriculum was implemented during the rst week. The post-training objective structured clinical examination (OSCE) global ratings were obtained, as well as the sonographic examinations by the residents. The participants were categorized into 4 groups: group 1 performed PoCUS before and after the curriculum; group 2 performed only after the curriculum; group 3 performed only before the curriculum; the last did not perform any examinations. Results: Two hundred and thirty-nine residents participated. The median global rating was 4 (interquartile ranges, 4-5). A signicantly increasing number of residents integrated PoCUS into patient care (64 vs. 170, p=0.037) with acceptable image quality after the curriculum. Nearly 30% of residents did not use PoCUS although no differences existed in the OSCE global ratings. Group 1 performed more cases with better image quality and using ≥ 2 ultrasound applications (8/100 vs. 82/359, p=0.0009) after the curriculum. Group 1 performed more cases with better image quality than group 2 after the curriculum. In groups 3 and 4, “no suitable cases” (25/69), and “choosing other imaging priorities” (43/69) were the main feedbacks for not performing PoCUS. Conclusions: A focused training had a substantial positive impact on the utilization of PoCUS of the residents. The global ratings could not predict whether to integrate PoCUS into clinical practice. Further studies would be needed for the sustained effects of the curriculum and how to encourage these residents in groups 3 and 4 to use PoCUS. Clinical trial after correlation to the PoCUS. 30%


Background
Point-of-care ultrasound (PoCUS), referring to "the stethoscope of the 21st century" [1], is de ned as ultrasonography performed by the physician at the bedside [2]. It provides real-time, dynamic images that physicians can correlate with the patient's presenting symptoms, serving as a valuable adjunct for patient care [3]. The use of PoCUS has been increasing robustly in emergency medicine (EM) in recent decades.
Proliferating PoCUS use and an evolving evidence base for different applications [4] have promoted the incorporation of ultrasound (US) training into the medical education of EM.
PoCUS education is a new scope of EM education. The Accreditation Council for Graduate Medical Education (ACGME) has recommended that all EM residents attain competency in the use of US [5].
Objective structured clinical examination (OSCE) is recommended by the American College of Emergency Physicians (ACEP) policy statement to assess for US competency [6]. However, the sub-competency in the EM Milestones document suggests that a resident should consistently utilize and integrate appropriate US applications into clinical management [7]. Little has been known about the actual use of PoCUS by the residents [8]. The results of the OSCE would not readily translate to the use of PoCUS in the clinical decision. Whether the seamless penetration into clinical practice could occur after a US curriculum in a high-volume emergency department (ED), is uncertain.
In Taiwan, after completing medical school, graduates must participate in the one-year postgraduate general medicine training as the rst postgraduate year (PGY-1) residents before they start a minimum of 3-year residency in a specialty. One-month ED training was included. Since 2013, a novel PoCUS curriculum was started for the PGY-1 residents in the ED of the National Taiwan University Hospital.
This study aimed to investigate the clinical utilization of PoCUS of the PGY-1 residents.

Study setting and design
The prospective observational study was conducted at the ED of the National Taiwan University Hospital, a tertiary medical center in Taiwan, between July 2015 and October 2017. The ED has an annual uptake of approximately 85,000 patients. The study protocol was approved by the institutional review board of the hospital (201412004RIND) and registered at ClinicalTrials.gov (NCT03738033). Written informed consent was obtained from the participants.
Eight to ten PGY-1 residents attended the ED training every month. Every PGY-1 resident had 18 eighthour working shifts during their training month and evaluated 10-15 non-critical patients under the supervision of the certi ed emergency physicians on each shift. A novel PoCUS curriculum including 30min didactics and 2-hour hands-on training on a live healthy model volunteer was implemented for the PGY-1 residents during the rst week. Because abdominal discomfort was the leading ED symptoms, the content of the curriculum included the extended focused assessment of sonography for trauma (eFAST) to detect intraperitoneal uid, pericardial effusion, pleural effusion, and pneumothorax, and sonography for urinary tract to detect hydronephrosis, gall bladder to detect acute cholecystitis, and abdominal aorta to detect abdominal aortic aneurysm. The ratio of the instructor to participant is less than 1:5. The instructors are expert sonographers, board-certi ed by the Taiwan Society of Ultrasound in Medicine, and had over 10 years of experience in sonographic examinations. A US machine (SSA-660A, Canon, Japan) equipped with a 2-5 MHz curvilinear transducer was used for training.
The PGY-1 residents completed the curriculum and received an OSCE for post-curriculum assessment immediately. It consisted of standardized questions for image acquisition and interpretation with points for technique, image quality, and correct interpretation of anatomy of FAST, kidney, gall bladder, and aorta. A global rating score using a Likert 5-point scale (unsatisfactory=1, needs improvement=2, satisfactory=3, high satisfactory=4, outstanding=5) [9] was given by the instructor at the scene and by the other instructor that did not involve in training through video review independently. The faces of the PGY-1 residents in the video were covered. The score given by the two instructors was averaged.
There were two US machines (SSA-550A, SSA-660A, Canon, Japan) equipped with 2-5 MHz curvilinear transducers kept ready in use in the ED clinics. Also, the reporting documentation was put in a plastic bag, along with the machine. The sonographic examinations performed by the PGY residents were obtained at the end of the one-month ED training, including indication, scanning targets, sonographic ndings, sonographic diagnosis, and management. The accuracy of sonographic diagnosis was de ned as the agreement between the sonographic diagnosis made by the PGY and by the attending physician. The images were reviewed by another two instructors not involving in training blindly and independently.
The quality of the images was categorized using a 5-point Likert rating scale. Point 1 indicated no recognizable structures, no objective data can be gathered; point 2 indicated minimally recognizable structures but insu cient for diagnosis; point 3 indicated minimal criteria met for diagnosis, recognizable structures but with some technical or other aws; point 4 indicated minimal criteria met for diagnosis, all structures imaged well and diagnosis easily supported; point 5 indicated minimal criteria met for diagnosis, all structures imaged with excellent image quality and diagnosis completely supported [10]. Another instructor not responsible for the curriculum would interview the PGY-1 residents who did not perform clinical sonographic examinations after the curriculum and their feedbacks was obtained.

Selection of Participants
The PGY-1 residents attending the curriculum were included. Those not attending the curriculum were excluded.

Data collection
The demographic data of the PGY-1 residents were obtained, including age, gender, and prior US experience. The global ratings of the OSCE were collected, as well as the sonographic examinations including indication, scanning targets, sonographic ndings, sonographic diagnosis on shifts. Based on the clinical sonographic examinations, the residents could be categorized into 4 groups. Group 1 indicated the residents performed PoCUS before and after the curriculum; Group 2 indicated that those performed PoCUS only after the curriculum; Group 3 indicated that those performed PoCUS only before the curriculum; the last indicated those did not perform any examinations. The scanning targets were categorized into a single application (ex, FAST) and more than 2 applications (ex, FAST+gall bladder) according to the ACEP statement [6].

Outcomes
The primary outcome was the utilization of PoCUS of the PGY-1 residents during clinical practice.

Statistical analysis
All data were analyzed by SAS software (SAS 9.4, Cary, North Carolina, USA). Categorical data were expressed in counts and proportions, while continuous data were expressed in medians and interquartile ranges (IQRs). Categorical variables were compared using a Chi-square test and ANOVA. Continuous variables were examined using Wilcoxon's rank-sum test. Intraclass correlation (ICC) with one-way random effects was used to assess inter-rater reliability for those global rating scores and imaging quality scores by two evaluators.
The linear regression models were applied to identify the factors associated with numbers and the accuracy of the sonographic examinations after the curriculum. The polytomous regression models were applied to investigate the factors associated with the image quality of the sonographic examinations after the curriculum. The covariates included age, gender, prior US experience, and global ratings. Also, the factors associated with the global ratings were investigated using the polytomous regression models.
The covariates included age, sex, and prior experience. A p-value of less than 0.05 was considered statistically signi cant.

Results
During the study period, 239 PGY-1 residents attended one-month ED training, and all of them (100%) participated in this study and completed the OSCE. A total of 901 sonographic examinations were collected and reviewed. There was a good ICC coe cient of 0.86 (95% CI, 0.68-0.96) for the OSCE global rating score and 0.84 (95% CI, 0.63-0.95) for the image quality of the sonographic examinations. Table 1 lists the demographic data of residents. The majority were novice sonographers. The median OSCE global rating score was 4. An increasing number of PGY-1 residents integrate PoCUS into patient care on their shifts after the curriculum (64 vs. 170, p=0.037). Also, more increasing case numbers with better image quality and more accurate diagnoses were noted after the curriculum (Table 2). After adjusting age, gender, and prior US experience, a higher global rating (coe cient, 1.38±1.16, p=0.03) was signi cantly associated with the better image quality of the sonographic examinations after the curriculum. However, there was no impact on the numbers of the cases and the accuracy.
Based on the clinical sonographic examinations, the participants were categorized into 4 groups (Table   3). Among the groups, there were no differences in the global ratings. Group 1 residents performed more cases with better image quality after the curriculum, compared with those before the curriculum. Also, group 1 residents performed the examinations using ≥2 US applications after the curriculum (before vs. after, 8/100 vs. 82/359, p=0.0009) ( Table 4). Comparing with the sonographic examinations after the curriculum in groups 1 and 2, group 1 performed more cases (p<0.0001) with better image quality (p=0.03) although the accuracy was unremarkable. Tables 4 and 5 list the US indication and scanning targets of the sonographic examinations in groups 1 and 2.
Sixty-nine residents in groups 3 and 4 did not perform sonographic examinations after the curriculum. "No suitable cases for examinations" (25/69), and "having other imaging priorities" (43/69) were the main feedbacks of these residents.

Discussion
While clinical integration is essentially addressed during PoCUS education, little has been known about the actual use of PoCUS by the residents after training [8]. This study provides evidence regarding the real US performance of the inexperienced PGY-1 residents after a short US curriculum. The results showed the US curriculum had a substantial effect on increasing the clinical integration of PoCUS. A signi cantly increased number of PGY-1 residents used PoCUS during clinical practice after the curriculum. Also, the global ratings had no obvious correlation to the clinical integration of PoCUS. Nearly 30% of them did not use PoCUS although their OSCE performance was good.
Time constraints would be a barrier for ED medical education. Previous studies suggested that short courses of focused US training were of acceptable quality, even in paramedics with no prior ultrasound experience [11][12][13]. Also, US skills would decay at different times for different organ images such as a signi cant decline of the ability to identify A-lines of lung over time, but no decline in cardiac image recognition [14]. The content of the US curriculum was designated as the combination of unique abdominal US techniques because abdominal disorders were frequently encountered in our ED.
The introduction of the OSCE showed comparable improvements in trainee performance and understanding feeling con dent in their ability to utilize US [7]. There is evidence that global rating scales capture diverse levels of pro ciencies better than checklists, and are easy for examiners to use [15]. Also, global ratings were shown to yield reliable data when compared with checklist scores for the assessment of residents in the eld of medicine [13,16]. The skills needed for US are complex and include hand-to-eye coordination, image optimization, and image interpretation [17], and clinical correlation. Therefore, this study used global ratings for OSCE performance instead of checklist scores due to the psychometric properties [16] and higher levels of expertise on US performance. There were 4 different patterns of the behavior of doctors after US training. This short US curriculum helped inexperienced doctors to integrate PoCUS into clinical practice, especially in groups 1 and 2. The residents used self-formatted combinations of US techniques that the FAST and the renal US were the most common contents. The results showed more than 70% of residents had a desire to change the behavior to be willing to integrate PoCUS into daily practice. Although residents in groups 1 and 2 had progressed, group 1 residents seemed to be more active and motivated in performing sonographic examinations, compared with those in group 2.
The ED is a unique learning environment different from any other setting in the hospital. The high volume and acuity of patients in the ED provide opportunities to repeatedly practice and to facilitate the accumulation of clinical experience for young staffs, crossing the gap from the level 2 (learning level) to the level 3 and 4 (behavior and result levels) easily [18]. However, the characteristics of ED patients that each resident encountered are diverse and varied. The time spent with an individual patient is limited. In our ED, the PGY-1 residents could evaluate 200-250 patients during the training month. Although the number of scanning cases performed by the PGY-1 residents increased after the curriculum, the median was 3 cases. It implied that the PGY-1 residents used PoCUS in highly selected, maximally bene cial patients.
The potential barriers exist for utilizing PoCUS in clinical settings. The frequent challenges are US equipment and lack of con dence or experience in clinical integration [8,19]. Also, the lack of evidence for clinical utility for certain patients, a trade-off between performing US and other clinical processes, and the inability to use US for decision-making were reported [8,20]. In our study, the facility was settled that the US machines were easily accessible. All of the sonographic examinations were under the supervision of the certi ed ED physicians. The working climate was designated to be friendly for using US, but some barriers still existed. As the feedbacks from groups 3 and 4, the residents could choose other imaging modalities such as computed tomography or magnetic resonance imaging for patient care, whatever autonomously or by the attending physician's suggestion. It implied the gap between PoCUS and comprehensive imaging modalities would be hardly crossed in a time-limited and stressful environment. Moreover, how to encourage groups 3 and 4 residents to use PoCUS in clinical practice would be an important issue in future studies.
This study had some limitations. First, it was conducted in an academic medical center with active US training. External generalization would be further explored. Although the distribution of ED patients varies from hospital to hospital, however, this study provided novel evidence regarding how inexperienced residents integrate PoCUS in their primary care after a tailored educational curriculum. The design and contents of the curriculum were based on the frequently encountered complaints in our ED, and more than 70% of residents used the techniques in their clinical practice. Second, this study investigated the short-term effects of the US curriculum. Insights into long-term retention of US skills and subsequent time to skill degradation were not evaluated. Future longitudinal studies would be needed for these residents for their specialties after one-year general medicine training, especially in groups 3 and 4. Whether the specialty would have an impact on the utilization of PoCUS was wondered. Third, the sonographic examinations were based on written documentation. The percentage of the residents utilizing US would be under-estimated if they performed the sonographic examinations but the results couldn't be seen in the paper records or the electronic medical records. Fourth, the change in the management plan after PoCUS was not evaluated in this study because the plan was in uenced by the sonographic ndings and made by the attending physician. Last, a more structured survey for groups 3 and 4 would be needed for a more detailed understanding of the reasons for not using PoCUS. Availability of data and materials: All data generated or analyzed during this study are included in this published article.
Competing interests: The authors declare that they have no competing interests.
Funding: no funding.
Authors' contributions: WC and HP conceived the study and designed the trial. WC, CH, KM, MC, CY, and HP acquisition of the data. WC and HP analysis and interpretation of the data. WC drafted the manuscript, and all authors contributed substantially to its revision. HP critically revised the manuscript for important intellectual content and takes responsibility for the paper as a whole. All authors read and approved the nal manuscript.   Cases, n --Imaging quality --Accuracy --*Presented as median (interquartile ranges, IQRs). † OSCE, Objective Structured Clinical Examination. ‡ Comparing between the performance before and after the curriculum in the group 1.