“To Do or Not To Do?” Clinical Integration of Point- Of-Care Ultrasound After Training: A Prospective Study


 Background: Point-of-care ultrasound (PoCUS) can serve as a valuable adjunct for patient care. This study aimed to investigate the clinical integration of PoCUS of the first post-graduate year (PGY-1) residents after a PoCUS curriculum.Methods: The prospective study was conducted at the emergency department (ED) of the National Taiwan University Hospital between July 2015 and October 2017. Every PGY-1 resident had one-month ED training and a PoCUS curriculum was implemented during the first week. The post-training objective structured clinical examination global ratings were obtained, as well as the sonographic examinations the residents performed on shifts. Four groups could be identified: group 1 performed PoCUS before and after the curriculum; group 2 performed only after the curriculum; group 3 performed only before the curriculum; the last did not perform any examinations.Results: 239 residents participated. The median global rating was 4 (interquartile ranges, 4-5). After the curriculum, an increasing number of residents integrated PoCUS into patient care (64 vs. 170, p=0.037) with acceptable image quality. No differences existed in the global ratings between 4 groups, however, nearly 30% of residents did not use PoCUS during their shifts. Group 1 performed more cases with better image quality and using ≥2 US applications (8/100 vs. 82/359, p=0.0009), compared with those before the curriculum. Comparing with those in groups 1 and 2, group 1 performed more cases (p<0.0001) with better image quality (p=0.03). “No suitable cases” (25/69), and “choosing other imaging priorities” (43/69) were the main feedbacks in groups 3 and 4.Conclusions: A short focused US training had an immediate positive impact on OSCE global ratings and integration of PoCUS of PGY-1 residents in the ED. The global ratings could not predict whether to use PoCUS in the ED practice. Although the working climate was designated to be friendly for utilizing US, a substantial percentage of residents did not use PoCUS. Further longitudinal studies would be needed for the sustained effects of the curriculum.Trial registration: NCT03738033.


Introduction
Point-of-care ultrasound (PoCUS), referring to "the stethoscope of the 21st century" [1], is de ned as ultrasonography performed by the physician at the bedside [2]. It provides real-time, dynamic images that physicians can correlate with the patient's presenting symptoms, serving as a valuable adjunct for patient care [3]. The use of PoCUS has been increasing robustly in emergency medicine (EM) in recent decades.
Proliferating PoCUS use and an evolving evidence base for different applications [4] have promoted the incorporation of ultrasound (US) training into the medical education of EM.
PoCUS education is a new scope of EM education. The Accreditation Council for Graduate Medical Education (ACGME) has recommended that all EM residents attain competency in the use of US [5].
Objective structured clinical examination (OSCE) is recommended by the American College of Emergency Physicians (ACEP) policy statement to assess for US competency [6]. However, the sub-competency in the EM Milestones document suggests that a resident should consistently utilize and integrate appropriate US applications into clinical management [7]. Little has been known about the actual use of PoCUS by the residents [8]. The results of the OSCE would not readily translate to the routine use of US in the clinical decision. Whether the seamless penetration into clinical practice could occur after a US curriculum in a high-volume emergency department (ED), is uncertain.
In Taiwan, after completing medical school, graduates must participate in the one-year postgraduate general medicine training as the rst postgraduate year (PGY-1) residents before they start a minimum of 3-year residency in a specialty. One-month ED training was included. Since 2013, a novel PoCUS curriculum was started for the PGY-1 residents in the ED of the National Taiwan University Hospital (NTUH).
This study aimed to investigate the clinical integration of PoCUS on shifts of the PGY-1 residents and the correlation with their OSCE performance after the US curriculum.

Study setting and design
The prospective observational study was conducted at the ED of the NTUH, a tertiary medical center in Taiwan, between July 2015 and October 2017. The ED has an annual uptake of approximately 85,000 patients. The study protocol was approved by the institutional review board of the hospital (201412004RIND) and registered at ClinicalTrials.gov (NCT03738033). Written informed consent was obtained from the participants.
Eight to 10 PGY-1 residents attended the ED training every month. Every PGY-1 residents had 18 eighthour working shifts during their training month and evaluated 10-15 non-critical patients under the supervision of the certi ed emergency physicians on each shift. A novel PoCUS curriculum including 30min didactics and 2-hour hands-on training on a live healthy model volunteer was implemented for the PGY-1 residents during the rst week. Because abdominal discomfort was the leading ED symptoms, the content of the curriculum included the extended focused assessment of sonography for trauma (eFAST) to detect intraperitoneal uid, pericardial effusion, pleural effusion, and pneumothorax, and sonography for urinary tract to detect hydronephrosis, gall bladder to detect acute cholecystitis, and abdominal aorta to detect abdominal aortic aneurysm. The ratio of the instructor to participant is less than 1:5. The instructors are the expert sonographers, board-certi ed by the Taiwan Society of Ultrasound in Medicine, and had over 10 years of experience in sonographic examinations. A US machine (SSA-660A, Canon, Japan) equipped with 2-5 MHz curvilinear transducers was used for training.
The PGY-1 residents completed the curriculum and received an OSCE for post-curriculum assessment immediately. It consisted of standardized questions for image acquisition and interpretation with points for technique, image quality, and correct interpretation of anatomy of FAST, kidney, gall bladder, and aorta. A global rating score using a Likert 5-point scale (unsatisfactory=1, needs improvement=2, satisfactory=3, high satisfactory=4, outstanding=5) [9] was given by the instructor at the scene and by the other instructor that did not involve in training through video review independently. The faces of the PGY-1 residents in the video were covered. The score given by the two instructors was averaged.
There were two US machines (SSA-550A, SSA-660A, Canon, Japan) equipped with 2-5 MHz curvilinear transducers kept ready in use in the ED clinics. Also, the reporting documentation was put in a plastic bag, along with the machine. The sonographic examinations the residents had performed clinically were obtained at the end of the ED training, including indication, scanning targets, sonographic ndings, sonographic diagnosis, and management. The accuracy of sonographic diagnosis was de ned as the agreement between the sonographic diagnosis and the discharge/admission diagnosis made by the attending physician. The images were reviewed by another two instructors not involving in training blindly and independently. The quality of the images was categorized using a 5-point Likert rating scale. Point 1 indicated no recognizable structures, no objective data can be gathered; point 2 indicated minimally recognizable structures but insu cient for diagnosis; point 3 indicated minimal criteria met for diagnosis, recognizable structures but with some technical or other aws; point 4 indicated minimal criteria met for diagnosis, all structures imaged well and diagnosis easily supported; point 5 indicated minimal criteria met for diagnosis, all structures imaged with excellent image quality and diagnosis completely supported [10]. Another instructor not responsible for the curriculum would interview the PGY-1 residents who did not perform clinical sonographic examinations after the curriculum and their feedbacks was obtained.

Selection of Participants
The PGY-1 residents attending the curriculum and completing the OSCE were included. Those attending the curriculum but not completing the OSCE were excluded.

Data collection
The demographic data of the PGY-1 residents were obtained, including age, gender, and prior US experience. The global ratings of the OSCE were collected, as well as the sonographic examinations including indication, scanning targets, sonographic ndings, sonographic diagnosis on shifts. Based on the clinical sonographic examinations, the residents could be categorized for 4 groups. The group 1 indicated the residents performed PoCUS during their shifts before and after the curriculum; the group 2 indicated that those performed PoCUS only after the curriculum; the group 3 indicated that those performed PoCUS only before the curriculum; the last group indicated those did not perform any examinations. The scanning targets were categorized into a single application (ex, FAST) and more than 2 applications (ex, FAST+gall bladder) according to the ACEP statement [6].

Outcomes
The primary outcome was the clinical integration of PoCUS on shifts of the PGY-1 residents and the correlation with their OSCE performance.

Statistical analysis
All data were analyzed by SAS software (SAS 9.4, Cary, North Carolina, USA). Categorical data were expressed in counts and proportions, while continuous data were expressed in medians and interquartile ranges (IQRs). Categorical variables were compared using a Chi-square test and ANOVA. Continuous variables were examined using Wilcoxon's rank-sum test. Intraclass correlation (ICC) with one-way random effects was used to assess inter-rater reliability for those global rating scores and imaging quality scores by two evaluators.
The linear regression models were applied to identify the factors associated with numbers and accuracy of the sonographic examinations after the curriculum. The polytomous regression models were applied to investigate the factors associated with the image quality of the sonographic examinations after the curriculum. The covariates included age, gender, prior US experience, and global ratings. Also, the factors associated with the global ratings were investigated using the polytomous regression models. The covariates included age, sex, and prior experience. A p-value of less than 0.05 was considered statistically signi cant.

Results
During the study period, 239 PGY-1 residents attended one-month ED training, and all of them (100%) participated in this study. There was a good ICC coe cient of 0.86 (95% CI, 0.68-0.96) for the OSCE global rating score and of 0.84 (95% CI, 0.63-0.95) for the image quality of the clinical sonographic examinations. Table 1 lists the demographic data of residents. The majority were novice sonographers. The median OSCE global rating score was 4. An increasing number of PGY-1 residents integrate PoCUS into patient care on their shifts after the curriculum (64 vs. 170, p = 0.037). Also, more increasing case numbers with better image quality and more accurate diagnoses were noted after the curriculum ( Table 2). After adjusting age, gender, and prior US experience, a higher global rating (coe cient, 1.38 ± 1.16, p = 0.03) was signi cantly associated with the better image quality of the sonographic examinations after the curriculum. However, there was no impact on the numbers of the cases and the accuracy.  Based on the clinical sonographic examinations, 4 groups of residents could be identi ed (Table 3). Among the groups, there were no differences in the global ratings. Group 1 residents performed more cases with better image quality after the curriculum, compared with those before the curriculum. Also, group 1 residents performed the examinations using ≥ 2 US applications after the curriculum (before vs. after, 8/100 vs. 82/359, p = 0.0009) (Fig. 1, Table 4). Comparing with the sonographic examinations after the curriculum in groups 1 and 2, group 1 residents performed more cases (p < 0.0001) with better image quality (p = 0.03) although the accuracy was unremarkable. Tables 4 and 5 list the US indication and scanning targets of the sonographic examinations in groups 1 and 2.    Sixty-nine residents in groups 3 and 4 did not perform US clinically after the curriculum. "No suitable cases for examinations" (25/69), and "having other imaging priorities" (43/69) was the main feedbacks of these residents.

Discussion
While clinical integration is essentially addressed during PoCUS education, little has been known about the actual use of PoCUS by the PGY-1 residents after training [8]. This study provides evidence regarding the real US performance of the inexperienced PGY-1 residents after a short US curriculum. The results showed the US curriculum had an immediate positive effect on OSCE performance and a substantial effect on increasing the clinical integration of PoCUS. A signi cantly increased number of PGY-1 residents used PoCUS during clinical practice after the curriculum. However, nearly 30% of them did not use PoCUS although their OSCE performance was good. It revealed that the global ratings had no obvious correlation to the clinical integration of PoCUS.
Time constraints would be a barrier for ED medical education. Previous studies suggested that short courses of focused US training were of acceptable quality, even in paramedics with no prior ultrasound experience [11][12][13]. Also, US skills would decay at different times for different organ images such as a signi cant decline of the ability to identify A-lines of lung over time, but no decline in cardiac image recognition [14]. The content of the US curriculum was designated as the combination of unique abdominal US techniques because abdominal disorders were frequently encountered in our ED.
The introduction of the OSCE showed comparable improvements in trainee performance and understanding feeling con dent in their ability to utilize US [7]. There is evidence that global rating scales capture diverse levels of pro ciencies better than checklists, and are easy for examiners to use [15]. Also, global ratings were shown to yield reliable data when compared with checklist scores for assessment of residents in the eld of medicine [13,16]. The skills needed for US are complex and include hand-to-eye coordination, image optimization, and image interpretation [17], and clinical correlation. Therefore, this study used global ratings for OSCE performance instead of checklist scores due to the psychometric properties [16] and higher levels of expertise on US performance.
There were different patterns of the behavior of doctors after US training. It could be observed that the US curriculum helped the inexperienced doctors to integrate PoCUS into the clinical practice in groups 1 and 2. The residents used self-formatted combinations of US techniques that the FAST and the renal US were the most common contents. More than 70% of residents had a desire to change the behavior to be willing to integrate PoCUS into daily practice. Although residents in both groups had progressed, group 1 residents seemed to be more active and motivated in performing sonographic examinations, compared with those in group 2.
The ED is a unique learning environment different from any other setting in the hospital. The high volume and acuity of patients in the ED provide opportunities to repeatedly practice and to facilitate the accumulation of clinical experience for young staffs, crossing the gap from the level 2 (learning level) to the level 3 and 4 (behavior and results levels) easily [18]. However, the characteristics of ED patients that each resident encountered are diverse and varied. The time spent with an individual patient is limited. In our ED, the PGY-1 residents could evaluate 200-250 patients during the training month. Although the number of scanning cases performed by the PGY-1 residents increased after the curriculum, the median was 3 cases. It implied that the PGY-1 residents used PoCUS in highly selected, maximally bene cial patients.
The potential barriers exist for utilizing PoCUS in clinical settings. The frequent challenges are US equipment and lack of con dence or experience in clinical integration skills [8,19]. Also, the lack of evidence for clinical utility for certain patients, a trade-off between performing US and other clinical processes, and the inability to use US for decision-making were reported [8,20]. In our study, the facility was settled that the US machines were easily accessible. All of the sonographic examinations were under the supervision of the certi ed ED physicians. The working climate was designated to be friendly for using US, but some barriers still existed. As the feedbacks from groups 3 and 4, the residents could choose other imaging modalities such as computed tomography or magnetic resonance imaging for patient care, whatever autonomously or by the attending physician's suggestion. It implied the gap between PoCUS and comprehensive imaging modalities would be hardly crossed in a time-limited and stressful environment.
This study had some limitations. First, it was conducted in an academic medical center with an active US training. External generalization would be further explored. Although the distribution of ED patients varies from hospital to hospital, however, this study provided novel evidence regarding how inexperienced residents integrate PoCUS in their primary care after a tailored educational curriculum. The design and contents of the curriculum were based on the frequently encountered complaints in our ED, and more than 70% of residents used the techniques in their clinical practice. Second, this study investigated the immediate effects of the US curriculum. Insights into long-term retention of US skills and subsequent time to skill degradation were not evaluated. Future longitudinal studies would be needed for these residents for their specialties after one-year general medicine training, especially in groups 3 and 4. Whether the specialty would have an impact on the clinical integration of PoCUS was wondered. Third, the sonographic examinations were based on written documentation. The percentage of the residents utilizing US would be under-estimated if they performed the sonographic examinations but the results couldn't be seen in the paper records or the electronic medical records. Last, a more structured survey for groups 3 and 4 would be needed for a more detailed understanding of the reasons for not using PoCUS clinically.

Conclusions
A short focused US training had an immediate positive impact on OSCE global ratings and integration of PoCUS of PGY-1 residents in the ED. The global ratings could only evaluate whether the procedure meets the criterion, but could not predict whether to use PoCUS in the ED practice. Although the working climate was designated to be friendly for utilizing US, a substantial percentage of residents did not use PoCUS.
Further longitudinal studies would be needed for the sustained effects of the curriculum.

Declarations
Ethics approval and consent to participate: The study protocol was approved by the institutional review board of the ethics committee of the NTUH (201412004RIND). Written informed consents were obtained from the participants.
Consent for publication: Written informed consents were obtained from the participants.
Availability of data and materials: All data generated or analysed during this study are included in this published article.

Figure 1
The numbers of sonographic examinations performed by group 1 residents. They used more than 2 applications after training (black columns).