Serious Illness Communication Skills Training for Emergency Physicians and Advanced Practice Providers: A Multi-Method Assessment of the Reach and Effectiveness of the Intervention

Background EM Talk is a communication skills training program designed to improve emergency providers’ serious illness conversational skills. Using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework, this study aims to assess the reach of EM Talk and its effectiveness. Methods EM Talk is one of the components of Primary Palliative Care for Emergency Medicine (EM) intervention. It consisted of one 4-hour training session during which professional actors used role-plays and active learning to train providers to deliver serious/bad news, express empathy, explore patients’ goals, and formulate care plans. After the training, emergency providers filled out an optional post-intervention survey, which included course reflections. Using a multi-method analytical approach, we analyzed the reach of the intervention quantitatively and the effectiveness of the intervention qualitatively using conceptual content analysis of open-ended responses. Results A total of 879 out of 1,029 (85%) EM providers across 33 emergency departments completed the EM Talk training, with the training rate ranging from 63–100%. From the 326 reflections, we identified meaning units across the thematic domains of improved knowledge, attitude, and practice. The main subthemes across the three domains were the acquisition of discussion tips and tricks, improved attitude toward engaging qualifying patients in serious illness (SI) conversations, and commitment to using these learned skills in clinical practice. Conclusion Effectively engaging qualifying patients in serious illness conversations requires appropriate communication skills. EM Talk has the potential to improve emergency providers’ knowledge, attitude, and practice of SI communication skills. Trial registration: NCT03424109


Introduction
More than half of seriously ill older adults visit the Emergency Department (ED) in the last six months of life. 1,2It is estimated that between 50 and 60 percent of seriously ill older adults do not have advanced directives 3,4 and are at risk of receiving care inconsistent with their wishes. 5The ED presents an opportunity to engage these patients in discussions focused on goals of care, advanced directives, and willingness to obtain hospice and palliative care.
][8] Emergency Medicine (EM) providers particularly tend to avoid such conversations as they are more likely to assume that they are better suited to provide life-prolonging interventions and providers of other specialties are better equipped to handle such conversations. 9ducational interventions aimed at improving physicians' communication skills in SI conversations gained increasing attention about two decades ago, 10 with the VitalTalk being one of the most commonly used educational training models. 11,12A few specialty-focused adaptations of the VitalTalk had emerged over time such as OncoTalk for oncology providers, 13,14 Geritalk for geriatric providers, 15,16 and EM Talk for emergency providers. 177][18] Earlier studies have reported that the OncoTalk training was associated with a substantial increase in oncology providers acquiring new skills in delivering bad news and transitioning qualifying patients to palliative care. 18,19Additionally, the Geritalk training has been associated with substantial improvement in self-reported preparedness and practice of engaging in SI conversations. 20,21like medical specialties with a controlled patient-provider environment like primary care and oncology, navigating SI conversations in the ED environment requires additional skills in engaging patients and caregivers in a fast-paced environment while maintaining patient privacy.Furthermore, EM providers are required to demonstrate competency in interpersonal and communication skills including delivering bad news and resolving con icts, consistent with the guidelines from the Accreditation Council of Graduate Medical Education. 22With EM Talk focused on providing communication skills training as part of the educational intervention of a multi-center National Institute of Health grant, 23 it is unknown how effective the educational intervention improved the knowledge, attitude, and practice of EM providers.
5][26] The two-decade-old RE-AIM framework is a planning and evaluation tool commonly used to assess project implementation across clinical, public health, and health behavior-focused research. 24For this study, we focused on assessing the intervention's reach -de ned as the absolute number of persons who participated in the intervention, and the intervention's effectiveness -de ned as the impact of the intervention on individual outcome measures. 24Therefore, the aim of this study, is to assess the reach of EM Talk and its effectiveness in improving knowledge, attitude, and practice among EM providers.

Study Design
We employed a multi-method approach to assess the reach and effectiveness of the EM Talk intervention in providing SI communication skills for full-time EM physicians and advanced practice providers (hereafter referred to as EM providers).The advanced practice providers involved in SI communication skills training were those involved in the care of high-acuity patients.Consistent with this multi-method research design, 27 the reach of the intervention was assessed quantitatively using a cross-sectional study design while the effectiveness was assessed qualitatively using a conceptual content analytical design. 24We de ned the reach of the EM Talk intervention as (1) the absolute number and proportion of representative EM providers across each participating ED that obtained the SI communication skill training, (2) the estimated number of seriously ill patients potentially reached, and (3) the estimated yearly number of patients each trained provider will reach across each ED.We de ned the effectiveness of the EM Talk intervention as the self-reported thematic domains of improved knowledge, attitude, and practice of SI communication skills.The unit of analysis of the quantitative and qualitative studies was at the institutional and individual levels, respectively.This study followed the Standards for Reporting Qualitative Research (SRQR) guideline. 28

Study Population
The study population was full-time EM providers across 33 EDs enrolled in the Primary Palliative Care for Emergency Medicine (PRIM-ER) study.The PRIM-ER study is a cluster-randomized pragmatic trial that assesses the impact of EM provider interventions on healthcare utilization and outcomes among seriously ill older adults that visit the ED. 23The PRIM-ER intervention consists of (1) education in palliative and end-of-life care EM providers and emergency nurses, (2) communication skill training and simulation workshop for EM providers (using the EM Talk training) and emergency nurses (using the End-of-Life Nursing Education Consortium (ELNEC) training), and (3) the integration of a clinical decision support tool to identify and engage seriously ill older adults in SI conversations.We had reported the reach of the ELNEC intervention and emergency nurses' perceived barriers and solutions to conducting SI conversations in the ED. 29The current study focuses on the reach and effectiveness of EM Talk.

EM Talk Intervention
EM Talk was a one-day 4-hour SI communication skill training session, delivered both in-person and virtually.Consistent with our cluster-randomized stepped wedge design, 23  The rst half of the session comprised large group lectures and the second half of the session focused on small group practice sessions on skills in delivering bad news and discussing goals of care and re ective exercises.Each session was facilitated by two VitalTalk-trained personnel.Details of the EM Talk course description have been published earlier. 17

Quantitative Data Analysis
We obtained administrative data from each ED of the PRIM-ER study and the Centers for Medicare & Medicaid Services (CMS).Using the administrative data, we computed the counts of the EM providers that completed the EM Talk training and generated the sum of EM providers in each participating ED.Using the CMS data, we generated the yearly number of seriously ill patients that visit the ED by computing the mean of the number of qualifying seriously ill patients that visited the participating sites between 2018 and 2020.A qualifying seriously ill patient is a patient, 66 years or older, that visited the ED within the study period with a life-limiting illness identi ed using a GAGNE index (a measure of one-year mortality) greater than 6. 23,30 We de ned the proportion of EM providers trained as the number of EM providers trained divided by the total number of EM providers in the participating EDs (Table 1).We de ned the estimated number of seriously ill patients as the yearly average number of qualifying seriously ill patients in each ED multiplied by the proportion of EM Talk-trained providers in the ED.We de ned the yearly seriously ill patient and EM provider ratio as the average yearly average of the seriously ill patients reached divided by the number of EM providers trained.

Qualitative Data Analysis
Consistent with a conceptual content analytical approach, we identi ed codes that fell into three a priori-de ned thematic domains of improved knowledge, attitude, and practice.The knowledge, attitude, and practice (KAP) theory, a commonly used theoretical model to assess behavior change, divides the steps in behavioral change into knowledge acquisition, belief and attitude generation, and practice creation.We selected the KAP theory as the conceptual model to assess the effectiveness of the EM Talk intervention since the intent of the intervention was to equip EM providers with communication skills (knowledge acquisition), create a simulated practice experience (attitude generation) so that they can effectively engage seriously ill patients on discussions around goals of care (practice creation).
Data for the qualitative analysis was from three open-ended questions in the EM Talk post-training survey -designed consistent with the requirement of continuing medical education assessment (Appendix 1). 31The questions are as follows: (1) In the space below, please re ect on your personal experience with this educational intervention; (2) What changes will you make to how you identify patients and/or family members who may be ready to discuss goals of care and/or palliative care options?(3) What changes will you make to how you counsel patients and/or families about end-of-life care?Responses to these questions were optional and were pre xed as "A", "B", and "C".
Using each respondent's sentences as the unit of analysis, the coding team, made up of three coders (two males (OA, AG), one female (RK), all MDs), inductively and deductively identi ed codes and meaning units after an initial textual immersion. 32A codebook was generated after analyzing the responses to the rst open-ended question and the codebook was continuously modi ed throughout the analytical phase (Table 2). 33Each coder coded the qualitative data pool independently and nal codes were agreed upon through voice voting during coding and debrie ng meetings.After an initial round of coding (open coding), the coding team performed focused coding, during which the initial codes were merged and recategorized.Meaning units (exemplary sentence or phrasal codes) were generated from the sentences through the use of in-vivo, structural, and process coding techniques, and their counts were reported in tables. 34Subthemes were identi ed by pooling codes with similar meaning units. 32Synonym rule: For items that have implicit meanings, a synonym of the anchor word or phrase is applied and the sentence is re-assessed and categorized as either "yes, code is exact", "no, code is different", or "close code but not exact".
We employed several methods to ensure methodological and interpretive rigor.To ensure credibility, the coding team reported the nal codebook created after a series of debrie ng and coding meetings. 35The open-ended questions that informed the responses provide information on the dependability of the study and the details of the study participants and the source of data provide information on the transferability of our ndings. 36By reporting the counts of the meaning units of each theme and using quotes from the participants to explain the thematic domain, we ensured the con rmability of the study. 37man Subject Concern Across the 879 EM providers who completed the survey, we 117 open-ended responses (Fig. 1).Sentences from 60 respondents were coded under the improved knowledge domain while sentences from 45 and 25 respondents were coded under the improved attitude and improved practice domains, respectively.With some sentences producing multiple codes across the thematic domains, the code counts exceeded 117 (Table 4).

Improved Knowledge
The theme of improved knowledge was referenced by 60 respondents.The most common subthemes that emerged from these responses were the acquisition of tips and tricks of SI communication (n = 44) and acquired general yet useful knowledge (n = 14).A less common subtheme was the development of empathy skills (n = 3) (Table 5).

Acquired tips and tricks for SI communication
A majority of respondents acknowledged that they "learned some really valuable tools" (B254) and that "the tips and tricks provided were concise and therefore relatively easy to remember with regular practice/use" (B256).One provider recounted: "I did learn some helpful skills that I will try to bring into my practice."(A64) To these trained EM providers, the SI communication skills taught in the course were viewed as "techniques to talk to the family of palliative patients" (A35).One provider highlighted the importance of this skill based on the frequency of contact with SI patients and their caregivers in the EDs: "This was a useful educational intervention to ED providers who often have to have end-of-life discussions with patients and families in an emergent setting."(A34)

Acquired general useful knowledge
In contrast to EM providers that speci ed speci c skills the EM Talk provided, some providers reported a general improvement in their knowledge of palliative care.For some, the training was "a pretty good learning experience for me" (C314) while another provider feels the training "really helped me grow as a provider" (B194).A provider shared: "I learned more than I thought I would, made me think about these issues more than I had before."(A49)

Acquired empathy skills
The acquisition of empathy skills was the least identi ed subtheme, with one provider identifying empathy skills as a takeaway from the training.
"Learned a lot about empathetic skills that I can use in daily practice" (A81)

Improved Attitude
The theme of improved attitude was present in 46 responses.The most common subtheme that was identi ed was improved attitudes toward engaging in hospice and palliative care discussions (n = 30).Less frequently identi ed subthemes included attitude towards improving patient care (n = 10) and attitude towards receiving future training on SI conversations (n = 5) (Table 5).

Improved attitude toward engaging in SI conversations
The improved attitude towards engaging in SI conversations referred to being "more comfortable and at ease with end-of-life conversations" (B190).For some EM providers, the training helped them "realize the importance of having discussions with family early/often regarding goals of care for their loved ones" (B188).
Some EM providers, however, discussed the deliberate attempt of the EM provider "to slow down and listen to your patients and family members" (B166).The importance of being intentional about listening was stated by one of the EM providers: "Patients end up being more satis ed when you listen and they feel as if their needs and concerns are being addressed" (C311) A few EM providers stated that the training helped increase the motivation to engage in SI conversations.For example, one provider wrote about negative past experiences and how the course made them feel more con dent with such conversations: "…due to time constraints and some negative patient interactions regarding the goal of care discussions, I was initially resistant but now motivated and optimistic in my ability to navigate these talks" (A63).

Improved attitude toward patient care
Improved attitude towards patient care referred to the EM providers "see(ing) the value [the training] brings to patients and their families" (B208).The training provided an opportunity for self-re ection and assessment with one provider stating that "I identi ed various areas in which I can improve not only my communication in end-of-life discussions but also with all my patients" (B258).The awareness of how the training may improve patient care served as a motivation for some EM providers to practice SI conversations.
"[The course] pushed my comfort level with these discussions and has motivated me to practice and improve (A127)."

Improved attitude towards future training on SI conversations
A few EM providers re ected on the EM Talk training and stated that "[the training] is extremely applicable to our practice.I would recommend all EM doctors undergo training such as this" (B152).Other EM providers referred to the effectiveness of the small group discussion format and the ability to download the VitalTalk app for future reference.
"I had a great time in the small groups practicing di cult conversations.I also was happy to get the app downloaded to keep some very useful tools on hand" (A84).

Improved Practice
The theme of improved practice was referenced by 25 respondents.The majority of these re ected the subtheme of commitment to using acquired skills in clinical practice (n = 20) while a minority of respondents (n = 5) stated that were already utilizing taught skills in clinical practice (Table 5).

Commitment using acquired in clinical practice
The to acquired skills in clinical practice was indicated by providers who shared a plan to "incorporating this style of talking about goals of care with my patients and families" (A63).A provider acknowledged the ease of acquiring SI conversation skills and that it might take some time for the skill to become second nature.
"It [the training] was interesting and the tool is easy to follow so it should be easy to incorporate into practice.I suspect it will be more comfortable with time and eventually become second nature" (A82).

Already utilizing taught in clinical practice
A few EM providers expressed that, between the training completion and survey completion, they had been in clinical scenarios where they had to use some of the SI conversational skills taught.One provider stated that "I feel better about approaching end-of-life discussions and have had some success in my recent practice" (A69).Also, another provider attributed the success in navigating SI conversations he recently experienced to the training he received.
"The very next day I had a patient/family interaction that I was able to identify and navigate because of the training" (A104)

Discussion
We report that across the 33 EDs enrolled in the PRIM-ER study, over 85 percent of the EM providers completed the EM Talk training and we estimate that these trained providers will reach approximately 16,389 seriously ill older adult patients that visit the ED.Also, we report that across the thematic domains of improved knowledge, attitudes, and practice, the EM providers reported that the training improved their SI communication skills, improved their attitude towards engaging qualifying patients in SI conversations, and encouraged their commitment to using these learned skills in clinical practice.
The extensive reach of the EM Talk training is noteworthy and this success is the re ection of the commitment of the departmental leadership of each site and their willingness to integrate the training into the educational curriculum in their departments.Also, we selected EM physician champions that were tasked with disseminating the information about the EM Talk training and facilitating attendance.The selection of appropriate and in uential clinical champions is pivotal to the successful engagement and training of providers.Earlier studies have reported that clinical champions are instrumental in the quicker initiation of interventions, assist in overcoming institutional barriers, and can motivate and involve staff in clinical trials. 39,40 report that every year, a trained EM provider will reach between ve and 115 seriously ill older adult patients depending on the ED volume, patient mix, and geographic setting, among other factors.2][43][44] Engaging in SI conversations is never an easy task, and creating an enabling environment within the ED for EM providers to engage in such conversations may lighten the burden of delivering bad news and engaging patients in end-of-life goal discussions.
Earlier studies have reported that some of the barriers EM providers and emergency nurses face in engaging qualifying patients in SI conversations include lack of privacy, limited patient engagement time, and the fast-paced ED work culture. 9,29,45,46 Talk was designed to provide SI communication skills training to EM providers.Consistent with the goal of the intervention, the EM providers reported that they acquired tips and tricks in navigating SI conversations, are willing to engage qualifying patients in SI conversations, and have the intent to incorporate these learned skills in clinical practice.The observed harmony between the expected goal and self-reported outcome may be explained by the evidenced-based pedagogical technique employed in delivering the EM Talk training.VitalTalk -the parent program from which EM Talk emerged has consistently prioritized role play and small group learning sessions as a bedrock of successful training sessions.11,12 Similarly, other authors that taught GeriTalk -another derivative of the VitalTalk, reported that Geriatric and Palliative Medicine fellows had high levels of satisfaction after they underwent the training.16,20 Similarly, Berg et al. 47 reported that Oncology fellows self-reported signi cant improvement in SI communication skills after undergoing OncoTalk training.
This study has its limitations.Although a large proportion of full-time EM providers completed the training, it is unlikely that all EM providers will embrace and utilize the SI communication skill in their practice.The estimated average of seriously ill patients that would be reached yearly, therefore, represents the best-case scenario.On the other hand, EM providers may learn from one another and the training and knowledge may even spread to those who are not formally trained-i.e., adoption of behavior due to peer in uence.There is a possibility that attitude and practice towards engaging qualifying patients in SI conversations will differ by age, race/ethnicity, religious a liation, and years of practice.Third, differences in the pedagogical styles of the different facilitators may positively or negatively in uence the knowledge, attitude, and practice of EM providers toward engaging qualifying patients in SI conversations.Also, the EM Talk started as an in-person training program but was transitioned into an online training session.It is unknown to what extent this change in pedagogy affects the reach and effectiveness of the intervention.Despite these limitations, this study is one of the few that assessed the effectiveness of EM Talk training across the domains of knowledge, attitude, and practice.Also, this is one of the few studies that used the RE-AIM framework to assess the reach and effectiveness of a provider-focused intervention.Furthermore, this study is strengthened by its spread across over 30 EDs and its large sample size. Figures EM Talk training occurred sequentially across 33 EDs for three years (2019 to 2021).Before each training session, an EM Champion -an in uential EM provider, was selected to encourage and mobilize EM providers for the training and organize the training logistics in his or her ED.
Within a week after the SI communication skill training, EM providers completed a selfadministered post-training survey and received a ve-unit continuing medical education (CME) credit and a $67 gift card for their time.

Table 1
Statistical De nitions of the Reach of the Primary Palliative Care for Emergency Medicine (PRIM-ER) study

Table 2
Exclude if the statement refers to the course and does not re ect on individual or group improved clinical practice or skill acquisition.For implicit coding: Exclude "close code but not exact" and "no, code is different" after applying the synonym rule

Table 3
Ethical approval was obtained from the New York University Grossman School of Medicine Institutional Review Board total of 879 out of 1029 EM providers (85%) completed the EM Talk training (Table3).The proportion of EM providers that had the training across the 33 EDs ranged from 63-100%.Between 2018 and 2020, a total of 2,698,198 unique patients, 66 years and older, visited the 33 EDs at least once.Of this population, the number (and proportion) of unique seriously ill patients (GAGNE score > 6) was 57,136 (2.1%).The yearly average of seriously ill patients across the 33 EDs was 19,045.We estimated that 16,389 seriously ill patients would have been reached across all 33 EDs, assuming a 100% practice rate among the trained EM providers.Also, we estimated that every year, one trained EM provider will reach an average of 19 qualifying seriously ill patients and the number will vary from 5 to 115 across the 33 EDs.Reach of the EM Talk Training Across the Participating Emergency Departments (ID: i18-00607) and the PRIM-ER study protocol is reported on ClinicalTrials.gov(NCT03424109).38ResultsQuantitativeResults: Reach of Intervention A Average SI Patients Qualifying Index Visits: Number of patients 66 years and older with an index ED visits who had a GAGNE index of six or higher.The average is calculated by dividing 2018, 2019, and 2020 counts by 3.Estimated SI Patients Reached/Year = Percent Trained * SI Patients Qualifying Index Visits; Yearly SI Patient: EM Provider ratio = Estimated SI Patients Reached/Number of EM Providers Trained; OSU; Ohio State Univerisy: UF: University of Florida; UCSF: University of San Francisco; NYU: New York University; Univ of Penn: University of Pennsylvania Average SI Patients Qualifying Index Visits: Number of patients 66 years and older with an index ED visits who had a GAGNE index of six or higher.The average is calculated by dividing 2018, 2019, and 2020 counts by 3.Estimated SI Patients Reached/Year = Percent Trained * SI Patients Qualifying Index Visits; Yearly SI Patient: EM Provider ratio = Estimated SI Patients Reached/Number of EM Providers Trained; OSU; Ohio State Univerisy: UF: University of Florida; UCSF: University of San Francisco; NYU: New York University; Univ of Penn: University of Pennsylvania

Table 4
*Themes in bold; **Multiple coding categories across subthemes account for the sum exceeding the total

Table 5
Apriori themes, emerged subthemes and the associated meaning units