1. Program Settings and Participation
Moore’s Level One: The Cancer ECHO Program Participation
In the first year of Cancer ECHO (from September 17, 2019, to September 16, 2020), 22 Cancer ECHO sessions were held. Each typical 1.5-hour session consists of a 20-minute didactics and a 1-hour case discussion. There was a total of 22 didactics delivered by the Hub team with a blend of topics in screening, prevention, and survivorship. All but one of the 17 case discussions were regarding cancer survivorship. 147 unique individuals attended the Cancer ECHO at least once during this period. Among the 147 attended individuals, 16 (10.8%) were PCPs (physicians and advanced practice providers). There was an average of 2.5 PCPs (17.2%) and 12 other professionals participating in each session. 90.9% of sessions had at least one PCP spoke participant. The most common non-PCP attendees were cancer educators, navigators, public health workers, or administrators.
Study participation
We interviewed 15 program participants (12 Spokes and 3 Hub members) and 7 PS. Among them, 5 Spoke, and 6 PS were PCPs. 90.9% sessions had at least one of the 12 Spoke interviewees, and 63.6% of sessions had at least one of our 5 Spoke PCP interviewees. We surveyed 14 Spoke and 16 PS members. 9 of the survey respondents participated in interviews. Additional file 1 summarized the demographics of the interview and survey participants.
2. Moore’s Level Two: Satisfaction
14 Spokes rated their satisfaction with the program in the survey on a scale of 1 to 5 (1=far short of expectations, 2=short of expectations, 3=equals expectations, 4=exceeds expectations, 5=far exceeds expectations). The program met 100% of and exceeded 88.9% of both of PCPs and non-PCPs’ expectations. The overall satisfaction from 11 program participants was 3.9 ± 0.7. (See Table 1)
Table 1
Survey Results of The Self-Reported Outcomes
Framework
|
Role
|
Overalla: Mean (SD)
|
PCPa: Mean (SD)
|
Non-PCPa: Mean (SD)
|
Range
|
Moore’s Level 2b
|
Satisfaction
|
Spoke
|
3.9 (0.7)
|
3.8 (0.9)
|
4.0 (2.9)
|
1-5
1 = far short of expectations
5 = far exceeds expectations
|
Moore’s Level 3b
|
Knowledge Improvement
|
Spoke
|
4.3 (0.8)
|
3.8 (0.8)
|
4.5 (2.9)
|
1-5
1 = definitely not
5 = definitely yes
|
PS
|
3.7 (1.1)
|
2.8 (0.7)
|
4.1 (2.1)
|
Moore’s Level 4b
|
Confidence Improvement
|
Spoke
|
3.7 (1.1)
|
3.8 (0.8)
|
3.7 (1.5)
|
PS
|
3.1 (1.0)
|
2.6 (0.7)
|
3.6 (1.6)
|
Moore’s Level 5b
|
Practice Improvement
|
Spoke
|
3.7 (1.1)
|
4.0 (0.8)
|
3.6 (1.6)
|
PS
|
3.3 (1.0)
|
2.8 (0.7)
|
3.8 (2.0)
|
Maslach Burnout Inventory (MBI)c
|
Emotional exhaustion
|
Spoke
|
4.2 (1.3)
|
4.0 (0.8)
|
4.3 (1.8)
|
1-7
1 = strongly improved
7 = strongly worsened
|
PS
|
4.3 (1.3)
|
4.0 (1.1)
|
4.2 (1.6)
|
Insensitivity
|
Spoke
|
3.3 (1.6)
|
3.3 (1.0)
|
3.3 (1.8)
|
PS
|
3.3 (1.4)
|
3.2 (1.6)
|
3.3 (1.4)
|
aThe sample size (N of Spoke =14, N of PS =16) was too small to demonstrate any statistical significance. |
bMoore’s Evaluation Framework for Continuing Medical Education. |
cMaslach Burnout Inventory 2 single-items adapted measures. |
PCP, primary care provider; PS, potential spoke. |
Overall, the interviewees were satisfied with the program. Of the 12 Spoke (S) and 3 Hub (H) members we interviewed regarding why they liked the program, 6 (4 PCPs) interviewees mentioned the conversational format, 3 (2 PCPs) connecting to the multidisciplinary community, and 5 (3 PCPs) the real-world experiences they gained from the program. In terms of the aspects of the program that did not meet their expectations, Spoke 4 [MD] said “it just didn't seem as engaging.” The Hub members also shared their side of challenges. H2 [MD]: “We're having a tough time getting spoke sites to find good cases to present.”
3. Moore’s Level Three to Five: Effects on Knowledge, Confidence, and Practice
14 Spokes and 16 PSs self-rated their knowledge, confidence, and practice changes from any source during the period of the first year of the Cancer ECHO implementation (see Table 1). On a scale of 1 to 5 (1 = definitely not, 5 = definitely yes), the Spokes had higher ratings in their improvements in knowledge, confidence, and practice than the PS group (4.8 versus 3.7, 3.7 versus 3.1, 3.7 versus 3.3, respectively). Overall, the differences between Spokes and PSs were bigger among the PCPs (with an average increase of 1.1 points) than the non-PCPs (with an average increase of 0.1 points) for all three aspects. Of note, none of these differences were statistically significant.
S11 [MD] said “knowledge change, that is tremendous.” S12 [MD] said, “I can say that my knowledge, confidence has increased, has increased dramatically.” Their practice benefited from the program by having more options and developing a deeper empathy of what their patients were going through. S6 [FNP] shared how her practice changed through presenting a case discussion, “I think it's refreshing to hear a different perspective. So, in the case I actually presented…it was very challenging, and I was prepared to just say, OK, we've done all we can with this person, and now that the ball's in their court. But after talking with several people who participated [in the Cancer ECHO] that week, they really empathized and really brought forward potentially other barriers and other factors that I had not really considered. So, ...when I left that presentation, made me think, OK, I need to be more gracious and more patient, and stick with this case instead of at this point just kind of being done. So that was helpful.”
There are three themes that emerged from the interviews that described the strengths of the Cancer ECHO, which were consistent with the three themes from the interview answers for satisfaction. Table 2 shows quotes representative of these themes. Most of the participants highlighted their changes from learning from other participants, exploring more care options through collaboration, adapting the new resources and ideas into practice, and having better communications with patients. Some interviewees, especially PCPs, disproportionately favored the case discussion over the didactic component. Some interviewees did not think the Cancer ECHO changed their practice yet, citing several reasons: (1) their participation in the program was still too limited to see changes, and they faced barriers to maintain high participation, such as time commitment; (2) some of the topics were not directly pertinent to their practices; (3) since the learning happens from discussion with other participants, the quality of the program was related to the program participants. H3 [MD] mentioned: ‘I don't think we are capturing primary care providers.’ The Spoke PCPs thought the didactics were less effective, which is consistent with the perspectives of PS PCPs that didactics were less engaging.
Table 2
Emerged Themes of the Program Strengths in the Qualitative Results
Themes
|
Interviewees
|
Sample quotes
|
Format:
Conversational and interactive (in the case discussion component)
|
Providersa: 12 out of 13 (4/5 Spoke, 6/6 PS, 2/2 Hub)
Non-Providersa: 7 out of 9 (Except S3 [MPH] likes didactics better, S9 [CHES] likes didactics and case discussion equally)
|
Spokes: 9 out of 12
Potential Spokes: 7 out of 7
Hubs: 3 out of 3
|
S6 [FNP]: “It's been good to hear the (didactics) presentations, which is what I initially thought was gonna be the most helpful. But actually, the case presentation and discussion component where you have a question and answer has been surprisingly more beneficial, I think, at times and help create change to my practice.”
PS6 [DDS]: “It just seemed really interactive. ”
|
Content:
Real-world experiences
|
Providersa: 13 out of 13 (5/5 Spoke, 6/6 PS, 2/2 Hub)
Non-providersa: 7 out of 9 (Except S3 [MPH] likes didactics better, PS2 [LCSW] did not mention this theme)
|
Spokes: 10 out of 12
Potential Spokes: 6 out of 7
Hubs: 3 out of 3
|
S5 [MD]: “I probably like the case discussions the best, honestly, just because I think when you start with the case, it triggers a memory in my mind as a patient who is like that or something. Whereas if it's just a didactic session, it's helpful and nice, but it's harder to put into context. It's easier for me to lose interest.”
PS7 [MD]: “I like the idea of having a case presented, which is great.”
|
Participant and community:
A nonjudgmental, safe learning environment and support from a multi-disciplinary community
|
Providersa: 13 out of 13 (5/5 Spoke, 6/6 PS, 2/2 Hub)
Non-providersa: 8 out of 9 (Except S7 [MCHES] did not mention this theme)
|
Spokes: 11 out of 12
Potential Spokes: 7 out of 7
Hubs: 3 out of 3
|
S11 [MD]: “So that's what I like about the ECHO, and they have a different perspective, and it's not just physicians. It's either a social worker, which I rarely hear in practice…and you get to understand how important they are, or your physical therapists or your rehab people. The nurse you see most of the time, but we don't really get their perspective on what's going on. That makes it more of a complete, almost team effect on the patient.”
PS4 [MD]: “Everyone on is invited to talk, there is no judgment.”
|
aProviders include the primary care providers and specialist physicians in the Spoke, Potential Spoke, and the Hub groups. |
CHES, Certified Health Education Specialist; DDS, Doctor of Dental Surgery; ECHO, Extension for Community Healthcare Outcomes; FNP, Family Nurse Practitioner; MD, Doctor of Medicine; MPH, Master of Public Health; MCHES, Master of Certified Health Education Specialist; PS, potential spoke; S, spoke. |
4. Additional Measurement of the Effects on Burnout
We used changes in emotional exhaustion and insensitivity to evaluate the program impact. (See Table 1) Though many PCPs did not report significant emotional exhaustion changes, they did speak highly of the program's potential to improve their burnout levels with features that contribute to the program effectiveness in Table 2. As H3 [MD] described, “medicine is draining when you feel like you have nothing else to offer.” S5 [MD] thought having a safe place, such as the Cancer ECHO program, to discuss the mistakes they have made or dealt with could help emotional exhaustion. Our study participants felt the program was helpful for them to empathize with patients and be more understanding and sensitive to what their patients were going through.