Study Design and Participants
We conducted a cross-sectional study of attendees at the 27th Mayo Clinic Internal Medicine Board Review. This week-long course is a high-yield intensive program designed to assist learners with the American Board of Internal Medicine (ABIM) Initial and Recertification Examinations and to provide a relevant review for daily practice. Historically, most attendees are physicians who will be taking one of these examinations. The Mayo Clinic Internal Medicine Board Review offers 52.5 category 1 CME credits and consists of 57, 30-60 minute podium presentations. This setting was selected due to its generalizability in CME.
Mayo Clinic faculty members are selected to present based on their topic-related expertise. Course content is determined by a planning committee of generalists, specialists, and course directors. Learning objectives are given to each presenter by the course directors. This study was deemed exempt by the Mayo Clinic Institutional Review Board.
Learner Engagement Instrument (LEI) Development
A Learner Engagement Instrument (LEI) was developed to evaluate learners’ engagement in CME. To provide content validity, study authors and experts in the field [C.R.S., S.L.B., A.P.S., L.W.R., T.J.B., C.M.W.] reviewed conceptual frameworks on engagement and previously published engagement instruments. 13, 14, 18-20, 25, 28-31 Items were generated from both the conceptual frameworks and from the previously published instruments. Items were then revised through an iterative process until a consensus was achieved among the study authors. The conceptual framework included three domains of engagement: emotional engagement, behavioral engagement, and cognitive engagement. Cognitive engagement was divided into cognitive in-class and cognitive out-of-class, as supported by previous literature. 13 An eight-item instrument was created and blueprinted to the three domains as shown in figure 1. The eight items were: 1) I enjoyed this presentation 14, 18, 2) I was interested in this presentation13, 14, 3) I participated in this presentation19, 29, 4) I avoided distractions13, 20 5) I was an active learner14, 29, 6) I was absorbed in this presentation13, 27, 7) I will apply this presentation to my practice19, 28, 8) I am motivated to learn more about this topic. 14, 19, 28
CME Teaching Effectiveness (CMETE) Instrument
All presentations were evaluated using a previously validated CME teaching effectiveness (CMETE) instrument. 32 The CMETE instrument is a unidimensional survey containing eight items on a five point ordinal scale (1 = strongly disagree, 2= disagree, 3 = neutral, 4 = agree, 5 = strongly agree). The instrument scores previously demonstrated content, internal structure and relations to other variable validity evidence. 32 Learners completed both the CMETE and LEI after each lecture presentation. To prevent survey fatigue, two authors (C.P.W. and T.J.B.) from the original CMETE validation study reviewed the CMETE instrument to decrease the number of items. Through an iterative process, the CMETE was decreased from eight items to four items. The items with the highest interclass-correlation from the initial CMETE validation study were retained. 32 The final four items were: 1) Speaker presented information in a clear and organized manner, 2) Examples or cases were given that facilitated my understanding, 3) The slides added to the effectiveness of the presentations, and 4) Speaker included opportunities to learn interactively.
Data Collection
At the CME course, all attendees were invited to participate in the study. Attendees were reminded to download the conference app that collected: 1) demographic questions, 2) the CME Teaching Effectiveness (CMETE) Instrument for each presentation, and 3) The Learner Engagement Instrument (LEI) for each presentation. Previous research supports the response process validity evidence for the app used in this study. 32 All information collected remained anonymous. Participant demographic characteristics include age (years), gender (male, female), practice type (academic, group, solo, government or military, retired), specialty (family medicine, internal medicine, etc.), time spent providing patient care per week (hours), and years in clinical practice (years).
Burnout was assessed using a previously validated 2-item instrument. 33 Burnout was selected as a measure since previous research has demonstrated an inverse relationship to engagement 27. Participants recorded their CMETE instrument scores and LEI responses within the app, which anonymously linked to their demographic data.
Presenter variables included: age (years), academic rank, medical student teaching within the last year (yes or no), resident teaching within the last year (yes or no), fellowship teaching within the last year (yes or no), CME teaching within the last year (yes, no), and provision of direct patient care within the last year (yes or no).
Presentation characteristics included number of slides, summary slides, time of day (morning or afternoon), day of conference, and use of a pearls format. As outlined in prior research, slides which included titles of “pearls/clinical pearls” were counted as using the pearls format 34. A presentation was considered to have a summary slide if a slide listed “Key points”, “Outline”, “Take home message”, “Learning points”, “Some sort of summarized descriptions of the objectives”, or “Conclusions.”
Statistical Analysis
The data were entered and processed by the Mayo Clinic Survey Research Center. Categorical variables were presented percentages and numbers. To build validity to internal structure, an exploratory factor analysis was performed. The minimal proportion criteria were used for factor extracting. Items with factor loadings ≥ 0.50 were retained. Internal consistency reliability for items comprising each factor was calculated with a Cronbach alpha of > 0.7 as acceptable. 35
To account for the clustering of multiple ratings by students completing more than 1 evaluation, we generated an adjusted correlation matrix using mixed model approach. This adjusted correlation matrix was then used to perform confirmatory factor analysis with orthogonal rotation. In addition, for a sensitivity analysis, we also performed factor analysis using an unadjusted correlation matrix (Table 2).
For relations to other variables validity, associations between LEI scores and participant, presenter and presentation characteristics were determined using the Wilcoxon statistic. The level of statistical significance was set at an alpha of 0.05. Statistical analyses were conducted using SAS version 9.4 (SAS Institute Inc., Cary, NC).