This preprint is under consideration at BMC Medical Education. Preprints are preliminary reports that have not undergone peer review. They should not be considered conclusive, used to inform clinical practice, or referenced by the media as validated information.
Learn more about In Review
Research article
The Relationship between Learner Engagement and Teaching Effectiveness: A Novel Assessment of Student Engagement in Continuing Medical Education
Christopher R. Stephenson
Mayo Clinic
Corresponding Author
ORCiD: https://orcid.org/0000-0001-8537-392X
License:
This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License
Background: Continuing medical education (CME) often uses passive educational models including lectures. However, numerous studies have questioned the effectiveness of these less engaging educational strategies. Studies outside of CME suggest that engaged learning is associated with improved educational outcomes. However, measuring participants’ engagement can be challenging. We developed and determined the validity evidence for a novel instrument to assess learner engagement in CME.
Methods: We conducted a cross-sectional validation study at a large, didactic-style CME conference. Content validity evidence was established through review of literature and previously published engagement scales and conceptual frameworks on engagement, along with an iterative process involving experts in the field, to develop an eight-item Learner Engagement Instrument (LEI). Response process validity was established by vetting LEI items on item clarity and perceived meaning prior to implementation, as well as using a well-developed online platform with clear instructions.. Internal structure validity evidence was based on factor analysis and calculating internal consistency reliability. Relations to other variables validity evidence was determined by examining associations between LEI and previously validated CME Teaching Effectiveness (CMETE) instrument scores. Following each presentation, all participants were invited to complete the LEI and the CMETE.
Results: 51 out of 206 participants completed the LEI and CMETE (response rate 25%) Correlations between the LEI and the CMETE overall scores were strong (r=0.80). Internal consistency reliability for the LEI was excellent (Cronbach’s alpha=0.96). To support validity to internal structure, a factor analysis was performed and revealed a two dimensional instrument consisting of internal and external engagement domains. The internal consistency reliabilities were 0.96 for the internal engagement domain and 0.95 for the external engagement domain.
Conclusion: Engagement, as measured by the LEI, is strongly related to teaching effectiveness. The LEI is supported by robust validity evidence including content, response process, internal structure, and relations to other variables. Given the relationship between learner engagement and teaching effectiveness, identifying more engaging and interactive methods for teaching in CME is recommended.
Keywords
CME; Engagement, Validity, Teaching Effectiveness
Figure 1
Figure 2
Continuing Medical Education (CME) courses are frequently used by physicians to maintain up-to-date professional competence and performance. 1 However, numerous studies have questioned the effectiveness of CME teaching methods. 2-5 Traditional CME courses use mostly passive educational models. 6 Although there is evidence that interactive teaching methods may be more effective than lectures, lectures remain the primary teaching modality in CME 1, 7, 8. Ideally, education should be an active process where students are engaged to learn. 9, 10
Student engagement is a complex phenomenon that involves both physical and psychological constructs. It is defined as the amount of energy students physically and psychologically expend whereby they stay attentive, involved, and motivated to learn. 11, 12 Studies outside of CME demonstrate that more engaged students have higher course satisfaction and achievement of course learning objectives. 13, 14 Studies of college-level courses have reported that more engaged learners are also high achievers. 15, 16 In medical education, learner engagement in online CME courses is associated with improved patient outcomes including improved diabetic metrics. 17 Additionally, interactive teaching methods in CME are associated higher self-reported learning scores. 7 Despite studies suggesting that more engaged learners are more successful, research on learning engagement for in-person medical education, specifically continuing medical education, is limited.
Previously published conceptual frameworks divide engagement into multiple domains including behavioral engagement, psychological or emotional engagement and cognitive engagement. 11, 12, 14, 18-24 Additionally, these domains can be separated into in-class and out-of-class components. 13, 14
Behavioral engagement involves the physical actions of the learner. 14, 18, 20 This includes attendance, participation in class activities, and attentiveness. 18, 20 For example, a student who demonstrates high behavioral engagement would attend didactic sessions or workshops, avoid distractions, and actively participate by taking notes, using audience response methods, or participating in group discussions19, 21, 23, 24.
Psychological engagement refers to the positive and negative emotions a learner experiences in response to the teacher, peers, and the education environment. 13, 14, 18, 20 A student with high emotional engagement would enjoy the education session and not focus on the passage of time. Low emotional engagement would include experiences and expressions of boredom, anger, anxiety or sadness.1920, 24
Cognitive engagement involves an ability to engage in self-regulated learning and an appreciation for the value of learning. In cognitive engagement, students are motivated to learn, both in and out of the classroom. 13, 14, 20 Cognitive engagement also involves the perceived relevance of material to the learner’s experience. A student with high cognitive engagement would appreciate the applicability of material to future practice and would be motivated to learn more about the material, even outside of class.19, 24
There are several engagement assessment instruments which have been published for secondary school and college-level students. However, these instruments have items and domains that are specific to these populations and that may not be relevant to practicing physicians attending CME.13, 14, 20, 25 Furthermore, these instruments are cumbersome, having numerous items which would be impractical for assessing learner engagement following multiple lectures or workshops in CME populations. Since engaged learners are linked to higher academic achievement, measuring learner engagement in CME should be pursued.7, 13-17 However, instruments to measure learner engagement in medical education, specifically CME, are lacking. Using Messick’s framework for validity of psychometric assessment 26, we aimed to develop a novel instrument to assess learner engagement in CME, supporting 1) content validity using previously published conceptual frameworks and prior instruments 13, 14, 18 and item development through an iterative process with experts in the field, 2) internal structure validity by performing factor analysis and calculations of internal consistency, and 3) relations to other variables by exploring associations with engagement scores, teaching methods, and burnout. Based on prior research, we hypothesized that learner engagement would consist of behavioral, psychological, and cognitive domains, be directly related to teaching effectiveness, and be inversely related to learner burnout. 27
Study Design and Participants
We conducted a cross-sectional study of attendees at the 27th Mayo Clinic Internal Medicine Board Review. This week-long course is a high-yield intensive program designed to assist learners with the American Board of Internal Medicine (ABIM) Initial and Recertification Examinations and to provide a relevant review for daily practice. Historically, most attendees are physicians who will be taking one of these examinations. The Mayo Clinic Internal Medicine Board Review offers 52.5 category 1 CME credits and consists of 57, 30-60 minute podium presentations. This setting was selected due to its generalizability in CME.
Mayo Clinic faculty members are selected to present based on their topic-related expertise. Course content is determined by a planning committee of generalists, specialists, and course directors. Learning objectives are given to each presenter by the course directors. This study was deemed exempt by the Mayo Clinic Institutional Review Board.
Learner Engagement Instrument (LEI) Development
A Learner Engagement Instrument (LEI) was developed to evaluate learners’ engagement in CME. To provide content validity, study authors and experts in the field [C.R.S., S.L.B., A.P.S., L.W.R., T.J.B., C.M.W.] reviewed conceptual frameworks on engagement and previously published engagement instruments. 13, 14, 18-20, 25, 28-31 Items were generated from both the conceptual frameworks and from the previously published instruments. Items were then revised through an iterative process until a consensus was achieved among the study authors. The conceptual framework included three domains of engagement: emotional engagement, behavioral engagement, and cognitive engagement. Cognitive engagement was divided into cognitive in-class and cognitive out-of-class, as supported by previous literature. 13 An eight-item instrument was created and blueprinted to the three domains as shown in figure 1. The eight items were: 1) I enjoyed this presentation 14, 18, 2) I was interested in this presentation13, 14, 3) I participated in this presentation19, 29, 4) I avoided distractions13, 20 5) I was an active learner14, 29, 6) I was absorbed in this presentation13, 27, 7) I will apply this presentation to my practice19, 28, 8) I am motivated to learn more about this topic. 14, 19, 28
CME Teaching Effectiveness (CMETE) Instrument
All presentations were evaluated using a previously validated CME teaching effectiveness (CMETE) instrument. 32 The CMETE instrument is a unidimensional survey containing eight items on a five point ordinal scale (1 = strongly disagree, 2= disagree, 3 = neutral, 4 = agree, 5 = strongly agree). The instrument scores previously demonstrated content, internal structure and relations to other variable validity evidence. 32 Learners completed both the CMETE and LEI after each lecture presentation. To prevent survey fatigue, two authors (C.P.W. and T.J.B.) from the original CMETE validation study reviewed the CMETE instrument to decrease the number of items. Through an iterative process, the CMETE was decreased from eight items to four items. The items with the highest interclass-correlation from the initial CMETE validation study were retained. 32 The final four items were: 1) Speaker presented information in a clear and organized manner, 2) Examples or cases were given that facilitated my understanding, 3) The slides added to the effectiveness of the presentations, and 4) Speaker included opportunities to learn interactively.
Data Collection
At the CME course, all attendees were invited to participate in the study. Attendees were reminded to download the conference app that collected: 1) demographic questions, 2) the CME Teaching Effectiveness (CMETE) Instrument for each presentation, and 3) The Learner Engagement Instrument (LEI) for each presentation. Previous research supports the response process validity evidence for the app used in this study. 32 All information collected remained anonymous. Participant demographic characteristics include age (years), gender (male, female), practice type (academic, group, solo, government or military, retired), specialty (family medicine, internal medicine, etc.), time spent providing patient care per week (hours), and years in clinical practice (years).
Burnout was assessed using a previously validated 2-item instrument. 33 Burnout was selected as a measure since previous research has demonstrated an inverse relationship to engagement 27. Participants recorded their CMETE instrument scores and LEI responses within the app, which anonymously linked to their demographic data.
Presenter variables included: age (years), academic rank, medical student teaching within the last year (yes or no), resident teaching within the last year (yes or no), fellowship teaching within the last year (yes or no), CME teaching within the last year (yes, no), and provision of direct patient care within the last year (yes or no).
Presentation characteristics included number of slides, summary slides, time of day (morning or afternoon), day of conference, and use of a pearls format. As outlined in prior research, slides which included titles of “pearls/clinical pearls” were counted as using the pearls format 34. A presentation was considered to have a summary slide if a slide listed “Key points”, “Outline”, “Take home message”, “Learning points”, “Some sort of summarized descriptions of the objectives”, or “Conclusions.”
Statistical Analysis
The data were entered and processed by the Mayo Clinic Survey Research Center. Categorical variables were presented percentages and numbers. To build validity to internal structure, an exploratory factor analysis was performed. The minimal proportion criteria were used for factor extracting. Items with factor loadings ≥ 0.50 were retained. Internal consistency reliability for items comprising each factor was calculated with a Cronbach alpha of > 0.7 as acceptable. 35
To account for the clustering of multiple ratings by students completing more than 1 evaluation, we generated an adjusted correlation matrix using mixed model approach. This adjusted correlation matrix was then used to perform confirmatory factor analysis with orthogonal rotation. In addition, for a sensitivity analysis, we also performed factor analysis using an unadjusted correlation matrix (Table 2).
For relations to other variables validity, associations between LEI scores and participant, presenter and presentation characteristics were determined using the Wilcoxon statistic. The level of statistical significance was set at an alpha of 0.05. Statistical analyses were conducted using SAS version 9.4 (SAS Institute Inc., Cary, NC).
Demographics are reported in table 1. A total of 51 out of 206 participants completed the LEI, CMETE and demographic survey (response rate 25%). However, since participants completed surveys on up to 57 lectures, 2,486 LEI and CMETE surveys were submitted out of a total of 2,907 possible surveys for the 51 participants (86% of possible presentations). Of the 51 participants, 30 (59%) were male and 40 (78%) were internal medicine physicians. Over half of respondents (57%) were over the age of 50. The majority of participants were from the Midwestern United States (n=35, 69%).
Table 1
|
Demographics of Participants |
|
|
|
Total |
|
Practice Setting |
|
|
Academic |
10 (19.6%) |
|
Govt/Military |
3 (5.9%) |
|
Group |
30 (58.8%) |
|
Solo |
3 (5.9%) |
|
other |
5 (9.8%) |
|
|
|
|
Specialty |
|
|
IM |
40 (78.4%) |
|
FM |
2 (3.9%) |
|
Medical |
8 (15.7%) |
|
Non-Medical |
1 (2.0%) |
|
|
|
|
Practice Location |
|
|
Northeast |
2 (3.9%) |
|
Southeast |
9 (17.6%) |
|
Midwest |
35 (68.6%) |
|
Southwest |
3 (5.9%) |
|
West |
2 (3.9%) |
|
|
|
|
Reason for Enrolling |
|
|
New board certification |
15 (29.4%) |
|
Renewal of certification |
19 (37.3%) |
|
Not passing prior certification |
1 (2.0%) |
|
General knowledge |
16 (31.4%) |
To support validity to internal structure, an exploratory factor analysis of the LEI was performed as shown in table 2. A sensitivity analysis accounting for the clustering of multiple ratings by students completing multiple evaluations showed no differences between the factor analyses using the adjusted or unadjusted correlation matrixes. Items with a factor loading of 0.5 were retained with no items needing to be removed. Two factors were identified. One factor, which consisted of 4 items, included the domains of emotional engagement and cognitive out-of-class. We termed this new combined domain, “internal engagement.” Internal engagement reflected items which were internal to the learner, such as their enjoyment of the topic and motivation for self-directed learning and applicability of the material. The other factor comprised of the remaining 4 items and including the domains of behavioral engagement and cognitive in-class. We termed this new combined domain, “external engagement.” External engagement consisted of items external to the learner, which could be observed, such as participation during the course and being an active, absorbed learner. The internal consistency reliabilities (Cronbach alpha) were 0.96 for the internal engagement domain and 0.95 for the external engagement domain. Internal consistency was 0.96 for all 8 items overall. The abbreviated 4-item CMETE instrument’s internal consistent was 0.97 which is similar to previously published studies on this instrument’s performance 32.
Table 2
|
Exploratory Factor Analysis of Learner Engagement Instrument (Unadjusted) |
||
|
Questions |
Factor 1 |
Factor 2 |
|
Enjoyed presentation |
0.7013 |
0.4627 |
|
Interested presentation |
0.7471 |
0.4983 |
|
Participated presentation |
0.3751 |
0.6803 |
|
Avoided distractions |
0.4372 |
0.7046 |
|
Active listener |
0.4940 |
0.7487 |
|
Absorbed presentation |
0.4710 |
0.7223 |
|
Apply presentation practice |
0.7357 |
0.4172 |
|
Motivated learn more |
0.7499 |
0.4420 |
|
Exploratory Factor Analysis of Learner Engagement Instrument (Adjusted for Correlation) |
|||
|
Questions |
Factor 1 |
Factor 2 |
|
|
Enjoyed presentation |
0.7120 |
0.4467 |
|
|
Interested presentation |
0.7598 |
0.4761 |
|
|
Participated presentation |
0.3619 |
0.6996 |
|
|
Avoided distractions |
0.4364 |
0.7106 |
|
|
Active listener |
0.5102 |
0.7334 |
|
|
Absorbed presentation |
0.4533 |
0.7372 |
|
|
Apply presentation practice |
0.7377 |
0.4115 |
|
|
Motivated learn more |
0.7546 |
0.4337 |
|
To support relations to other variables validity, the LEI was compared to the CMETE, presentation characteristics, and burnout. Mean scores for the LEI and CMETE are reported in table 3. The LEI was highly correlated to the CMETE (R=0.80). Internal engagement correlated with teaching effectiveness (R= 0.80) stronger than external engagement (R=0.73) There were no differences in the LEI or CMETE scores based on demographics including age, sex, practice location, specialty or reason for enrolling in the course.
Table 3
|
Results of Learner Engagement Instrument and CME Teaching Effectiveness Instrument |
|
|
|
Total |
|
Learner Engagement instrument (LEI) |
|
|
Mean (SD) |
35.6 (7.3) |
|
Median |
38.1 |
|
Q1, Q3 |
34.0, 39.8 |
|
Range Internal consistency |
(0.0-40.0) 0.96 |
|
Factor 1 of LEI (Internal Engagement) |
|
|
Mean (SD) |
17.9 (3.7) |
|
Median |
19.4 |
|
Q1, Q3 |
17.4, 19.8 |
|
Range Internal consistency |
(0.0-20.0) 0.96 |
|
Factor 2 of LEI (External Engagement) |
|
|
Mean (SD) |
17.7 (3.7) |
|
Median |
19.0 |
|
Q1, Q3 |
16.4, 19.9 |
|
Range Internal consistency |
(0.0-20.0) 0.95 |
|
CME Teaching Effectiveness Instrument |
|
|
Mean (SD) |
18.1 (3.8) |
|
Median |
19.4 |
|
Q1, Q3 |
17.6, 19.9 |
|
Range Internal consistency |
(0.0-20.0) 0.97 |
Presentations on the first day of conference had lower mean LEI scores compared to later days (-0.21 on a 5 point scale, p<0.001). Presentations on the first day of conference also had lower CMETE scores (-0.19 on a 5 point scale, p<0.001). Otherwise, presentation characteristics, including use of humor, number of slides, summary slides or use of pearl format and time of day did not influence CMETE or LEI scores. Learners with burnout tended to have higher learning engagement scores, but the results were not significant (-0.79 on a 5 point scale, p=0.41).
The Learner Engagement Instrument (LEI) is an innovative method for measuring engagement in the CME population. The LEI demonstrated strong internal structure validity evidence including a two-factor model that measures engagement in both psychological and behavioral domains. We also identified relations to other variables evidence in terms of a strong relationship between learner engagement and teaching effectiveness, which suggests that effective teachers are more likely to engage their learners.
The LEI is supported by validity evidence according to Messick’s framework for assessment. 26 Regarding content validity evidence, the LEI underwent a rigorous design process that incorporated previously published literature and conceptual frameworks on engagement, along with input from experts and an iterative process of item development. Additionally, to preserve the integrity of the data and support validity to response process, we used a well-developed online platform to collect information with clear instructions was utilized. Additionally, prior to implementation, all LEI items were pilot tested on physicians outside of the study with feedback given on clarity and the perceived meaning of the items. All items were entered and analyzed using a dedicated survey research center. Finally, internal structure validity evidence was supported by factor analysis demonstrating a multidimensional assessment of learner engagement in CME, as well as excellent internal consistency reliability.
Furthermore, relation to other variables validity was supported by substantial correlations between overall LEI and CMETE scores. In summary, effective teachers had more engaged learners. Since the CMETE included items such as speaker clarity, use of case-based presentations, organized slides, and opportunities to learner interactively, faculty should be encouraged to employ these strategies routinely in their teaching sessions. Presentations on the first day of the course had slightly lower LEI scores than presentations on later days. However, this may be related to teaching effectiveness since these presentations also had lower CMETE scores.
We observed a trend towards increased burnout and more engagement rather than less engagement as was previously hypothesized. However, it’s possible that participants with higher burnout tend to exhibit higher focus, or perhaps participants with higher burnout have reasons to be more concerned about their future performance on certification examinations. Furthermore, the high-stakes nature of a board review course may contribute to increased burnout. More study is needed to identify if engagement and burnout are related in a CME population. Studying engagement and burnout in a lower stakes CME environment should also be considered.
Though we expected three domains, our exploratory factor analysis of the LEI revealed a two-dimensional instrument. A two-domain conceptual framework for engagement is also supported in the literature12. The revised framework for engagement based on our factor analysis is shown in figure 2. The items of emotional engagement and cognitive out-of-class engagement loaded together in the factor analysis. These domains focused on items internal to the learner, which cannot be seen by an outside observer. This includes a learner’s enjoyment and interest in the topic as well as the learner’s motivation to learn and apply the topic in their practice. 19, 23, 24Given that these items were internal preferences or motivations, we termed this new domain, “Internal Engagement.”
Additionally, behavioral engagement and cognitive in-class engagement loaded together in the factor analysis. Both these domains focused on the external actions of the learner while in class. This includes participating, avoiding distractions, listening and being absorbed in the presentation.19, 23, 24 Given that these items were in-class actions which could be observed by an external observer, we termed this engagement, “External Engagement.”
The domains of internal and external engagement parallel concepts in self-determination theory. Self-determination theory describes how humans are motivated to learn and includes concepts of amotivation, extrinsic motivation and intrinsic motivation. 36 Intrinsic motivation is internal and includes the learner’s personal interest and enjoyment of the material. 37 This is similar to our identified domain of internal engagement. A learner who has high internal engagement may find a topic enjoyable and align with their personal interests. Conversely, extrinsic motivation describes external pressures that influence a learner’s motivation to learn. This could include career goals, rewards, or societal demands. 37 A learner who has a high external engagement may be cognizant of the social pressures to avoid distractions and be an active participant, consequently also having high external engagement in the presentation. More research is needed to further explore the overlap between these theories on engagement and motivation to learn.
There are several reasons to explain why a third domain wasn’t identified in our factor analysis. First, previously published engagement inventories at the high school and college-level include more items for each domain. 13, 14, 25 Given that the LEI had only eight items, there may be too few items to discriminate between various domains of engagement. Second, as outlined in previous conceptual frameworks, there may only be two domains of engagement, a psychological-based domain and behavioral-based domain, with cognitive engagement existing in an overlap between the two. Third, CME attendees may be inherently different from college-level students and the domain of cognitive engagement may not be relevant to this population. Lastly, as noted above, our framework for engagement may overlap with the concepts of motivations to learn where internal and external engagement relate to intrinsic and extrinsic motivation to learn.
Our study has several strengths. As previously noted, the LEI is supported by strong validity evidence using Messick’s framework to design the instrument and underwent a vigorous design process. 26 Our sample was collected in a large, didactic-style CME course, which is representative of most types of CME. Last, although our response rate was modest, over 2500 LEI and CMETE surveys were answered. There are also several limitations. As noted above, our response rate was low to modest, which likely introduces selection bias. It is possible that only the most engaged learners completed the survey, biasing our results. This could artificially inflate learner engagement scores and skew our results. Furthermore, we were unable to collect comparative information on nonresponders. Studying the engagement of these initial nonresponders could be a potential area for future research. Additionally, engagement isn’t dichotomous and we didn’t identify a score cutoff differentiating between highly engaging presentations and non-engaging ones.
This smaller sample size may have also contributed to a type II error for burnout if our sample size was not adequate to detect a relationship between engagement and burnout. Future research on the relationship between burnout and engagement should be sought with larger sample sizes. Furthermore, additional study on engagement and relationship to teaching effectiveness in other populations, such as undergraduate and graduate medical education should be pursued.
Engagement is an important construct for learner retention, but measuring engagement can be challenging. Overall, the LEI demonstrated strong validity evidence and showed that more effective teachers, as assessed by students' evaluations, were associated with higher student engagement. Given the relationship between learner engagement and teaching effectiveness, identifying more engaging and interactive methods for teaching in CME is recommended.
Continuing Medical Education (CME)
American Board of Internal Medicine (ABIM)
Learner Engagement Instrument (LEI)
Continuing Medical Education Teaching Effectiveness (CMETE)
Ethics Approval and Consent to Participate: Mayo Clinic’s Institutional Review Board reviewed the study protocol and deemed the study exempt. A written letter was given to all participants stating that by completing the LEI questionnaires, participants were implying consent to participate in this study.
Consent for publication: Not applicable
Availability of data and materials: Data for this study is available at the following hyperlink: https://www.dropbox.com/s/wrkjc29g1gv41cd/2018%20Engagement%20Dataset.xlsx?dl=0
Competing interests: The authors declare that they have no competing interests.
Funding: None.
Authors contributions: Authors CRS, SB, AS, LR, TB and CW designed the LEI after reviewing the literature and participated in the iterative design process. Authors CRS and SB collected the data at the CME course. Authors CDS and JM performed statistical analysis. All authors contributed to the manuscript including creation and revision. All authors read and approved the final manuscript.
Acknowledgements: Not applicable
- Cervero RM, Gaines JK. The impact of CME on physician performance and patient health outcomes: an updated synthesis of systematic reviews. Journal of Continuing Education in the Health Professions. 2015;35:131-138.
- Warden GL, Mazmanian PE, Leach DC. Redesigning continuing education in the health professions: Washington, DC: Institute of Medicine, National Academy of Sciences; 2010.
- Nissen SE. Reforming the continuing medical education system. Jama. 2015;313:1813-1814.
- Campbell EG, Rosenthal M. Reform of continuing medical education: investments in physician human capital. JAMA. 2009;302:1807-1808.
- Davis DA, Prescott J, Fordis Jr CM, et al. Rethinking CME: an imperative for academic medicine and faculty development. Academic Medicine. 2011;86:468-473.
- Mazmanian PE, Davis DA. Continuing medical education and the physician as a learner: guide to the evidence. Jama. 2002;288:1057-1060.
- Stephenson CR, Wang AT, Szostek JH, et al. Flipping the continuing medical education classroom: Validating a measure of attendees' perceptions. Journal of Continuing Education in the Health Professions. 2016;36:256-262.
- Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. Int J Technol Assess Health Care. 2005;21:380-385.
- Michael J. Where's the evidence that active learning works? Advances in physiology education. 2006;30:159-167.
- Prober CG, Heath C. Lecture halls without lectures—a proposal for medical education. N Engl J Med. 2012;366:1657-1659.
- Bernard JS. Student Engagement: A Principle-Based Concept Analysis. International journal of nursing education scholarship. 2015;12.
- Astin AW. Student involvement: A developmental theory for higher education. Journal of college student personnel. 1984;25:297-308.
- Burch GF, Heller NA, Burch JJ, Freed R, Steed SA. Student engagement: Developing a conceptual framework and survey instrument. Journal of Education for Business. 2015;90:224-229.
- Gunuc S, Kuzu A. Student engagement scale: development, reliability and validity. Assessment & Evaluation in Higher Education. 2015;40:587-610.
- Williams RL, Eggert AC. Notetaking in college classes: Student patterns and instructional strategies. The Journal of General Education. 2002:173-199.
- Kiewra KA. A review of note-taking: The encoding-storage paradigm and beyond. Educational Psychology Review. 1989;1:147-172.
- Crenshaw K, Curry W, Salanitro AH, et al. Is physician engagement with Web-based CME associated with patients’ baseline hemoglobin A1c levels? The Rural Diabetes Online Care study. Academic medicine: journal of the Association of American Medical Colleges. 2010;85:1511.
- Fredricks JA, Blumenfeld PC, Paris AH. School engagement: Potential of the concept, state of the evidence. Review of educational research. 2004;74:59-109.
- Appleton JJ, Christenson SL, Kim D, Reschly AL. Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology. 2006;44:427-445.
- Alrashidi O, Phan H, Ngu B. Academic Engagement: An Overview of Its Definitions, Dimensions, and Major Conceptualisations2016.
- Moreira P, Cunha D, Inman RA. An integration of multiple student engagement dimensions into a single measure and validity-based studies. Journal of Psychoeducational Assessment. 2019:0734282919870973.
- Whitney BM, Cheng Y, Brodersen AS, Hong MR. The scale of student engagement in statistics: Development and initial validation. Journal of Psychoeducational Assessment. 2019;37:553-565.
- Pickering JD, Swinnerton BJ. Exploring the dimensions of medical student engagement with technology‐enhanced learning resources and assessing the impact on assessment outcomes. Anatomical Sciences Education. 2019;12:117-128.
- Trowler V. Student engagement literature review. The higher education academy. 2010;11:1-15.
- Kuh GD. The national survey of student engagement: Conceptual and empirical foundations. New directions for institutional research. 2009;2009:5-20.
- Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American psychologist. 1995;50:741.
- Schaufeli WB, Salanova M, González-Romá V, Bakker AB. The measurement of engagement and burnout: A two sample confirmatory factor analytic approach. Journal of Happiness studies. 2002;3:71-92.
- Reeve J, Tseng C-M. Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology. 2011;36:257-267.
- O’Malley KJ, Moran BJ, Haidet P, et al. Validation of an observation instrument for measuring student engagement in health professions settings. Evaluation & the health professions. 2003;26:86-103.
- Ema P, Rovan D, Ljubin Golub T. Development and validation of the Academic Engagement Scale (AES)2016.
- Kuh GD. Assessing what really matters to student learning inside the national survey of student engagement. Change: The Magazine of Higher Learning. 2001;33:10-17.
- Wittich CM, Mauck KF, Mandrekar JN, et al. Improving participant feedback to continuing medical education presenters in internal medicine: a mixed-methods study. Journal of general internal medicine. 2012;27:425-431.
- West CP, Dyrbye LN, Sloan JA, Shanafelt TD. Single item measures of emotional exhaustion and depersonalization are useful for assessing burnout in medical professionals. Journal of general internal medicine. 2009;24:1318.
- Kopecky SL, Litin SC. Clinical pearls in cardiology. Mayo Clinic Proceedings. Vol 85: Mayo Foundation; 2010:473.
- DeVellis RF. Guidelines in scale development. Scale Development: Theory and Applications. Newbury Park, Calif: Sage. 1991;5191.
- Deci EL, Koestner R, Ryan RM. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological bulletin. 1999;125:627.
- Cook DA, Artino Jr AR. Motivation to learn: an overview of contemporary theories. Medical education. 2016;50:997-1014.
- Alrashidi O, Phan HP, Ngu BH. Academic engagement: an overview of its definitions, dimensions, and major conceptualisations. International Education Studies. 2016;9:41.
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Under Review
Version 2
Posted 31 Aug, 2020
No community comments so far
Review #1 received
Received 17 Sep, 2020
Reviewer #1 agreed
On 15 Sep, 2020
Reviewers invited
Invitations sent on 14 Sep, 2020
Editor assigned
On 27 Aug, 2020
Submission checks complete
On 27 Aug, 2020
Editor invited
On 27 Aug, 2020
No comments provided
Editorial decision: Major revision
On 25 Jul, 2020
Review #1 received
Received 20 Jul, 2020
Review #2 received
Received 15 Jul, 2020
Reviewer #2 agreed
On 01 Jul, 2020
Reviewers invited
Invitations sent on 29 Jun, 2020
Reviewer #1 agreed
On 29 Jun, 2020
Editor assigned
On 20 Jun, 2020
Submission checks complete
On 20 Jun, 2020
Editor invited
On 20 Jun, 2020
First submitted
On 18 Jun, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Learn more about our company.
This preprint is under consideration at BMC Medical Education. Preprints are preliminary reports that have not undergone peer review. They should not be considered conclusive, used to inform clinical practice, or referenced by the media as validated information.
Learn more about In Review
Research article
The Relationship between Learner Engagement and Teaching Effectiveness: A Novel Assessment of Student Engagement in Continuing Medical Education
Christopher R. Stephenson
Mayo Clinic
Corresponding Author
ORCiD: https://orcid.org/0000-0001-8537-392X
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Under Review
Version 2
Posted 31 Aug, 2020
No community comments so far
Review #1 received
Received 17 Sep, 2020
Reviewer #1 agreed
On 15 Sep, 2020
Reviewers invited
Invitations sent on 14 Sep, 2020
Editor assigned
On 27 Aug, 2020
Submission checks complete
On 27 Aug, 2020
Editor invited
On 27 Aug, 2020
No comments provided
Editorial decision: Major revision
On 25 Jul, 2020
Review #1 received
Received 20 Jul, 2020
Review #2 received
Received 15 Jul, 2020
Reviewer #2 agreed
On 01 Jul, 2020
Reviewers invited
Invitations sent on 29 Jun, 2020
Reviewer #1 agreed
On 29 Jun, 2020
Editor assigned
On 20 Jun, 2020
Submission checks complete
On 20 Jun, 2020
Editor invited
On 20 Jun, 2020
First submitted
On 18 Jun, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
License:
This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License
Background: Continuing medical education (CME) often uses passive educational models including lectures. However, numerous studies have questioned the effectiveness of these less engaging educational strategies. Studies outside of CME suggest that engaged learning is associated with improved educational outcomes. However, measuring participants’ engagement can be challenging. We developed and determined the validity evidence for a novel instrument to assess learner engagement in CME.
Methods: We conducted a cross-sectional validation study at a large, didactic-style CME conference. Content validity evidence was established through review of literature and previously published engagement scales and conceptual frameworks on engagement, along with an iterative process involving experts in the field, to develop an eight-item Learner Engagement Instrument (LEI). Response process validity was established by vetting LEI items on item clarity and perceived meaning prior to implementation, as well as using a well-developed online platform with clear instructions.. Internal structure validity evidence was based on factor analysis and calculating internal consistency reliability. Relations to other variables validity evidence was determined by examining associations between LEI and previously validated CME Teaching Effectiveness (CMETE) instrument scores. Following each presentation, all participants were invited to complete the LEI and the CMETE.
Results: 51 out of 206 participants completed the LEI and CMETE (response rate 25%) Correlations between the LEI and the CMETE overall scores were strong (r=0.80). Internal consistency reliability for the LEI was excellent (Cronbach’s alpha=0.96). To support validity to internal structure, a factor analysis was performed and revealed a two dimensional instrument consisting of internal and external engagement domains. The internal consistency reliabilities were 0.96 for the internal engagement domain and 0.95 for the external engagement domain.
Conclusion: Engagement, as measured by the LEI, is strongly related to teaching effectiveness. The LEI is supported by robust validity evidence including content, response process, internal structure, and relations to other variables. Given the relationship between learner engagement and teaching effectiveness, identifying more engaging and interactive methods for teaching in CME is recommended.
Figure 1
Figure 2
Continuing Medical Education (CME) courses are frequently used by physicians to maintain up-to-date professional competence and performance. 1 However, numerous studies have questioned the effectiveness of CME teaching methods. 2-5 Traditional CME courses use mostly passive educational models. 6 Although there is evidence that interactive teaching methods may be more effective than lectures, lectures remain the primary teaching modality in CME 1, 7, 8. Ideally, education should be an active process where students are engaged to learn. 9, 10
Student engagement is a complex phenomenon that involves both physical and psychological constructs. It is defined as the amount of energy students physically and psychologically expend whereby they stay attentive, involved, and motivated to learn. 11, 12 Studies outside of CME demonstrate that more engaged students have higher course satisfaction and achievement of course learning objectives. 13, 14 Studies of college-level courses have reported that more engaged learners are also high achievers. 15, 16 In medical education, learner engagement in online CME courses is associated with improved patient outcomes including improved diabetic metrics. 17 Additionally, interactive teaching methods in CME are associated higher self-reported learning scores. 7 Despite studies suggesting that more engaged learners are more successful, research on learning engagement for in-person medical education, specifically continuing medical education, is limited.
Previously published conceptual frameworks divide engagement into multiple domains including behavioral engagement, psychological or emotional engagement and cognitive engagement. 11, 12, 14, 18-24 Additionally, these domains can be separated into in-class and out-of-class components. 13, 14
Behavioral engagement involves the physical actions of the learner. 14, 18, 20 This includes attendance, participation in class activities, and attentiveness. 18, 20 For example, a student who demonstrates high behavioral engagement would attend didactic sessions or workshops, avoid distractions, and actively participate by taking notes, using audience response methods, or participating in group discussions19, 21, 23, 24.
Psychological engagement refers to the positive and negative emotions a learner experiences in response to the teacher, peers, and the education environment. 13, 14, 18, 20 A student with high emotional engagement would enjoy the education session and not focus on the passage of time. Low emotional engagement would include experiences and expressions of boredom, anger, anxiety or sadness.1920, 24
Cognitive engagement involves an ability to engage in self-regulated learning and an appreciation for the value of learning. In cognitive engagement, students are motivated to learn, both in and out of the classroom. 13, 14, 20 Cognitive engagement also involves the perceived relevance of material to the learner’s experience. A student with high cognitive engagement would appreciate the applicability of material to future practice and would be motivated to learn more about the material, even outside of class.19, 24
There are several engagement assessment instruments which have been published for secondary school and college-level students. However, these instruments have items and domains that are specific to these populations and that may not be relevant to practicing physicians attending CME.13, 14, 20, 25 Furthermore, these instruments are cumbersome, having numerous items which would be impractical for assessing learner engagement following multiple lectures or workshops in CME populations. Since engaged learners are linked to higher academic achievement, measuring learner engagement in CME should be pursued.7, 13-17 However, instruments to measure learner engagement in medical education, specifically CME, are lacking. Using Messick’s framework for validity of psychometric assessment 26, we aimed to develop a novel instrument to assess learner engagement in CME, supporting 1) content validity using previously published conceptual frameworks and prior instruments 13, 14, 18 and item development through an iterative process with experts in the field, 2) internal structure validity by performing factor analysis and calculations of internal consistency, and 3) relations to other variables by exploring associations with engagement scores, teaching methods, and burnout. Based on prior research, we hypothesized that learner engagement would consist of behavioral, psychological, and cognitive domains, be directly related to teaching effectiveness, and be inversely related to learner burnout. 27
Study Design and Participants
We conducted a cross-sectional study of attendees at the 27th Mayo Clinic Internal Medicine Board Review. This week-long course is a high-yield intensive program designed to assist learners with the American Board of Internal Medicine (ABIM) Initial and Recertification Examinations and to provide a relevant review for daily practice. Historically, most attendees are physicians who will be taking one of these examinations. The Mayo Clinic Internal Medicine Board Review offers 52.5 category 1 CME credits and consists of 57, 30-60 minute podium presentations. This setting was selected due to its generalizability in CME.
Mayo Clinic faculty members are selected to present based on their topic-related expertise. Course content is determined by a planning committee of generalists, specialists, and course directors. Learning objectives are given to each presenter by the course directors. This study was deemed exempt by the Mayo Clinic Institutional Review Board.
Learner Engagement Instrument (LEI) Development
A Learner Engagement Instrument (LEI) was developed to evaluate learners’ engagement in CME. To provide content validity, study authors and experts in the field [C.R.S., S.L.B., A.P.S., L.W.R., T.J.B., C.M.W.] reviewed conceptual frameworks on engagement and previously published engagement instruments. 13, 14, 18-20, 25, 28-31 Items were generated from both the conceptual frameworks and from the previously published instruments. Items were then revised through an iterative process until a consensus was achieved among the study authors. The conceptual framework included three domains of engagement: emotional engagement, behavioral engagement, and cognitive engagement. Cognitive engagement was divided into cognitive in-class and cognitive out-of-class, as supported by previous literature. 13 An eight-item instrument was created and blueprinted to the three domains as shown in figure 1. The eight items were: 1) I enjoyed this presentation 14, 18, 2) I was interested in this presentation13, 14, 3) I participated in this presentation19, 29, 4) I avoided distractions13, 20 5) I was an active learner14, 29, 6) I was absorbed in this presentation13, 27, 7) I will apply this presentation to my practice19, 28, 8) I am motivated to learn more about this topic. 14, 19, 28
CME Teaching Effectiveness (CMETE) Instrument
All presentations were evaluated using a previously validated CME teaching effectiveness (CMETE) instrument. 32 The CMETE instrument is a unidimensional survey containing eight items on a five point ordinal scale (1 = strongly disagree, 2= disagree, 3 = neutral, 4 = agree, 5 = strongly agree). The instrument scores previously demonstrated content, internal structure and relations to other variable validity evidence. 32 Learners completed both the CMETE and LEI after each lecture presentation. To prevent survey fatigue, two authors (C.P.W. and T.J.B.) from the original CMETE validation study reviewed the CMETE instrument to decrease the number of items. Through an iterative process, the CMETE was decreased from eight items to four items. The items with the highest interclass-correlation from the initial CMETE validation study were retained. 32 The final four items were: 1) Speaker presented information in a clear and organized manner, 2) Examples or cases were given that facilitated my understanding, 3) The slides added to the effectiveness of the presentations, and 4) Speaker included opportunities to learn interactively.
Data Collection
At the CME course, all attendees were invited to participate in the study. Attendees were reminded to download the conference app that collected: 1) demographic questions, 2) the CME Teaching Effectiveness (CMETE) Instrument for each presentation, and 3) The Learner Engagement Instrument (LEI) for each presentation. Previous research supports the response process validity evidence for the app used in this study. 32 All information collected remained anonymous. Participant demographic characteristics include age (years), gender (male, female), practice type (academic, group, solo, government or military, retired), specialty (family medicine, internal medicine, etc.), time spent providing patient care per week (hours), and years in clinical practice (years).
Burnout was assessed using a previously validated 2-item instrument. 33 Burnout was selected as a measure since previous research has demonstrated an inverse relationship to engagement 27. Participants recorded their CMETE instrument scores and LEI responses within the app, which anonymously linked to their demographic data.
Presenter variables included: age (years), academic rank, medical student teaching within the last year (yes or no), resident teaching within the last year (yes or no), fellowship teaching within the last year (yes or no), CME teaching within the last year (yes, no), and provision of direct patient care within the last year (yes or no).
Presentation characteristics included number of slides, summary slides, time of day (morning or afternoon), day of conference, and use of a pearls format. As outlined in prior research, slides which included titles of “pearls/clinical pearls” were counted as using the pearls format 34. A presentation was considered to have a summary slide if a slide listed “Key points”, “Outline”, “Take home message”, “Learning points”, “Some sort of summarized descriptions of the objectives”, or “Conclusions.”
Statistical Analysis
The data were entered and processed by the Mayo Clinic Survey Research Center. Categorical variables were presented percentages and numbers. To build validity to internal structure, an exploratory factor analysis was performed. The minimal proportion criteria were used for factor extracting. Items with factor loadings ≥ 0.50 were retained. Internal consistency reliability for items comprising each factor was calculated with a Cronbach alpha of > 0.7 as acceptable. 35
To account for the clustering of multiple ratings by students completing more than 1 evaluation, we generated an adjusted correlation matrix using mixed model approach. This adjusted correlation matrix was then used to perform confirmatory factor analysis with orthogonal rotation. In addition, for a sensitivity analysis, we also performed factor analysis using an unadjusted correlation matrix (Table 2).
For relations to other variables validity, associations between LEI scores and participant, presenter and presentation characteristics were determined using the Wilcoxon statistic. The level of statistical significance was set at an alpha of 0.05. Statistical analyses were conducted using SAS version 9.4 (SAS Institute Inc., Cary, NC).
Demographics are reported in table 1. A total of 51 out of 206 participants completed the LEI, CMETE and demographic survey (response rate 25%). However, since participants completed surveys on up to 57 lectures, 2,486 LEI and CMETE surveys were submitted out of a total of 2,907 possible surveys for the 51 participants (86% of possible presentations). Of the 51 participants, 30 (59%) were male and 40 (78%) were internal medicine physicians. Over half of respondents (57%) were over the age of 50. The majority of participants were from the Midwestern United States (n=35, 69%).
Table 1
|
Demographics of Participants |
|
|
|
Total |
|
Practice Setting |
|
|
Academic |
10 (19.6%) |
|
Govt/Military |
3 (5.9%) |
|
Group |
30 (58.8%) |
|
Solo |
3 (5.9%) |
|
other |
5 (9.8%) |
|
|
|
|
Specialty |
|
|
IM |
40 (78.4%) |
|
FM |
2 (3.9%) |
|
Medical |
8 (15.7%) |
|
Non-Medical |
1 (2.0%) |
|
|
|
|
Practice Location |
|
|
Northeast |
2 (3.9%) |
|
Southeast |
9 (17.6%) |
|
Midwest |
35 (68.6%) |
|
Southwest |
3 (5.9%) |
|
West |
2 (3.9%) |
|
|
|
|
Reason for Enrolling |
|
|
New board certification |
15 (29.4%) |
|
Renewal of certification |
19 (37.3%) |
|
Not passing prior certification |
1 (2.0%) |
|
General knowledge |
16 (31.4%) |
To support validity to internal structure, an exploratory factor analysis of the LEI was performed as shown in table 2. A sensitivity analysis accounting for the clustering of multiple ratings by students completing multiple evaluations showed no differences between the factor analyses using the adjusted or unadjusted correlation matrixes. Items with a factor loading of 0.5 were retained with no items needing to be removed. Two factors were identified. One factor, which consisted of 4 items, included the domains of emotional engagement and cognitive out-of-class. We termed this new combined domain, “internal engagement.” Internal engagement reflected items which were internal to the learner, such as their enjoyment of the topic and motivation for self-directed learning and applicability of the material. The other factor comprised of the remaining 4 items and including the domains of behavioral engagement and cognitive in-class. We termed this new combined domain, “external engagement.” External engagement consisted of items external to the learner, which could be observed, such as participation during the course and being an active, absorbed learner. The internal consistency reliabilities (Cronbach alpha) were 0.96 for the internal engagement domain and 0.95 for the external engagement domain. Internal consistency was 0.96 for all 8 items overall. The abbreviated 4-item CMETE instrument’s internal consistent was 0.97 which is similar to previously published studies on this instrument’s performance 32.
Table 2
|
Exploratory Factor Analysis of Learner Engagement Instrument (Unadjusted) |
||
|
Questions |
Factor 1 |
Factor 2 |
|
Enjoyed presentation |
0.7013 |
0.4627 |
|
Interested presentation |
0.7471 |
0.4983 |
|
Participated presentation |
0.3751 |
0.6803 |
|
Avoided distractions |
0.4372 |
0.7046 |
|
Active listener |
0.4940 |
0.7487 |
|
Absorbed presentation |
0.4710 |
0.7223 |
|
Apply presentation practice |
0.7357 |
0.4172 |
|
Motivated learn more |
0.7499 |
0.4420 |
|
Exploratory Factor Analysis of Learner Engagement Instrument (Adjusted for Correlation) |
|||
|
Questions |
Factor 1 |
Factor 2 |
|
|
Enjoyed presentation |
0.7120 |
0.4467 |
|
|
Interested presentation |
0.7598 |
0.4761 |
|
|
Participated presentation |
0.3619 |
0.6996 |
|
|
Avoided distractions |
0.4364 |
0.7106 |
|
|
Active listener |
0.5102 |
0.7334 |
|
|
Absorbed presentation |
0.4533 |
0.7372 |
|
|
Apply presentation practice |
0.7377 |
0.4115 |
|
|
Motivated learn more |
0.7546 |
0.4337 |
|
To support relations to other variables validity, the LEI was compared to the CMETE, presentation characteristics, and burnout. Mean scores for the LEI and CMETE are reported in table 3. The LEI was highly correlated to the CMETE (R=0.80). Internal engagement correlated with teaching effectiveness (R= 0.80) stronger than external engagement (R=0.73) There were no differences in the LEI or CMETE scores based on demographics including age, sex, practice location, specialty or reason for enrolling in the course.
Table 3
|
Results of Learner Engagement Instrument and CME Teaching Effectiveness Instrument |
|
|
|
Total |
|
Learner Engagement instrument (LEI) |
|
|
Mean (SD) |
35.6 (7.3) |
|
Median |
38.1 |
|
Q1, Q3 |
34.0, 39.8 |
|
Range Internal consistency |
(0.0-40.0) 0.96 |
|
Factor 1 of LEI (Internal Engagement) |
|
|
Mean (SD) |
17.9 (3.7) |
|
Median |
19.4 |
|
Q1, Q3 |
17.4, 19.8 |
|
Range Internal consistency |
(0.0-20.0) 0.96 |
|
Factor 2 of LEI (External Engagement) |
|
|
Mean (SD) |
17.7 (3.7) |
|
Median |
19.0 |
|
Q1, Q3 |
16.4, 19.9 |
|
Range Internal consistency |
(0.0-20.0) 0.95 |
|
CME Teaching Effectiveness Instrument |
|
|
Mean (SD) |
18.1 (3.8) |
|
Median |
19.4 |
|
Q1, Q3 |
17.6, 19.9 |
|
Range Internal consistency |
(0.0-20.0) 0.97 |
Presentations on the first day of conference had lower mean LEI scores compared to later days (-0.21 on a 5 point scale, p<0.001). Presentations on the first day of conference also had lower CMETE scores (-0.19 on a 5 point scale, p<0.001). Otherwise, presentation characteristics, including use of humor, number of slides, summary slides or use of pearl format and time of day did not influence CMETE or LEI scores. Learners with burnout tended to have higher learning engagement scores, but the results were not significant (-0.79 on a 5 point scale, p=0.41).
The Learner Engagement Instrument (LEI) is an innovative method for measuring engagement in the CME population. The LEI demonstrated strong internal structure validity evidence including a two-factor model that measures engagement in both psychological and behavioral domains. We also identified relations to other variables evidence in terms of a strong relationship between learner engagement and teaching effectiveness, which suggests that effective teachers are more likely to engage their learners.
The LEI is supported by validity evidence according to Messick’s framework for assessment. 26 Regarding content validity evidence, the LEI underwent a rigorous design process that incorporated previously published literature and conceptual frameworks on engagement, along with input from experts and an iterative process of item development. Additionally, to preserve the integrity of the data and support validity to response process, we used a well-developed online platform to collect information with clear instructions was utilized. Additionally, prior to implementation, all LEI items were pilot tested on physicians outside of the study with feedback given on clarity and the perceived meaning of the items. All items were entered and analyzed using a dedicated survey research center. Finally, internal structure validity evidence was supported by factor analysis demonstrating a multidimensional assessment of learner engagement in CME, as well as excellent internal consistency reliability.
Furthermore, relation to other variables validity was supported by substantial correlations between overall LEI and CMETE scores. In summary, effective teachers had more engaged learners. Since the CMETE included items such as speaker clarity, use of case-based presentations, organized slides, and opportunities to learner interactively, faculty should be encouraged to employ these strategies routinely in their teaching sessions. Presentations on the first day of the course had slightly lower LEI scores than presentations on later days. However, this may be related to teaching effectiveness since these presentations also had lower CMETE scores.
We observed a trend towards increased burnout and more engagement rather than less engagement as was previously hypothesized. However, it’s possible that participants with higher burnout tend to exhibit higher focus, or perhaps participants with higher burnout have reasons to be more concerned about their future performance on certification examinations. Furthermore, the high-stakes nature of a board review course may contribute to increased burnout. More study is needed to identify if engagement and burnout are related in a CME population. Studying engagement and burnout in a lower stakes CME environment should also be considered.
Though we expected three domains, our exploratory factor analysis of the LEI revealed a two-dimensional instrument. A two-domain conceptual framework for engagement is also supported in the literature12. The revised framework for engagement based on our factor analysis is shown in figure 2. The items of emotional engagement and cognitive out-of-class engagement loaded together in the factor analysis. These domains focused on items internal to the learner, which cannot be seen by an outside observer. This includes a learner’s enjoyment and interest in the topic as well as the learner’s motivation to learn and apply the topic in their practice. 19, 23, 24Given that these items were internal preferences or motivations, we termed this new domain, “Internal Engagement.”
Additionally, behavioral engagement and cognitive in-class engagement loaded together in the factor analysis. Both these domains focused on the external actions of the learner while in class. This includes participating, avoiding distractions, listening and being absorbed in the presentation.19, 23, 24 Given that these items were in-class actions which could be observed by an external observer, we termed this engagement, “External Engagement.”
The domains of internal and external engagement parallel concepts in self-determination theory. Self-determination theory describes how humans are motivated to learn and includes concepts of amotivation, extrinsic motivation and intrinsic motivation. 36 Intrinsic motivation is internal and includes the learner’s personal interest and enjoyment of the material. 37 This is similar to our identified domain of internal engagement. A learner who has high internal engagement may find a topic enjoyable and align with their personal interests. Conversely, extrinsic motivation describes external pressures that influence a learner’s motivation to learn. This could include career goals, rewards, or societal demands. 37 A learner who has a high external engagement may be cognizant of the social pressures to avoid distractions and be an active participant, consequently also having high external engagement in the presentation. More research is needed to further explore the overlap between these theories on engagement and motivation to learn.
There are several reasons to explain why a third domain wasn’t identified in our factor analysis. First, previously published engagement inventories at the high school and college-level include more items for each domain. 13, 14, 25 Given that the LEI had only eight items, there may be too few items to discriminate between various domains of engagement. Second, as outlined in previous conceptual frameworks, there may only be two domains of engagement, a psychological-based domain and behavioral-based domain, with cognitive engagement existing in an overlap between the two. Third, CME attendees may be inherently different from college-level students and the domain of cognitive engagement may not be relevant to this population. Lastly, as noted above, our framework for engagement may overlap with the concepts of motivations to learn where internal and external engagement relate to intrinsic and extrinsic motivation to learn.
Our study has several strengths. As previously noted, the LEI is supported by strong validity evidence using Messick’s framework to design the instrument and underwent a vigorous design process. 26 Our sample was collected in a large, didactic-style CME course, which is representative of most types of CME. Last, although our response rate was modest, over 2500 LEI and CMETE surveys were answered. There are also several limitations. As noted above, our response rate was low to modest, which likely introduces selection bias. It is possible that only the most engaged learners completed the survey, biasing our results. This could artificially inflate learner engagement scores and skew our results. Furthermore, we were unable to collect comparative information on nonresponders. Studying the engagement of these initial nonresponders could be a potential area for future research. Additionally, engagement isn’t dichotomous and we didn’t identify a score cutoff differentiating between highly engaging presentations and non-engaging ones.
This smaller sample size may have also contributed to a type II error for burnout if our sample size was not adequate to detect a relationship between engagement and burnout. Future research on the relationship between burnout and engagement should be sought with larger sample sizes. Furthermore, additional study on engagement and relationship to teaching effectiveness in other populations, such as undergraduate and graduate medical education should be pursued.
Engagement is an important construct for learner retention, but measuring engagement can be challenging. Overall, the LEI demonstrated strong validity evidence and showed that more effective teachers, as assessed by students' evaluations, were associated with higher student engagement. Given the relationship between learner engagement and teaching effectiveness, identifying more engaging and interactive methods for teaching in CME is recommended.
Continuing Medical Education (CME)
American Board of Internal Medicine (ABIM)
Learner Engagement Instrument (LEI)
Continuing Medical Education Teaching Effectiveness (CMETE)
Ethics Approval and Consent to Participate: Mayo Clinic’s Institutional Review Board reviewed the study protocol and deemed the study exempt. A written letter was given to all participants stating that by completing the LEI questionnaires, participants were implying consent to participate in this study.
Consent for publication: Not applicable
Availability of data and materials: Data for this study is available at the following hyperlink: https://www.dropbox.com/s/wrkjc29g1gv41cd/2018%20Engagement%20Dataset.xlsx?dl=0
Competing interests: The authors declare that they have no competing interests.
Funding: None.
Authors contributions: Authors CRS, SB, AS, LR, TB and CW designed the LEI after reviewing the literature and participated in the iterative design process. Authors CRS and SB collected the data at the CME course. Authors CDS and JM performed statistical analysis. All authors contributed to the manuscript including creation and revision. All authors read and approved the final manuscript.
Acknowledgements: Not applicable
- Cervero RM, Gaines JK. The impact of CME on physician performance and patient health outcomes: an updated synthesis of systematic reviews. Journal of Continuing Education in the Health Professions. 2015;35:131-138.
- Warden GL, Mazmanian PE, Leach DC. Redesigning continuing education in the health professions: Washington, DC: Institute of Medicine, National Academy of Sciences; 2010.
- Nissen SE. Reforming the continuing medical education system. Jama. 2015;313:1813-1814.
- Campbell EG, Rosenthal M. Reform of continuing medical education: investments in physician human capital. JAMA. 2009;302:1807-1808.
- Davis DA, Prescott J, Fordis Jr CM, et al. Rethinking CME: an imperative for academic medicine and faculty development. Academic Medicine. 2011;86:468-473.
- Mazmanian PE, Davis DA. Continuing medical education and the physician as a learner: guide to the evidence. Jama. 2002;288:1057-1060.
- Stephenson CR, Wang AT, Szostek JH, et al. Flipping the continuing medical education classroom: Validating a measure of attendees' perceptions. Journal of Continuing Education in the Health Professions. 2016;36:256-262.
- Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. Int J Technol Assess Health Care. 2005;21:380-385.
- Michael J. Where's the evidence that active learning works? Advances in physiology education. 2006;30:159-167.
- Prober CG, Heath C. Lecture halls without lectures—a proposal for medical education. N Engl J Med. 2012;366:1657-1659.
- Bernard JS. Student Engagement: A Principle-Based Concept Analysis. International journal of nursing education scholarship. 2015;12.
- Astin AW. Student involvement: A developmental theory for higher education. Journal of college student personnel. 1984;25:297-308.
- Burch GF, Heller NA, Burch JJ, Freed R, Steed SA. Student engagement: Developing a conceptual framework and survey instrument. Journal of Education for Business. 2015;90:224-229.
- Gunuc S, Kuzu A. Student engagement scale: development, reliability and validity. Assessment & Evaluation in Higher Education. 2015;40:587-610.
- Williams RL, Eggert AC. Notetaking in college classes: Student patterns and instructional strategies. The Journal of General Education. 2002:173-199.
- Kiewra KA. A review of note-taking: The encoding-storage paradigm and beyond. Educational Psychology Review. 1989;1:147-172.
- Crenshaw K, Curry W, Salanitro AH, et al. Is physician engagement with Web-based CME associated with patients’ baseline hemoglobin A1c levels? The Rural Diabetes Online Care study. Academic medicine: journal of the Association of American Medical Colleges. 2010;85:1511.
- Fredricks JA, Blumenfeld PC, Paris AH. School engagement: Potential of the concept, state of the evidence. Review of educational research. 2004;74:59-109.
- Appleton JJ, Christenson SL, Kim D, Reschly AL. Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology. 2006;44:427-445.
- Alrashidi O, Phan H, Ngu B. Academic Engagement: An Overview of Its Definitions, Dimensions, and Major Conceptualisations2016.
- Moreira P, Cunha D, Inman RA. An integration of multiple student engagement dimensions into a single measure and validity-based studies. Journal of Psychoeducational Assessment. 2019:0734282919870973.
- Whitney BM, Cheng Y, Brodersen AS, Hong MR. The scale of student engagement in statistics: Development and initial validation. Journal of Psychoeducational Assessment. 2019;37:553-565.
- Pickering JD, Swinnerton BJ. Exploring the dimensions of medical student engagement with technology‐enhanced learning resources and assessing the impact on assessment outcomes. Anatomical Sciences Education. 2019;12:117-128.
- Trowler V. Student engagement literature review. The higher education academy. 2010;11:1-15.
- Kuh GD. The national survey of student engagement: Conceptual and empirical foundations. New directions for institutional research. 2009;2009:5-20.
- Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American psychologist. 1995;50:741.
- Schaufeli WB, Salanova M, González-Romá V, Bakker AB. The measurement of engagement and burnout: A two sample confirmatory factor analytic approach. Journal of Happiness studies. 2002;3:71-92.
- Reeve J, Tseng C-M. Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology. 2011;36:257-267.
- O’Malley KJ, Moran BJ, Haidet P, et al. Validation of an observation instrument for measuring student engagement in health professions settings. Evaluation & the health professions. 2003;26:86-103.
- Ema P, Rovan D, Ljubin Golub T. Development and validation of the Academic Engagement Scale (AES)2016.
- Kuh GD. Assessing what really matters to student learning inside the national survey of student engagement. Change: The Magazine of Higher Learning. 2001;33:10-17.
- Wittich CM, Mauck KF, Mandrekar JN, et al. Improving participant feedback to continuing medical education presenters in internal medicine: a mixed-methods study. Journal of general internal medicine. 2012;27:425-431.
- West CP, Dyrbye LN, Sloan JA, Shanafelt TD. Single item measures of emotional exhaustion and depersonalization are useful for assessing burnout in medical professionals. Journal of general internal medicine. 2009;24:1318.
- Kopecky SL, Litin SC. Clinical pearls in cardiology. Mayo Clinic Proceedings. Vol 85: Mayo Foundation; 2010:473.
- DeVellis RF. Guidelines in scale development. Scale Development: Theory and Applications. Newbury Park, Calif: Sage. 1991;5191.
- Deci EL, Koestner R, Ryan RM. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological bulletin. 1999;125:627.
- Cook DA, Artino Jr AR. Motivation to learn: an overview of contemporary theories. Medical education. 2016;50:997-1014.
- Alrashidi O, Phan HP, Ngu BH. Academic engagement: an overview of its definitions, dimensions, and major conceptualisations. International Education Studies. 2016;9:41.