Introducing a clerkship curriculum based on entrustable professional activities: a pilot study


 BackgroundGeneral clerkship quality criteria have not been studied after introducing a curriculum based on entrustable professional activities (EPAs). Therefore, we conducted a pilot study to explore educational outcomes of an EPA-based clerkship curriculum reform.MethodsWe collected multiple self-assessment and evaluation data on levels 1-3 of the New World Kirkpatrick model (KPM). For level 1 (reaction) we analyzed curriculum evaluations and verbal and written students’ feedback. For level 2 (learning) pre- and post-clerkship self-assessments of perceived need for supervision for each EPA and students’ written learning reflections based on patient mix exposure were analyzed. For level 3 (behavior) we evaluated workplace-based assessments.ResultsThe first student cohort in the new EPA-based clerkship completed 180 self-assessments (18 per student) of need for supervision and 63 documented workplace-based assessments (average of 6.3 per student in four weeks). On KPM level 1 we saw a high overall satisfaction with the clerkship (average of 4.9, range: 4.0-5.0 on a 5-point Likert scale). In written evaluations and feedback rounds students pointed out the importance of structured bedside teaching in the first clerkship week. On KPM level 2 the overall decrease of self-assessed need for supervision before and after the clerkship was two supervision levels (from direct to indirect supervision) and statistically significant (p < 0.05). For three EPAs students reached indirect supervision levels. Learning reflections and patient mix analysis indicated that students were exposed to a wide range of diagnostic categories (ICD-10: F0 – F6) and received actionable feedback for communication skills. On KPM level 3 clinical supervisors predominantly used EPAs 1 (History taking), 2 (Assessing mental status) and 8 (Documentation and presentation) for workplace-based assessments. We saw a decreasing need for supervision from the first to the last week in the clerkship according to the supervisors’ judgements.ConclusionStudents reacted positively to introducing an EPA-based clerkship curriculum. The EPA-oriented formative self-assessments and workplace-based assessments seemed to support achievement of competency-based learning goals. However, more in-depth understanding of the entrustment process in the clerkship context is necessary to fully leverage the potential of an EPA-based clerkship.


Abstract
Background General clerkship quality criteria have not been studied after introducing a curriculum based on entrustable professional activities (EPAs). Therefore, we conducted a pilot study to explore educational outcomes of an EPA-based clerkship curriculum reform.

Methods
We collected multiple self-assessment and evaluation data on levels 1-3 of the New World Kirkpatrick model (KPM). For level 1 (reaction) we analyzed curriculum evaluations and verbal and written students' feedback. For level 2 (learning) pre-and post-clerkship self-assessments of perceived need for supervision for each EPA and students' written learning re ections based on patient mix exposure were analyzed. For level 3 (behavior) we evaluated workplace-based assessments.

Results
The rst student cohort in the new EPA-based clerkship completed 180 self-assessments (18 per student) of need for supervision and 63 documented workplace-based assessments (average of 6.3 per student in four weeks). On KPM level 1 we saw a high overall satisfaction with the clerkship (average of 4.9, range: 4.0-5.0 on a 5-point Likert scale). In written evaluations and feedback rounds students pointed out the importance of structured bedside teaching in the rst clerkship week. On KPM level 2 the overall decrease of self-assessed need for supervision before and after the clerkship was two supervision levels (from direct to indirect supervision) and statistically signi cant (p < 0.05). For three EPAs students reached indirect supervision levels. Learning re ections and patient mix analysis indicated that students were exposed to a wide range of diagnostic categories (ICD-10: F0 -F6) and received actionable feedback for communication skills. On KPM level 3 clinical supervisors predominantly used EPAs 1 (History taking), 2 (Assessing mental status) and 8 (Documentation and presentation) for workplace-based assessments. We saw a decreasing need for supervision from the rst to the last week in the clerkship according to the supervisors' judgements.

Conclusion
Students reacted positively to introducing an EPA-based clerkship curriculum. The EPA-oriented formative self-assessments and workplace-based assessments seemed to support achievement of competencybased learning goals. However, more in-depth understanding of the entrustment process in the clerkship context is necessary to fully leverage the potential of an EPA-based clerkship.

Background
A new competency-based national learning catalogue has been introduced in Switzerland for undergraduate medical education [1]. The PROFILES-catalogue (Principal Relevant Objectives and Framework for Integrated Learning and Education in Switzerland) includes nine entrustable professional activities (EPAs) and serves as a blueprint for clinical curricula in undergraduate medical education (UME). However, there is currently only limited evidence available on how to introduce and work with EPAs in clinical rotations, such as clerkships. While there has been attention to implementing EPA-based curricula from a general health professions and graduate medical education perspective, little attention has been paid to general quality criteria of clerkships after introducing EPAs [2,3].
Introducing an EPA-based clinical curriculum can involve selection and development of EPAs, implementation of new teaching activities and reforming assessment instruments. Clinical educators in different specialties started to work with EPAs in UME in the context of clerkships, subinternships, electives or boot-camps [4][5][6][7]. These EPA-based curricula are mostly not embedded in larger programmatic assessment frameworks [8] but are limited to few weeks of continuous or spaced clinical exposure in one clinical specialty [9,10]. Only few medical schools have started to implement comprehensive assessment structures to implement EPA-based UME curricula [11].
Early research in the context of EPA-based clerkship curricula has mostly focused on three aspects: development of specialty speci c EPAs [12], different implementation aspects of EPAs [13] and assessment of EPAs with supervision scales and actionable feedback [2,[14][15][16]. Although selfassessment of competence is not reliable [17], self-assessment of self-e cacy related constructs, such as perceived need for supervision, might be valuable in primarily formative assessment contexts. We did not nd studies that explored formative assessment aspects when introducing an EPA-based clerkship curriculum. Additionally, general quality criteria of clerkships were partly not considered after EPA-based curricular reforms in clerkships. These criteria include student satisfaction, perceived and observed learning progress, patient mix exposure and behavioral change [18,19].
Our goal for this pilot study was to better understand general quality criteria and associated educational outcomes when introducing EPAs to a clerkship curriculum. Therefore, we explored multiple data sources on the rst three levels of the New World Kirkpatrick model [19] of an EPA-based clerkship curriculum reform. The results from this study inform continuing curriculum reform and might be relevant in particular for clinical educators planning to or already implementing EPA-based clinical curricula as well as educators involved in larger UME curriculum design.

Development of an EPA-based clerkship curriculum
The theoretical models that informed the implementation process included concepts of entrustment in the workplace [14,20], programmatic assessment perspectives [21] and competency-based undergraduate learning and teaching [1,22,23]. The Swiss national catalogue (PROFILES) is based on nine core EPAs, which are not specialty-speci c [1]. A clerkship syllabus with nested EPAs for psychiatry was developed. An open-source learning management system (ILIAS) was used to collect selfassessment and evaluation data [24].

Setting and population
Medical students at the University of Bern, Switzerland have to complete ve mandatory core clerkships during their fourth year in a six-year medical curriculum (Bachelor and Master of Medicine). Each clerkship lasts four weeks and includes the specialties of internal medicine, gynecology, pediatrics, surgery and psychiatry. Our psychiatric teaching hospital (3'800 inpatients per year, > 10'000 outpatients per year) trains about ten students per month with each teaching ward taking on one to two medical students per month (n (approximately) = 100 per academic year). The interprofessional staff of one ward typically consists of one attending physician and two residents in addition to nurses, a psychologist, a social worker, and takes care of 20-22 hospitalized patients. We use a pass-fail-grading system for clerkship students with several mandatory formative assessments (at least four workplace-based assessments, four documented patient admissions, and one scienti c paper or clinical case presentation per student).

Sample
Our sample for this explorative pilot study consisted of ten medical students with an average age of 22.5 years, nine female students and one male. Eight students originated from primarily German speaking parts of the country and two from primarily French speaking parts. Two students reported prior psychiatry experience (one through clinical skills training, one through relatives with a psychiatric diagnosis). All students and clinical teaching staff were introduced through either a seminar or written information material about the EPA-based clerkship curriculum reform.

Pilot study
We used a mixed-methods explorative study design to describe the implementation process of an EPAbased core clerkship curriculum reform in psychiatry in February 2019. We used qualitative and quantitative program evaluation data collected from medical students, each spending one month on one of ten wards in a single academic teaching hospital in Bern, Switzerland. We used a published supervision scale for each nested psychiatry clerkship EPA [25] for self-assessments of need for supervision (six levels). We assessed the educational curriculum outcomes on levels 1-3 of the New World Kirkpatrick model [19]. On level 1 (satisfaction) we analyzed students' evaluation of the clerkship curriculum and students' in-between and end-of-clerkship feedback. On level 2 (learning) we analyzed self-ratings of perceived need for supervision for each EPA of the PROFILES catalogue before and after the clerkship rotation. In order to explore learning re ections based on patient mix exposure, we collected students' documentation of patient encounters including patient age, route of admission, diagnosis and perceived learning from the admission. On level 3 (behavioral change) we analyzed all supervisor rated workplace-based assessments. Assessment and evaluation data were either collected through paperbased evaluation forms or electronically via ILIAS throughout February 2019.

Ethics and analysis of data
The ethics committee of the canton of Bern reviewed the research design and exempted the study from additional ethical approval. Con dentiality and anonymity with regards to electronic data was maintained throughout the study. Any names or potentially identifying information were removed before analyzing the data and quotes were all translated from (Swiss-)German to English. We used the statistical software R to analyze supervision scale ratings with the Wilcoxon signed-rank test.

Results
New EPA-based psychiatry clerkship curriculum A clerkship curriculum based on the PROFILES catalogue was developed. Ten students participated in the new EPA-based psychiatry curriculum and spent four weeks (approximately 50 hours per week) on ten different wards. The core EPAs with nested psychiatry clerkship EPAs are shown in Table 1. Students were introduced to the clerkship structure in a clerkship orientation seminar at the rst day of the clerkship. In addition to working on the wards and taking part in routine ward and department meetings, students participated in six one-hour didactic clerkship seminars (on psychopathology, psychiatric interventions, old-age psychiatry, psychosis, stress-related disorders, personality disorders, a scienti c paper presentation exercise and a scienti c journal club). Furthermore, they participated in off-site visits to specialized addiction treatment facilities, which included a patients-as-teachers seminar and the child and adolescent psychiatric clinic. KPM Level 2 educational outcome (learning): Students' self-assessment of need for supervision regarding EPAs before and after the clerkship and learning re ections Each student rated his or her need for supervision for nine nested psychiatry clerkship EPAs in the beginning of the clerkship (asked on rst day) and the end of the clerkship (asked on last day) on a 6point Likert scale (range from 1 = "I can only observe this activity", to 2 = "I can do this only as a co-activity with the supervisor", 3 = "I can do this activity, if the supervisor is present", 4 = "I can do this, if the supervisor completely repeats the activity", 5 = "I can do this, if the supervisor repeats the important parts of the activity", 6 = "I can do this, if I can ask for help when I need it"). Changes differed across EPAs and intraindividually. The average change of supervision need for all nine EPAs was 2.9, which is equivalent to moving from observing an EPA to doing an EPA independently and having a resident check the activity.
However, the change in supervision level ranged from 1 to 6. Single EPA change (except of EPA 9) and overall change of self-assessed need for supervision was statistically signi cant (Wilcoxon signed rank test, p < 0.05). Results are shown in Figure 1.
With regards to personal learning re ections, each student documented four patient admissions including admission diagnosis and personal learning effect. The full range of psychiatric disorders was covered in the clerkship. Documented admissions included schizophrenic disorders (n = 11, 28%), affective disorders (n = 10, 25%), substance abuse disorders (n = 5, 13%), anxiety and stress related disorders (n = 5, 13%), personality disorders (n = 3, 8%), organic illness (n = 1, 3%), eating and impulse control disorders (n = 1, 3%), and two admissions did not contain a diagnosis (5%). The vast majority of written re ections addressed various aspects of communicating with psychiatric patients (e.g. "adjust communication strategy to a patient who is logorrheic and long-winded, in such a case use short and precise questions early on" or "learned how to deal with a patient who doesn't want to answer a question and refused a physical exam, communicate acceptance"). Fewer comments addressed speci c aspects of the mental status exam and planning and structuring a patient admission.
With regards to written feedback, 97% of assessment forms contained written feedback. All feedback forms were signed off by residents, except one which was signed off by a nurse. No assessment form was signed off by attending physicians, psychologists or social workers. Written feedback was predominantly formulated as a short positive a rmation of what was done well (e.g. "structured mental status examination" or "good communication during physical examination, empathic attitude"). Only few written feedbacks also contained actionable areas of future improvement (e.g. "try to interrupt logorrheic patient next time" or "add disorder speci c questions when interviewing relatives, such as memory function in suspected dementia"). Comparing and contrasting written feedback per student indicated that supervisors in some cases referred back to previous feedbacks and validated progress (e.g. from "practice written summary of mental status" to "precise summary of mental status, pay attention to relevant information for other team members, such as housing, nancial situation and family situation).

Discussion
We used mixed-methods to explore experiences with and educational outcomes of introducing an EPAbased clerkship curriculum on New World Kirpatrick's levels 1-3 [19]. Overall, the data from this pilot study indicate positive learning outcomes on various self-reported measures, including general clerkship satisfaction and a decrease in perceived need for supervision across nine EPAs. The trend for workplacebased assessments also indicated a decrease of need for supervision. However, clinical supervisors predominantly assessed EPAs 1 (Take a patient's psychiatric history), 2 (Assess mental status) and 8 (Document and present a clinical encounter with a psychiatric patient).

KPM Level 1 (satisfaction): Students' evaluation of the EPA-based clerkship curriculum experience
The overall students' experience with an EPA-based clerkship curriculum was positive and better as compared to our overall evaluation from the previous year of the same month. Studies and educational reports from other specialties revealed similar results concerning EPAs in their clerkship curricula [15,26,27]. A possible explanation might be, that structuring a clerkship curriculum around EPAs provides a learning scaffold that is closer to clinical working [28].
KPM Level 2 (learning): Students' self-assessment of need for supervision regarding EPAs before and after the clerkship and learning re ections Changes in perceived need for supervision were strongest and most homogenous for those EPAs (1, 2 and 8) that had been used for workplace-based assessments. EPAs 3 (Prioritize a psychiatric differential diagnosis), 4 (Order and interpret tests for psychiatric patients), 6 (Recognize and treat psychiatric emergencies) and 7 (Prescribe and develop a management plan for a psychiatric patient) showed very heterogenous changes. EPAs 5 (Initiate involuntary treatment) and 9 (Identify and report opportunities to improve patient safety in a psychiatric hospital) showed almost no change. Potential reasons include complexity of the tasks [29] or simply curricular gaps. Further research into relevant threshold concepts to effectively support students in mastering these EPAs is needed.
To explore learning progress of clerkship students we used the patient mix, a measure typically based on patient volume and patient diversity in terms of diagnoses [18] in combination with written learning re ections. The four most frequent diagnoses that students were entrusted to admit were schizophrenic disorders, affective disorders, substance abuse disorders, and anxiety and stress related disorders. Patient mix has been shown in a systematic review to correlate positively with self-reported learning outcomes, such as self-con dence, comfort-level, perceived quality of the learning experience, perceived effectiveness of the rotation and instructional quality [18]. Our chosen minimal patient volume (four admissions per student) in terms of documented patient admissions and resulting learning re ections did allow for several self-reported improvements in learning outcomes.

KPM Level 3 educational outcome (behavioral change): Workplace-based assessments of EPAs in the clerkship
Workplace-based assessments were frequently documented during the clerkship and showed a slight trend towards less direct supervision for three EPAs. The other six core EPAs were not assessed during the rst month of our EPA-based psychiatry clerkship. Other researchers similarly reported selective assessment of core EPAs or developing clerkship speci c sets of nested EPAs [4,30]. Probably the expectation of which EPAs should be assessed needs to be communicated more clearly to both trainees and clinical supervisors as described in the context of graduate medical education [31]. Barriers and facilitators to learning speci c EPAs as well as to assess these in UME need to be better understood.

Limitations
Our pilot study is limited by the small sample and these preliminary results need to be con rmed.
However, we used multiple evaluation strategies, including quantitative and qualitative data to explore early EPA-based clerkship curriculum innovations. Currently, we are collecting data for the full academic year to analyze the robustness of our ndings. We already were able to use insights from this pilot study to adapt our clerkship curriculum and work with teaching residents to inform curricular changes.

Implications for practice
Several aspects from our explorative study seem relevant for clerkship directors working with EPA-based curricula. On the most basic level, our study indicates that introducing an EPA-based clerkship curriculum is feasible in smaller scale clinical teaching contexts and that students reacted positively. This might be in particular interesting for departments planning to work with EPAs before longitudinal assessment strategies for EPA-based undergraduate medical education curricula have been implemented. Selfassessment of perceived need for supervision can effectively be used in this context to explore changes for speci c EPAs during a clerkship. Based on our observations the deliberate use of formative workplacebased assessments might be a potential driver of achieving learning goals with regard to EPAs.
With clinical residents as primary supervisors of clerkship students, they should receive adequate support, training and instruction for their role as clinical teachers. From a clerkship curriculum management perspective, using self-assessment of perceived need for supervision can help clerkship directors to identify weaknesses in the curriculum and adjust curriculum content accordingly. For example, we found that students seemed to struggle more with identifying differential diagnosis and formulating a management plan for psychiatric patients as compared to EPAs like taking the psychiatric history and assessing the mental status. Exploring underlying threshold concepts and adapting or designing new curricular elements to help students achieve these competencies are consequences of our pilot study.

Conclusion
Our pilot study indicates, that introducing an EPA-based clerkship curriculum can help to focus educational resources on competency-based learning goals. Teaching skills of clinical residents seem to be essential. Future research should further explore the entrustment process in the context of clerkships. Authors' contributions SP, CN and SH conceptualized the study. SP and AC were responsible for the data collection and analysis. SK, WS and CN contributed to curriculum design, data interpretation and manuscript revision. All authors were involved in the nal data synthesis and in drafting and revising the nal manuscript.   Changes of perceived need for supervision per EPA. Levels of supervision: 1 = "I can only observe this activity", to 2 = "I can do this only as a co-activity with the supervisor", 3 = "I can do this activity, if the supervisor is present", 4 = "I can do this, if the supervisor completely repeats the activity", 5 = "I can do this, if the supervisor repeats the important parts of the activity", 6 = "I can do this, if I can ask for help when I need it". Total of self-assessments: n = 180 (90 pre-and 90 post-clerkship). Statistical analysis:

Declarations
Wilcoxon signed-rank test.