Study design
This study is a quasi-randomized controlled educational research study comparing the effects of two different instructional approaches on the acquired practical skills during undergraduate surgical training.
Study participants
Study participants were undergraduate medical students at Goethe-University Hospital in Frankfurt in the fourth year of a 6-year program completing their obligatory surgical training. All participants were naïve regarding CMF-specific practical skills and knowledge.
Participation was voluntary and took place after written informed consent, which was revocable at any time. Students were blinded in relation to their knowledge of the didactic principles used during their training as well as affiliation to any study group. Basic data regarding student age, sex, and duration of the study were collected using a questionnaire.
The study was conducted according to the ethical principles of the Helsinki Declaration (Ethical Principles for Medical Research Involving Human Subjects), and the local ethics committee noted that no further approval was necessary.
Study protocol
The study was carried out within the ‘training week of practical clinical skills in surgery’ [22], which aims to teach students basic surgical competencies and prepare them for their upcoming surgical rotations. This takes place at the skills lab and consists of 12 teaching units for basic general and surgical skills from all surgical disciplines. The teaching content is based on the learning objectives for practical skills defined in the national competency-based catalog of learning objectives in surgery from the German Society of Surgery [23]. The training has a capacity of 64 students per week, with a maximum of eight students per group and tutor.
Following the first week of practical skills training, students pass through two weeks of surgical rotation at the university hospital or associated teaching hospitals with participation on the ward, the ambulance, and in the operating theatre to integrate and apply the acquired surgical knowledge and skills under supervision in daily clinical practice. In the two weeks of surgical rotation no additional CMF-specific training took place since only around 1 to 2 % of students passed their surgical rotation at the Department of Cranio-Maxillofacial and Facial Plastic Surgery.
Two fundamental clinical competences from the CMF spectrum, namely the performance of a structured facial examination (SFE, video at https://youtu.be/S-b3kIzmLQw, accessed 17.04.2020) and the packing of a modified Bellocq tamponade (BT, video at https://youtu.be/gtMq4044RlM, accessed 17.04.2020) were examined (Figure 1). These competencies were chosen since previous studies found medical students to have significant shortcomings in CMF-specific competencies [24–26], although consultations in the field are of considerably socioeconomic and numerical importance in the acute and normal care [28,29].
The existing instructional manual for the CMF unit was reworked and adapted for the study. Each manual consists of a detailed schedule and workflow, as well as a step-by-step checklist to ensure a standardized sequence of the training. For further quality assurance and standardization, an instructional video was designed for both skills [30]. All instructors involved received an online-training, where both skills were demonstrated and trained in each of the instructional approaches as well as the correct performance of each instructional approach.
Intervention
The assignment of students to one of the eight learning groups per training week with a maximum of six students per group who pass through the teaching units together occurred prior to the training week, independent of the authors and independent of study participation by the deanery. The allocation of the learning group in the study to the two instructional approaches was performed alternately.
‘Video See One, Do One Approach’
This approach is labeled as the main component of clinical-bedside teaching. Students learn by watching an expert explaining and demonstrating a skill. This is followed by the first independent performance of the skill, which is mostly with a patient [4]. As quality assurance and standardization for the demonstration of each skill from the trainers, each skill was videotaped based on the existing manual and checklist. The trainer demonstrated the video and explained the skill in detail, step-by-step. The explanations were predetermined in the manual and trained in the tutor training. Subsequently, students could practice the skill under the supervision on peers (SFE) or on training models (BT) and, if needed, receive correction from the tutor. Each student should practice each skill at least once. The training lasted for 60-min.
‘Video 4-step Approach’
For this study, the ‘4-step Approach’ – as described by Walker and Peyton [10] – was modified. For Step 1, the video was demonstrated without any comments. For Step 2, the trainer demonstrated the video and explained the skill step-by-step in detail. The explanations were predetermined in the manual and trained in the online-tutorial. For Step 3, the video was paused after each step, and students were chosen one-by-one by the trainer to explain the next instructional step of the video, which was then video-played. Step 4 was performed, as described by R. Peyton. Here, students explained every upcoming step of the procedure and after approval by the trainer carried it out under supervision.
Subsequently, students could practice the skills under the supervision and, if needed, receive correction from the tutor. Each student was advised to practice each skill at least once. The training lasted for 60-min.
Outcome measures
To assess the acquired competence in both skills of the study, the OSCE-format (one station for each skill) was used during the training week directly after the intervention (T1) and 5–13 (SD = 3.16) weeks later (T2), as part of the curricular and summative surgical OSCE. The surgical OSCE consists of 10 stations in total, two of them regarding the structured facial examination and the packing of a Belloq’s tamponade. A trinary scoring scale was used (0 points for not done, 1 point for done but incorrect, and 2 points for done and correct) for each checklist, which was based on the checklist used in the tutor manual. A timeframe of 5-min to complete each OSCE station was given. Subsequently, students received short feedback regarding their performance and suggestions for improvement at T1. During their training week, students were video-recorded (Camera System: Panasonic HC-X929) for later performance measurement. Due to data protection reasons the videos were blinded regarding the student names and were deleted after analysis. The checklists implemented were primarily piloted in previous undergraduate trainings and afterwards validated by two independent, blinded examiners. Therefore, inter-rater reliability was measured using Pearson’s correlation coefficient (r). In addition, the content validity was ensured through the creation as part of an expert workshop with didactic and surgical experts as well as through the repeated application and adaption in the context of previous studies [8,29,30] and OSCE exams. During their surgical OSCE, due to examination regulations, the performance was not video-recorded and measured by only one examiner for each skill using the same checklists. Examiners were two experienced surgeons working in CMF surgery. All examiners were blinded towards the students’ instructional approach and affiliation of the learning group. They received training before the OSCE to gain experience in the use of the checklist.
Data analysis
Microsoft Excel 2016 (Microsoft Office 2007, © Microsoft Corporation, Redmond, USA) for Mac and SPSS Statistics version 19 (IBM, Armonk, USA) were used for the statistical analysis and graphical display of data.
To test for normal distribution the Kolmogorov-Smirnov test for one sample was carried out. Since the data was not normally distributed the Mann-Whitney-U-Test for non-parametric data was used to test for significant differences in learning success between the ‘Video 4-step Approach’ and ‘See One, Do One Approach’. To test for performance differences within the respective groups at different times, the Friedman test for repeated measures was used. Furthermore, effect sizes were calculated using eta squared (η2), which is defined as the square of the correlation ratio (η), resulting in a unitless value that helps to interpret the effect size of observed results and hence the statistical power of a study. For most types of effect sizes, a larger absolute value indicates a stronger effect. As a rule of thumb values ≤ 0.01 indicate a relatively small effect, values ≤ 0.06 indicate a medium effect size and values ≥ 0.14 indicate a strong effect. Furthermore, it can be used as an additional control test since prior studies have shown that significant test results alone are not sufficient to interpret data and draw conclusions [31].
Pearson’s correlation coefficient (r) was used to calculate the inter-rater reliability between both raters at T1.
Additionally, each item of the checklists that were used to measure student performance was categorized into four (extraoral examination, neurological examination, midface examination, intraoral examination) or three (material preparation, insertion of the tamponade, fixation of the tamponade) subgroups. Averages from those subgroups were checked for significant differences using the Mann-Whitney-U-Test for non-parametric data.
Sample size estimation
Based on prior examination results from the years before the intervention, we estimated an average student performance of 70% with a standard deviation of 10% in the OSCE. A sample size of 88 (44 per study arm) was calculated based on the following parameters: mean ‘4-step Approach’ = 33, mean ‘See One, Do One’ = 30, SD = 10, alpha = 0.05, beta = 0.2, Power = 80%.