Study design
All of the ethical principles for medical research involving human subjects dictated by the 1975 declaration of Helsinki, as revised in 2013, were considered. Concordant with the Ethics Board at the University Medical School, no ethical permission was necessary to conduct the study. The study was prospective, blinded, randomized, and controlled with the following parallel arms (feedback methods):
Control group: Direct expert feedback (DEF)
Intervention group 1: Individualized video feedback (IVF)
Intervention group 2: Unsupervised video feedback (UVF)
The study measured the student’s improvement of performance and the time taken to perform a task directly after an introduction exercise (T0) and a second exercise after receiving feedback (T1). Finally, to measure the long-term learning retention, a final examination (T2) was performed six weeks after the post-test.
The assignment of students to one of the learning groups per training week with a maximum of six students per group who passed through the teaching units together occurred prior to the training week, independent of the authors and independent of study participation by the deanery. The allocation of the learning groups in the study to the three instructional approaches was performed alternately.
Study participants and study conduction
The study participants were fourth-year dentistry students from the University Goethe of Frankfurt in the period of 2018 - 2019 attending a compulsory internship which includes a five-day rotation through every section of the Department of Oral, Cranio-Maxillofacial and Facial Plastic Surgery, i.e. the operating room, the outpatient clinic and the emergency department. Before starting their rotation, students have to complete practical skills training which has been described in greater detail in a previous publication. 14Teaching was held in small groups ranging from four to six students by the same instructor (a 4th year resident who is responsible for the undergraduate education of the Department) throughout the course of the study. Prior to the study beginning, the instructor received training which included the learning objectives of the practical skills training, a tutor manual that included an explanation on how to correctly perform each skill as well as a timetable and blueprint and trained on the use of the five-step feedback sheet to give the students structured expert feedback. Sixty students signed an informed consent of participation after receiving an explanation of the study process and objectives from which they could withdraw at any time (Figure 1). They were instructed not to conduct additional training activities during the course of the study.
Pre-test evaluation
Practical skills training and measurements of pre-test evaluation
In the practical skills training, an emergency situation was simulated in which the students had to insert a periphery venous catheter. In the first exercise, it evaluated the correct use of gloves, the placement of a tourniquet, their knowledge of periphery-venous anatomy, preparing a sterile working surface, placing the catheter and the fixation of the catheter (Figure 2). The students performed the exercise on a phantom injection arm (Gaumard Scientific, USA). The second exercise involved the first aid treatment of a mandibular fracture using an interdental ligature technique (‘Ernst’ ligature). In this exercise, the correct identification of the fracture line, the placement of the ligature, cutting and twisting the endings of the wire and checking the stability of the ligature were evaluated (Figure 3). The students performed the exercise on a patient simulator (KaVo Dental GmbH, Biberach, Germany). Before taking part in the practical skills training, all of the students received instructions through a standardized teaching video for each skill. The videos included step-by-step instructions in real-time with comments, hints and frequent mistakes that should be avoided in the performance of the aforementioned skills. The process was based on the tutor’s manual and global rating scale.
Feedback Methods
After receiving the instructions, the students performed the skills by themselves for 30 minutes. Subsequently, the students were video recorded while performing each skill one last time as a performance measurement that was evaluated by the examiners (T0). This step was followed by the students individually receiving one of the feedback methods investigated in this study (T1). The time of the execution of the practical skills was also documented at T0 and T1.
Direct expert feedback (Control group)
In this group, the students were supervised by the instructor while performing the skills. During the 30 minutes of practice, the instructor observed each student one at a time performing the task at least once. This was followed by giving each student individual feedback using a five-step feedback sheet. The five steps in the feedback protocol assessed what went well, what could be improved, what went badly, what was missing, and what the take-home message was for each student. Immediately after the feedback, the students practiced again for 30 minutes before repeating the exercise while again being video recorded for the subsequent assessment (T1).
Individualized video feedback
In this group, feedback was given by the instructor using the same five step-feedback sheet after watching each student performance. The feedback sessions lasted for 30 minutes. Immediately after the feedback, the students practiced again for 30 minutes before repeating the exercise while again being video recorded for a later assessment (T1).
Unsupervised video feedback
As feedback, the students received once again the standardized video instructions and they were instructed to give themselves feedback using the same five step-feedback sheets. The feedback sessions were performed individually by each student and lasted for 30 minutes. Immediately after the feedback, the students practiced again for 30 minutes before repeating the exercise while again being video recorded for later assessment (T1).
Long-term retention
To measure long-term retention, an objective structured clinical examination (OSCE) format focused on OMS (OMS-OSCE) took place 6 weeks later (T2). 1514 During this time interval, the students did not perform any further exercises or receive any feedback. A regular OSCE is composed of eight 5-minute stations, with four of them verifying theoretical skills and four of them assessing practical skills. The practical stations assessed the task ‘’catheterization of a periphery venous catheter’’ and the second station evaluated the task ‘’interdental ligature’’, as described above. Again, the students were video recorded for later assessment (T2).
Performance measurement
The evaluation of their performance was done using a previously validated global rating scale (GRS; Figure 2 and 3).15 This consists of a trinary scoring scale (0 points for not done, 1 point for done but incorrect, and 2 points for done and correct) based on the checklist used in the tutor’s manual.1514By adding the aforementioned points, an average performance score was obtained. The global rating scales implemented were primarily piloted in previous undergraduate trainings and afterwards validated by two independent, blinded examiners. In addition, the content validity was ensured as part of an expert workshop with didactic and surgical experts as well as through its repeated application and adaption in the context of the previous studies 16–18 and OSCE exams. For the present study, two examiners received an educational course as calibration and to gain experience using the GRS. The inter-rater reliability was measured using Cohen’s kappa coefficient (κ = 0.84). The performance of the acquired competences in relation to both skills of the study was measured during the practical skills training (T0), directly after the intervention (T1) and 6 weeks later (T2). The examiners rated the student performance using video recordings of the student’s performance at each point in time (Camera System: Panasonic HC-X929, Osaka, Japan). All examiners had the opportunity to examine the videos only once and they were blinded toward the students’ instructional approach and their study group.
Statistical analysis
Microsoft Office 2016 (Microsoft Office 2007, © Microsoft Corporation, Redmond, USA) for Mac and SPSS Statistics version 19 (IBM, Armonk, USA) were used for the statistical analysis. The data collected was tested for normal Gaussian distribution using the Shapiro-Wilk normality test. The data from their performance was analyzed using two-way analysis of variance (ANOVA) with a Tukey multiple comparisons test done for all pairs. Time was analyzed using an unpaired two-tailed t-test (α = 0.05, 95% CI of diff.). Cohen’s d was used as an additional control test to support the interpretation of the data. A larger absolute value indicates a stronger effect. The results have been presented as the mean and standard deviation (SD), depicted in tables. Statistical significance was considered if p<0.05.
Sample size estimation
Based on the prior examination results from the years before the intervention and our null hypothesis that alternative feedback methods (IVF and UVF) are not inferior (no-inferiority study) when providing effective feedback compared to traditional methods (DEF), we estimated an average student performance of 70% with a standard deviation of 10% in the OSCE. With an average student number of 65 per semester, a sample size of 56 was calculated based on the following parameters: average student performance = 70%, alpha = 0.05, beta = 0.2 and power = 0.8.