Study Design and Setting
This study was designed as a single-blinded, parallel-group, randomised controlled trial with a 1:1 allocation ratio testing for non-inferiority video-delivered versus in-person feedback for improvement in medical student suturing skill acquisition. This manuscript is reported as per the noninferiority and equivalence randomized trials extension of the Consolidated Standards of Reporting Trials statement.(Piaggio et al., 2012) The protocol was not registered with a trial registry prior to commencement of enrollment.
Outcomes
Our primary outcome was improvement in suturing ability between pre- and post-feedback performance outcomes as measured by the University of Bergen suturing skills Assessment Tool (UBAT) score.(Sundhagen et al., 2018) A post-hoc analysis using a modified Objective Structured Assessment of Technical Skill (OSATS) score (Faulkner et al., 1996; Martin et al., 1997) was also used. See Appendix A for a summary of these tools. Secondary outcomes included student and instructor perceptions on feasibility and usefulness of remote smartphone-based feedback versus standard feedback methods assessed with questionnaires delivered to participants and instructors online using Google Forms (Alphabet Inc; California, USA), see Appendix B.
Participants and Randomization
Participants were recruited from first- and second-year medical students at McMaster University using a combination of email messages and social media requests. This population is consistent with that initially used in the development and validation of the UBAT score.
Informed consent was obtained from all participants prior to enrollment and a waiver of research ethics board review was granted by the Hamilton Integrated Research Ethics Board.
As this study was conducted during a COVID-19 lockdown period in Ontario, participants were initially randomized from September 2020 until November 2020 to either remote live feedback or remote recorded feedback arms using random block sizes of 2 to 4 with allocation concealment achieved through virtual pre-prepared sealed envelopes. Participants and feedback providers were blinded to their randomization until after submitting the pre-recorded feedback video. Once COVID restrictions allowed for resumption of in-person research, approximately during February 2021, additional participants were recruited for the control arm. Participant testing was performed asynchronously. See Figure 1 for diagrammatic overview of experiment. All participants completed a baseline survey prior to testing.
Video recording and feedback
Each participant was provided with a study kit including a suture pad, needle driver, forceps, 8 sutures (Artagia Inc; Toronto, Canada), and a smartphone recording mount (Neewer; Shenzhen, China; NW-35C). Participants were given 48 hours to complete a baseline questionnaire and suturing training module including instructional videos(Forte, 2018; Weinberg et al., 2014) as well as a description of a standardized suturing task (see Appendix A). They were also given instructions to set up their workspace for recording (Figure 2). In brief, the smartphone mount was used to hold the participant’s smartphone at a suitable height to record hand movements, tissue and instrument handling, and knot formation. In the event that a participant did not have a smartphone, or one with a suitable camera for the application, one was provided (Samsung Electronics Co., Ltd; Suwon, South Korea; A3 2020 Model).
Participants in the first round of recruitment (both video-delivered study arms) recorded themselves performing the standardized suturing task, then uploaded their recording into a Google Drive folder to be viewed immediately by their feedback provider (FP) for feedback. Once the FP was ready to provide feedback, participants’ randomization was revealed to both the FP and participant.
In the remote-live feedback arm, a Zoom (Zoom Video Communications; California, USA) link was sent to both participant and the FP to engage in a 15-minute live feedback session. Video capture was setup to allow both the participant and the FP to see each other’s hands and instruments in real-time. In the remote-recorded feedback arm, the FP was instructed to create a feedback video explaining how the participant can improve using the same recording setup. This feedback video was uploaded and provided to the participant within two hours. After feedback, participants had 48 hours to practice, record, and upload another performance of the standardized suturing task.
Participants in the control arm (recruited asynchronously once in-person educational activities were allowed in our jurisdiction) were provided with the same study materials and given 48 hours prior to receiving feedback to learn the standardized suturing task based on the training module, recording their pre-feedback performance prior to arrival. Participants again performed the standardized suturing task with a FP observing in the same room. FPs were then given 15 minutes to provide in-person feedback to each participant. This amount of training, but not individual feedback, for our control arm is similar to that given in the development and validation of the UBAT score(Sundhagen et al., 2018).
After feedback, participants in all study arms then had 48 hours to practice, record, and upload another performance of the standardized suturing task.
Feedback Providers
Twelve resident physicians in a surgical specialty assumed the role of feedback provider. FPs were instructed to provide feedback in the manner they usually would in a typical clinical or lab setting without score-specific training.
Suturing Performance Assessment
All suturing performance videos, both pre- and post-feedback, were assessed and scored by 2 surgical resident feedback providers using the UBAT and, in our post-hoc analysis, by 2 separate final-year surgical residents using the modified OSATS. Assessors were blinded to both the experimental arm and to whether the video was recorded pre- or post-feedback.
Experiential survey
Following completion of post-feedback videos, participants in all study groups answered a questionnaire exploring their experience with using the video-recording equipment, video-delivered feedback (where applicable), and their opinions on the suitability and feasibility of video-delivered feedback. Similar questionnaires were distributed to feedback providers and video assessors. See Appendix B for details pertaining to the survey.
Data Analysis
Analysis was performed by a blinded statistician using IBM SPSS V.25 (Statistical Product and Service Solutions, Chicago, Illinois, USA) and Microsoft Excel Analysis Toolpak (Microsoft Corporation, Redmont, Washington, USA). Baseline differences in demographics and stated interest in surgery for experimental and control arms were compared using descriptive statistics. Categorical and ordinal data were reported using calculated frequencies. Inter-rater reliability was assessed using Fleiss’ Kappa. Change in UBAT and OSATS scores were presented as numeric values with 95% confidence intervals (CI).
The expected improvement in the in-person control group was estimated at 4 points on the UBAT score, based on previous rates in the literature.(Almeland et al., 2020; Sundhagen et al., 2018) Non-inferiority was evaluated as described by Rothman(Rothmann & Tsou, 2003) by testing whether the 95% confidence interval for lower limit of the 95% CI for the video-delivered arms’ within-group improvement in UBAT and OSATS score excluded a 10%the control arm’s within-group improvement minus an inferiority margin between experimental and control groups.. An inferiority margin of 10 percentage points was chosen in consultation with local surgical experts and considered acceptable, given the generally low effect of feedback(Emmanuel et al., 2021) and a lack of standard guidelines for educational interventions as opposed to medical therapies for which regulatory bodies provide guidance.(Guidance for Industry Non-Inferiority Clinical Trials, 2010; Schumi & Wittes, 2011) To reach 80% power at a significance level of 0.025 for one-sided testing, 18 students per group were required.(Julious, 2004)