Intentional enterotomies: validation of a novel robotic surgery training exercise

While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick’s validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13 h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4 min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.


Introduction
Since the initial FDA approval for its use in 2000, robotassisted surgery has increased in prevalence, particularly in the field of general surgery [1]. A majority of the over one million robotic surgeries performed in 2019 were general surgery procedures [2][3][4]. While many surgeons cite the platform's visual and ergonomic advantages to laparoscopy, high-profile complications have led to increased public scrutiny regarding proper use of the technology [5,6]. A particular concern is the extent to which robotic surgeons are trained in the platform's use. In contrast to laparoscopy, where US general surgeons must complete didactic and skills-based curricula and testing for board certification, no such requirements exist for robotic-assisted surgery.
To address these deficiencies, academic surgery programs along with the groups behind Fundamentals of Robotic Surgery, Robotic Training Network and Fundamental Skills of Robotic Surgery have developed and implemented formalized robotic training programs that incorporate online training, console simulation, and live operating experience both at the beside and robot console [7][8][9][10][11][12][13][14][15][16]. However, variations in content and application are broad and there remains no consensus for robotic training standards.
Simulation such as virtual reality trainers, animal, cadaveric and inanimate tissue platforms plays a critical role in existing training curricula as it provides training in a lowstakes environment while allowing the trainee to improve their psychomotor and basic procedural skills. It remains difficult to discern the optimal training platform. Virtual reality trainers, for example, can cost up to $158,000 and are known to poorly emulate electrocautery [17][18][19]. Animal and cadaveric platforms also can be prohibitively expensive and can present ethical challenges for some programs/ surgeons. Inanimate training exercises bridge this gap by utilizing the already procured da Vinci system to provide high-fidelity training in a cost-effective manner, particularly those that use biotissue models versus traditional box trainers. A persistent need exists for inanimate tissue model that incorporates electrocautery for standardized robotic training curricula. Currently described cautery simulation platforms are expensive and cheaper high-fidelity cautery simulations are needed [19].
We developed such a model using a double-layered bowel model secured to a moistened household sponge that simulates skills such as tissue handling, camera control, suturing, and electrocautery. We then set out to establish the content, response process, internal structure and construct validity of the exercise using Messick's validity framework [20].

Methods
This research was approved by the 59th Medical Wing, Joint Base San Antonio and University of Texas Health at San Antonio Institutional Review Boards and determined to be exempt. This was a multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3). Participants were recruited via email and provided with informed consent. They were then separated into two cohorts (MS + PGY1 vs. PGY2/3). An inanimate tissue model was created using a doublelayered hydrogel bowel model produced by Lifelike BioTissue Inc. (London, Ontario, Canada) and assembled by research staff (Fig. 1). The base of the bowel model was sutured to a heavy-duty cleaning sponge and then marked with a longitudinal incision line measuring 5 cm with five pairs of dots evenly spaced 1 cm from each other and 5 mm from the incision line to mark locations for the suture needle to enter and exit the bowel. This was then secured to an electrocautery Bovie pad within a robotic abdominal dome trainer [21]. Participants were then asked to create a longitudinal enterotomy using electrocautery with monopolar scissors, followed by closure with interrupted suture using the dots as landmarks on where to enter and exit the model.
Participants first completed a pre-study survey on which they self-reported select demographic information and surgical experience. Prior to performing the exercise, participants were provided with a narrated video of one of the authors (R.L.) performing the exercise, which consisted of creating an enterotomy in the bowel model using the da Vinci Xi robot (Intuitive Surgical Inc., Sunnyvale, CA), followed by an interrupted suture closure along marked, dotted lines.
Participant performance was recorded and scored in real time by one of the research staff using a standardized scoring rubric based on errors committed. The video recording was then reviewed by a separate researcher to ensure inter-rater reliability for the error calculations. Errors were defined as the number of times the following occurred: instrument out of view of the camera, targets missed, torn suture, instrument collision and air knots tied. These errors were then tabulated together for a summative total error score. The amount of suture required to complete each exercise and time to completion-defined as initiation of enterotomy to the cutting of the last stitch-were also annotated.
Simulation recordings were then assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) rubric, a standardized clinical assessment tool for robotic surgical skills assessment [22,23]. GEARS assesses six separate domains: depth perception, bimanual dexterity, efficiency, autonomy, force sensitivity, and robotic control. Each domain is scored using a 5-point Likert-like scale with specific performance anchors at 1, 3 and 5. The GEARS score is the sum of each of these domains. In this analysis, autonomy was not assessed as participants performed the exercise by themselves in its entirety. Each video was evaluated by at least 30 reviewers using the Crowd-Sourced Assessment of Technical Skills (C-SATS ™ , Seattle, WA) platform [24][25][26][27]. Lastly, the participants were asked to fill out a survey evaluating their self-rated robotic surgical skill and confidence before and after the exercise along with its overall educational benefit, realism, and potential for skill transfer (Table 1). Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30, while each C-SATS evaluation cost ~ $200 per video.
Messick's validity framework was used to evaluate the model for content, response process, internal structure and construct validity [20,28]. Content validity is defined as the extent to which a measurement addresses all skills necessary for a particular domain of content [29]. Response process evidence refers to data integrity and how closely assessment scores reflect the observed performance of the trainee. Internal structure validity corresponds to the reliability and reproducibility of simulator scores between raters; this can be shown by establishing inter-rater reliability [30]. Finally, a simulation contains construct validity if it is able to differentiate between participants of different skill sets [31].
Differences between the outcomes among the two groups were identified by two-tailed unpaired T tests. Statistical analysis was performed using SPSS Statistical Software version 24 (IBM, Armonk, NY). Statistical significance was defined at a p < 0.05.

Results
Thirty-one subjects were recruited to participate in the inanimate tissue model exercise. The subjects ranged from medical students to general surgery residents (PGY1-3) at two different institutions. Exposure to and experience with minimally invasive surgical platforms varied significantly between the two cohorts ( Table 2). We noted statistically significant differences between the two when comparing the hours spent on the da Vinci Skills Simulator (dVSS) trainer (p = 0.003), number of bedside assists performed (p < 0.001), number of laparoscopic cases performed as surgeon junior (p < 0.001). and number of robotic cases performed as surgeon junior (p < 0.001).
Performance metrics from each cohort were aggregated together and compared. The PGY2/3 group completed the exercise faster (14.4 vs. 26.1 min, p < 0.001; Fig. 2a) and committed fewer total errors (11.9 vs. 21.5 errors, p = 0.018; Fig. 2b). The types of errors committed by each group is shown in Table 3. Inter-rater reliability of total error was established with Cronbach's alpha of 0.998.
The PGY2/3 group also received higher GEARS score (19.9 vs 18.5, p = 0.001; Fig. 2c). Components of the GEARS scores are shown in Table 4. Inter-rater reliability was established among the crowd-sourced assessors (~ 30-40 for each video) with Cronbach's alpha of 0.717. All participants were able to successfully create the enterotomy with electrocautery as described. The electrocautery had a melting effect on the LifeLike Biotissue double-layered hydrogel bowel model akin to using a monopolar energy device on the 'Cut' setting. The thermal spread was minimal and was limited to tissue contacted by the instrument directly. Before completing the exercise, how would you have rated your robotic surgical skill? 2 Before completing the exercise, how confident were you in performing robotic surgery? 3 Following the exercise, how would you rate your robotic surgical ability? 4 Following the exercise, how confident do you feel in your robotic surgical skill? 5 Overall, how would you rate the educational benefit of the exercise? 6 How effective do you think the inanimate training exercise was in training robotic skill? 7 How realistic was training on the inanimate model (i.e., appearance and tissue characteristics) in comparison to the operating room? 8 Training on the inanimate exercise taught useful skills that were transferable to the operating room The post-activity survey was completed by 23 of 31 participants. Based upon survey responses, 87.0% and 91.3% of participants reported improvement in robotic surgical ability and confidence, respectively. Overall, respondents rated the realism of the exercise 7.5/10, education benefits as 9.1/10 and effectiveness in teaching basic robotic skills 8.7/10 (Fig. 3). Furthermore, participants felt that the skills they learned on the inanimate model would transfer to the operating room.

Discussion
This prospective, multi-institutional study established the content, response process, internal structure and construct validity of a novel inanimate tissue model that incorporates electrocautery using Messick's validity framework [20,28]. The content validity was established through the post-exercise survey which demonstrated that participants found the model to contain a high degree of realism comparable to the operating room. Response process validity was achieved by using blinded raters to provide GEARs ratings for video performances. Internal structure validity was established by using GEARS ratings from crowdsourced assessors who have a published inter-rater reliability (IRR) [32]. Finally, construct validity was shown through the significant differences in objective metrics-GEARS scores, time to completion, and total errors committed-between the two cohorts of participants.
Medical simulation enables procedural and psychomotor skill acquisition in a low-stakes environment. Certain simulation platforms, however, contain barriers and limitations of use. While virtual reality trainers have robust validation data supporting their use, costs can be prohibitive. For example, the dVSS system costs $90,00 in addition to the investment needed for the da Vinci console (~ $500,000) for the trainer interface, while dV-Trainer costs range up to $158,000 [19]. Trainers like the dVSS system that also require concurrent console usage limiting training availability to time outside normal business hours, which can be problematic given residency work hour restrictions [33]. These systems are also known to poorly emulate electrocautery [18]. Similarly, box trainer exercises like the one developed and validated by the Fundamentals of Robotic Surgery Skills Curriculum group lack realism and tissue fidelity [14]. Other utilized forms of simulation training include live animal and cadaver laboratories, but concerns surrounding ethics, cost, and accessibility persist [34]. Fig. 2 a Average time to completion for each group during the exercise (minutes). Timer started at the initiation of the enterotomy and ended once the last suture was cut. p value obtained using two-tailed unpaired T test. b Average total number of errors committed by each group during the exercise. p value obtained using two-tailed unpaired T test. (c) Average GEARS scores for each of the groups. p value obtained using two-tailed unpaired T test Inanimate tissue models serve as a cost-effective and more accessible means to provide task-specific simulation training [35,36]. Tam et al. argue that virtual reality trainers are best for gaining familiarity to the platform, while biotissue tissue exercises like the one presented teach critical skills such as tissue handling, suturing, camera control, and electrocautery [37]. They showed face and content validity for an inanimate tissue model using material from LifeLike Biotissue Inc. to train surgical oncology fellows for robotic pancreaticoduodenectomy.
Examples of similar models in the literature are scarce due to the aforementioned barriers. Marecik et al. published a similar robotic suturing exercise using porcine intestine, but the need to harvest and freeze the bowel tissue raises ethical and tissue integrity concerns [38]. Other inanimate robotic training exercises are described in the literature such as a vaginal cuff model constructed from "beer huggies" and drills designed to replicate the Fundamentals of Laparoscopic Surgery curriculum [23,39]. This double-layered bowel model provides a more cost-effective and accessible means to conferring similar surgical skills. Single iteration use, after initial investment for reusable training materials, costs as little as $30 [40].
Limitations of this study include the small, yet significant, difference in GEARS scores among the two groups, the clinical significance of which remains to be seen. Work is ongoing to establish the construct validity of this scoring system in commonly performed robotic general surgery procedures, after which time we will be able to comment more definitively on the matter.
Next steps will involve incorporating this model and the assessment tools in a proficiency-based curriculum and utilizing GEARS scores as a metric for junior resident robotic skill evaluation. A proficiency-based curriculum utilizes expert derived performance goals as training endpoints [41]. These types of curricula would emphasize deliberate practice to a goal-directed learning experience rather than an arbitrary number of repetitions to ensure uniform skill development among trainees, which have been shown to more reliably confer higher levels of skill [42].

Conclusion
Using Messick's validity framework, this study established the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. This may also provide a means to determine trainee proficiency through time and error-based as well as crowd-sourced analysis. Consideration should be given to its addition to robotic surgery training curricula.