Comparing the Efficacy of Endovascular Simulation Training in different Training Formats: a randomized controlled trial

doi:10.21203/rs.3.rs-3771987/v1

Download PDF

Research Article

Comparing the Efficacy of Endovascular Simulation Training in different Training Formats: a randomized controlled trial

https://doi.org/10.21203/rs.3.rs-3771987/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 30 Jun, 2024

Read the published version in Annals of Vascular Surgery →

Version 1

posted

You are reading this latest preprint version

Background

High-fidelity endovascular simulators have demonstrated significant potential in enhancing interventionalists’ performance metrics, including procedure time, fluoroscopy time, and contrast volume usage. However, it is less clear whether trainee performance, when assessed using both multidimensional global and procedure-specific rating scales, improves in a manner consistent with these metric parameters. This study aimed to determine whether the structure of the training – conducted in block sessions or weekly intervals – impacts the effectiveness of the training.

Methods

20 students were enrolled in this prospective, randomized, controlled, single-center trial. They were randomly divided into block and weekly training groups, each undergoing 6 hours of supervised endovascular training on a Mentice simulator over a span of 3 weeks. At the outset and conclusion of their training, students performed a predefined peripheral endovascular intervention, which was assessed by an experienced interventionalist. Furthermore, participants were prompted to complete a survey at both the beginning and the end of the study.

Results

Both groups exhibited significant improvements in procedure time, fluoroscopy time, contrast volume usage, global rating scale scores, and performance-specific rating scale scores. However, no significant difference was observed between the two groups. The simulator training notably boosted the trainee’s self-confidence.

Conclusion

Simulator training not only enhances metric performance parameters, but also improves performance as assessed by both global rating scale and procedure-specific rating scale. Whether the training is conducted in block sessions or weekly intervals did not affect its efficacy. As such, the training format can be tailored to best fit the specific circumstances and setting.

Endovascular

Simulator

Simulation

Training

Rating Scale

RCT

Endovascular procedures have become an integral part of not only modern vascular surgery, cardiology, and interventional radiology but also of other disciplines, such as neurosurgery. The adoption is largely due to its minimally invasive nature and associated low mortality rate [1–3]. In this context, angiography simulators have become an increasingly vital tool for training budding doctors outside the patient environment. These simulators permit repeated practice, enabling trainees to learn from their errors without compromising patient safety [4–6]. Numerous studies, including a systematic review by See et al., reported not only significant improvements in metrics like procedure time, fluoroscopy time, and contrast volume usage but also in the trainees’s self-confidence levels [7–9]. Incorporating high-fidelity angiography simulators into surgical training, given their potential to boost an interventionalist’s performance, could possibly diminish perioperative risks for both patients and the medical staff. A lingering uncertainty, however, revolves around whether the training format – being in block sessions or weekly intervals – affects the efficacy of the practice. While many studies emphasize metric outcomes, such as procedure time (PT), contrast volume (CV) usage, and total fluoroscopy time (FT), a limited number delve into skill-level performance via global rating scales (GRS) or procedure-specific rating scale (PSRS) [10, 11]. The aim of this study was to discern if GRS and PSRS mirror improvements observed in the metric parameters and to determine potential disparities in learning progression and overall performance between the block training (BT) and weekly training (WT) cohorts.

Study design

Upon approval of the study design by the local ethics committee (ECS 1039/2021), 20 medical students from the Medical University of Innsbruck, Austria, in their sixth semester and beyond, agreed to participate in this prospective, randomized, controlled, single-center trial. Informed consent was obtained from all the participants. They received an initial didactic instruction which detailed the study objectives and introduced the foundational principles of endovascular intervention for patients with peripheral artery disease (PAD).

The participants were then randomized into either BT or WT formats, collaborating in pairs. Each pair underwent six hours of supervised training on an angiography simulator. The block format group completed this in two consecutive days with three-hour sessions each day, while the weekly format group attended two-hour sessions over three consecutive weeks giving a total of 6 hours training to each participant. Following a tutorial on the simulator’s correct usage and the standardized procedural sequence, participants executed an antegrade peripheral endovascular intervention, involving stenting of a short lesion in the superficial femoral artery (SFA).

Outcomes were divided into two categories: performance metrics recorded by the simulator (e.g., total PT, CV usage, and total FT) and a subjective skill assessment by an experienced endovascular surgeon who was blinded to the type of training the participants had performed. This subjective assessment featured two separate scoring systems: The Procedure Specific Rating Scale (PSRS), which outlined the exact procedural steps for a peripheral endovascular intervention as seen in Fig. 1 and instructed at the study’s onset, with a pinnacle score of fourteen points. And the Modified Reznick Scale (MRS), a Global Rating Scale (GRS) originally conceptualized by Reznik [12] for surgical performance, was adapted by Hislop et al. [9] for endovascular interventions (Fig. 2). It facilitates a multifaceted performance evaluation. Participants were rated in the four categories ‘Respect for Tissue’, ‘Time and Motion’, ‘Instrument Handling’, ‘Flow of Operation’, as well as overall performance and endproduct, with scoring ranging from one to five for each of those aspects. Consequently, a perfect score represents 30 points, symbolizing exemplary skill demonstration.

This assessment was performed before the start of the training as well as after completion of the training by an observer, being blinded to the group allocation.

Surveys

Participants were asked to fill out both entrance and exit surveys to collected data on demographics, musical or gaming backgrounds, interest in vascular surgery or interventional radiology, self-assessment of manual dexterity, prior endovascular experience and their comfort level in performing an endovascular procedure on a patient, even if supervised. Responses to both surveys were collected online via the REDCap electronic data capture tools (Vanderbilt University, Nashville Tennessee).

Endovascular Simulator

The study utilized the Mentice VIST G5 simulator (Mentice AB, Göteborg, Sweden) as seen in Fig. 3, a high-fidelity endovascular simulator. Designed to replicate modern catheterization laboratories, it features two monitors, a control box for fluoroscopy and table adjustments, foot pedals for managing fluoroscopy and cine loops, a contrast volume syringe, a stent release handle, and a pressure inflation device for balloon dilation (Fig. 1). This simulator allows for the use of actual catheters and provides enhanced haptic feedback, delivering a strikingly authentic procedural experience. Furthermore, its software presents a range of peripheral ilio-femoral endovascular intervention scenarios.

Data management and statistics

All research data were curated and overseen using the Research Electronic Data Capture (REDCap) tools (Vanderbilt University, Nashville Tennessee) provided by the Medical University of Innsbruck [13, 14].

For statistical assessment, SPSS 29.0 for Windows (SPSS, Chicago, USA) was used. A p-value of < 0.05 was defined as statistically significant, with all disclosed p-values being two-sided. The Kolmogorov-Smirnov test served as a ‘goodness of fit’ measure to evaluate normal distribution. When data displayed a normal distribution, the two-sample t-test was executed to identify statistical differences. For data not following a normal distribution, the Mann-Whitney U test was employed for independent samples, and the Wilcoxon signed-rank test for dependent ones. The Pearson correlation coefficient evaluated linear correlations between non-normally distributed data sets. In contrast, the Spearman’s rank correlation coefficient assessed monotonic relationships in normally distributed data. Categorical data, expressed as percentages, were compared using the chi-square test for independent samples and the McNemar test for dependent ones.

All participants completed the study and were therefore included in the analysis. Both BT and WT groups exhibited equal gender distributions post-randomization stratified by gender (80% male vs. 20% female). One participant within the WT group (5%) had prior experience with an angiography simulator. Both groups had similar study levels (BT: 8.0 semester, WT: 8.6 semester), with mean ages of 23.3 in BT and 24.1 in WT. In the first survey, only one student from the BT group (5%) felt confident enough to perform an endovascular intervention on a patient under supervision.

At the first supervised intervention, no significant differences between the groups were found and both groups (BT and WT) demonstrated significant improvement across all parameters (PT, FT, CV, MRS and PSRS) at the final intervention with details being described in Table 1 and Fig. 4 (a and b).

Table 1 Showing the results of the performance metrics (PT and FT given in minutes, CV in milliliters) as well as the achieved PSRS and MRS scores for both groups (BT and WT) at the first (= before training) and final (= after training) supervised intervention, the standard deviation added in parentheses. The p-values comparing first and final intervention are also stated (* = < 0.05; ** = < 0.01; *** = <0.001).

	BT			WT
	before training	after training	p-value	before training	after training	p-value
PT (min)	21.48 (± 5.18)	7.75 (± 2.17)	< 0.001***	22.08 (± 3.92)	9.15 (± 2.21)	< 0.001***
FT (min)	10.40 (± 4.51)	3.37 (± 0.84)	< 0.001***	10.32 (± 4.80)	3.53 (± 0.91)	< 0.001***
CV (ml)	49.94 (± 23.27)	33.08 (± 11.10)	0.047*	41.47 (± 14.41)	25.74 (± 7.12)	0.005**
PSRS	11.90 (± 1.37)	13.20 (± 0.79)	0.023*	10.80 (± 1.03)	13.50 (± 0.71)	0.004**
MRS	16.00 (± 2.98)	23.70 (± 3.65)	0.007**	14.20 (± 2.39)	24.10 (± 3.90)	0.005**

When comparing the rate of improvement between groups, no significant difference was observed, except in the PSRS scores. The WT group displayed superior progress in the PSRS due to a lower initial score, although the final PSRS averages between the two groups did not differ significantly (Table 2, Fig. 4 (c and d)).

Table 2 Comparing the improvements of the performance metrics (PT and FT given in minutes, CV in milliliters) as well as of the achieved PSRS and MRS scores for both groups (BT and WT) between first and final supervised intervention, the standard deviation added in parentheses. The p-values comparing the rates of improvement between the groups (BT and WT) are also stated (* = < 0.05; ** = < 0.01; *** = <0.001).

	Comparison of Improvements
	Improvement BT	Improvement WT	p-value
PT (min)	13.73 (± 4.66)	12.93 (± 3.36)	0.666
FT (min)	7.03 (± 4.22)	6.79 (± 4.70)	0.907
CV (ml)	16.86 (± 23.87)	15.73 (± 11.22)	0.894
PSRS	1.30 (± 1.34)	2.70 (± 0.82)	0.014*
MRS	7.70 (± 4.79)	9.90 (± 5.04)	0.33

While at the beginning of the training, only 5% of the probands felt confident to perform and intervention on a patient under supervision, this number rose to 85% overall (100% of the BT group and 70% of the WT group) by the conclusion of the training. This improvement of self-confidence was statistically significant in both groups (BT: p = 0.004; WT: p < 0.001, overall: p < 0.001).

When asked about the adequacy of the overall training duration (6 hours for each group), 80% of the BT group felt it insufficient, expressing a desire for extended training. In contrast, significantly fewer, namely only 40%, in the WT felt similarly.

A noticeable trend emerged with students identifying themselves in the survey as having superior motor skills. They generally outperformed their peers in initial interventions concerning PT (p = 0.228) and FT (p = 0.056), although this observation was not statistically significant.

The potential of simulator training in endovascular surgery to improve outcome parameters such as PT, FT and administered CV is well established [8]. However, whether there is a difference in efficacy between block training sessions and weekly ones remains unclear. Block sessions, commonly held during conferences or symposia, are more popular due to their convenience. However, the relevance of weekly sessions rises when considering long-term training for budding surgeons or interventional radiologists.

Our findings validate prior studies: simulator training effectively optimizes metric outcomes such as PT, FT, and CV. All participants demonstrated marked improvements at performed assessments.

Most past studies had a limitation noted by See et al.: they often overlooked the comprehensive performance evaluation through tools like the Global Rating Scale (GRS) and Procedure Specific Rating Scale (PSRS), focusing mainly on standard performance metrics. We incorporated the Modified Reznick Scale as a GRS and devised a PSRS for a specific endovascular procedure. Ensuring impartiality, different proctors handled the initial and final assessment with the latter being unaware of the student’s training history. Like the performance metrics, GRS and PSRS too exhibited significant enhancements, suggesting that simulator training refines holistic performance, from operation flow to instrument handling.

When juxtaposing BT and WT groups, the data did not show any notable disparity in terms of improvement between first and final assessment. This implies that if total training hours remain consistent, both training formats yield comparable results.

Numerous studies assessing motor skill acquisition and transfer of blocked and random or interleaved training schedules have shown, that the initial learning curve is equal in both groups, if not steeper in the BT group, but that interleaved training might improve retention and transfer, potentially due to better conveyance to long term memory, known as the contextual interference effect. ^15–18 Therefore, a follow-up study would be of interest, to evaluate the long-term learning effect of the simulator training in the BT and WT groups.

Interestingly, there was a significant difference in the subjective perception of sufficiency of the training. A large majority (80%) of the BT probands stated that they wished more training time. In the WT group this perception was with only 40% of probands significantly lower. Even though the total training time with 6 hours was the same for each group, it appears that a BT format is perceived as less extensive, albeit the objective improvement was the same as with the WT format.

Another remarkable observation is the boost in trainees’ self-confidence. Echoing findings from studies like Torres et al. [7], our research indicated that post-training, trainees felt more equipped to undertake an intervention on patients. Self-confidence, often a gauge for training success, not only heightens training satisfaction but also boosts the comfort level in seeking help in every-day situations [19, 20]. Clanton et al. further emphasized this by showcasing a direct correlation between confidence and competency post training, assessing the levels of confidence and competence in their surgical skills training research of 150 medical students over a 2-year workshop [21].

Simulator training serves as a valuable instrument in endovascular surgical training. It enhances not only tangible metrics like PT, FT and CV usage but also subjective performance evaluations such as GRS and PSRS. Furthermore, it boosts trainee’s self-confidence. Our findings indicate that the efficacy of block and weekly training formats is comparable. This flexibility allows instructors to tailor their training approach based on their specific environment and circumstances.

Funding and Competing Interests

The authors have no competing interests to declare that are relevant to the content of this article.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Compliance with Ethical Standards

There are no disclosures of potential conflicts of interest.
Informed consent was obtained of the study participants before their enrollment.

Ethics approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Ethics Committee of the Medical University Innsbruck.

Trial registration number: ECS 1039/2021

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Author Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by David Wippel, Werner Westreicher, Fabian Lamprecht, Michaela Kluckner, Maria Gummerer and Miar Ouaret. The first draft of the manuscript was written by David Wippel and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Acknowledgments

We kindly thank Professor Elke Gizewski, head of the Department of Radiology of the Medical University Hospital Innsbruck, for providing the endovascular simulator and the facilities for the duration of the study. Also, we kindly thank Christoph Wippel, MD, Msc for carefully revising and editing the final draft of the manuscript.

Greenhalgh RM, Brown LC, Kwong GP, et al. Comparison of endovascular aneurysm repair with open repair in patients with abdominal aortic aneurysm (EVAR trial 1), 30-day operative mortality results: randomised controlled trial. Lancet. 2004;364(9437):843-8.
Nasr MK, McCarthy RJ, Hardman J, et al. The increasing role of percutaneous transluminal angioplasty in the primary management of critical limb ischaemia. Eur J Vasc Endovasc Surg. 2002;23(5):398-403.
Peschillo S, Caporlingua A, Colonnese C, et al. Brain AVMs: and endovascular, surgical, and radiosurgical update. ScientificWorldJournal. 2014;2014:834931.
Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241(2):364-72.
Moorthy K, Vincent C, Darzi A. Simulation based training. BMJ. 2005;330(7490):493-4.
Desender L, Rancic Z, Aggarwal R, et al. Patient-specific rehearsal prior to EVAR: a pilot study. Eur J Vasc Endovasc Surg. 2013;45(6):639-47.
Torres IO, De Luccia N. A simulator for training in endovascular aneurysm repair: The use of three dimensional printers. Eur J Vasc Endovasc Surg. 2017;54(2):247-53.
See KW, Chui KH, Chan WH, et al. Evidence for Endovascular Simulation Training: A Systematic Review. Eur J Vasc Endovasc Surg. 2016;51(3):441-51.
Hislop SJ, Hsu JH, Narins CR, et al. Simulator assessment of innate endovascular aptitude versus empirically correct performance. J Vasc Surg. 2006;43(1):47-55.
Chaer RA, Derubertis BG, Lin SC, et al. Simulation improves resident performance in catheter-based intervention: results of a randomized, controlled study. Ann Surg. 2006;244(3):343-52.
Hseino H, Nugent E, Lee MJ, et al. Skills transfer after proficiency-based simulation training in superficial femoral artery angioplasty. Simul Healthc. 2012;7(5):274-81.
Reznick RK. Teaching and testing technical skills. Am J Surg. 1993;165(3):358-61.
Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building and international community of software platform partners. J Biomed Inform. 2019;95:103208.
Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-81.
Fazeli, David & Hamidreza, Taheri et al. (2017). Random Versus Blocked Practice to Enhance Mental Representation in Golf Putting. Perceptual and Motor Skills. 124. 003151251770410. 10.1177/0031512517704106.
North, Jamie, Bezodis, Neil et al. (2017). The effect of consistent and varied follow-through practice schedules on learning a table tennis backhand.
Srem-Sai, Medina & Hagan Jnr, John & Baba, Jatong & Schack, Thomas. (2019). Assessment of Random and Blocked Practice Schedules on Motor Skills' Acquisition, Retention and Transfer Among Selected Senior High School Students. American Journal of Sports Science. 7. 26-33. 10.11648/j.ajss.20190701.15.
Chalavi, Sima & Pauwels, Lisa et al. (2018). The Neurochemical Basis of the Contextual Interference Effect. Neurobiology of Aging. 66. 10.1016/j.neurobiolaging.2018.02.014.
Esterl RM, Henzi DL, Cohn SM. Senior medical student "Boot Camp": can result in increased self-confidence before starting surgery internships. Curr Surg. 2006;63(4):264-8.
Geoffrion R, Lee T, Singer J. Validating a self-confidence scale for surgical trainees. J Obstet Gynaecol Can. 2013;35(4):355-61.
Clanton J, Gardner A, Cheung M, et al. The relationship between confidence and competence in the development of surgical skills. J Surg Educ. 2014;71(3):405-12.

No competing interests reported.

Download PDF

Journal Publication

published 30 Jun, 2024

Read the published version in Annals of Vascular Surgery →

Version 1

posted

You are reading this latest preprint version

Comparing the Efficacy of Endovascular Simulation Training in different Training Formats: a randomized controlled trial

Status:

Journal Publication

Version 1

Abstract

Background

Methods

Results

Conclusion

Figures

Background

Methods

Results

Discussion

Conclusion

Declarations

Funding and Competing Interests

Compliance with Ethical Standards

Ethics approval

Consent to participate

Author Contributions

Acknowledgments

References

Additional Declarations

Status:

Journal Publication

Version 1