Data were collected at two general hospitals. The IRB (Institutional review board) approved the study at the participant institutions. Written informed consent was obtained from participants before inclusion in the study. From October 2018 to April 2019, a convenience sample consisting of 30 gynecology residents with no previous experience in hysteroscopy and 25 experts participated in the study. Experts were gynecologic surgeons who had received formal training in hysteroscopic procedures and had been practicing diagnostic and operative hysteroscopy for more than three years at the time they were enrolled in the study.
The Simulator
The anthropomorphic anatomical models used in the simulator were manufactured in acrylonitrile butadiene styrene (ABS) on a fusion deposition modeling (FDM) 3-D printer (3-D Printer Prusa I3 Rework, ITCTERM, Brazil). The 3-D model was scanned from a 9 × 7 × 6 cm pyriform mold simulating a human uterus. The models contained two hemicavities internally lined with textured, rose-colored acetic silicone to mimic the appearance of the human endometrium. A 2-cm diameter opening was created to allow the atraumatic passage of the hysteroscope.
Three uterine models were embedded in latex foam inside an opaque 30 cm-wide, 28 cm-long, 18-cm high cardboard box. Three 3-cm circular apertures were made on the longest box wall. The three apertures were framed with ABS ring-shaped connectors to which red-colored male condoms were attached and connected to the opening of the uterine models inside the box simulating vaginas (Fig. 1). Each uterine model in the simulator was designed to develop one of the basic hysteroscopic skills.
The first model was designed to develop intra-uterine navigation skills. It was equipped with nine distinct adhesive images positioned on the anterior, posterior, left lateral, right lateral walls, on the uterine fundus, and on the anterior and posterior tubal ostia, represented by two red markings inside the model.
The second uterine model was designed to develop direct biopsy skills. Five electronic touch-sensors (Micro Switch, KW-1 3pins, Dongnan, China) covered by a non-slip adhesive tape were connected to a microcontroller board, Arduino Uno (Smarts Projects, Ivrea, Italy) which was connected to a microcomputer via a USB (Universal Serial Bus) port. The sensors were positioned through openings in the model's wall in the uterine fundus and on the anterior, posterior, left and right walls of the inner surface of the model. A visual sign on the computer screen indicated which sensor had been pressed at each attempt.
The third uterine model was designed for developing foreign body removal skills. The model was equipped with five 8-mm diameter apertures on the model wall through which 8-mm diameter, 4-cm long mini balloons were inserted five millimeters into the model cavity.
Face and content validity
Face and content validities were based on expert perceptions of the realism of tasks performed on the simulator compared to procedures performed on real patients. Face validity was evaluated by assessing the degree of expert concordance with the following statements: (1) the appearance of the simulator's internal cavity resembles that of the human uterine body cavity; and (2) the procedures performed on the simulator resemble the office-based hysteroscopy procedures performed on human patients. Responses were measured on four-point unidirectional, forced-response scales, where the extreme points 1 and 4 indicated full disagreement and full agreement, respectively. Also, the realism of the global experience in performing the tasks on the simulator was assessed on a 10-point rating scale (1 = absolutely non-realistic; 10 = absolutely realistic).
Content validity was based on experts’ agreement with statements addressing the similarity between the tasks performed on the simulator, the hysteroscopic maneuvers performed in real patients, for example hand-eye coordination, accurate visualization through a 30° angle scope and the use of grasping forceps. Also, the utility of learning experiences in the simulator for training novices was used to assess the content validity of the simulator. Responses were measured on four-point unidirectional scales where the extreme points 1 and 4 indicated full disagreement and full agreement, respectively.
Construct validity
The ability to discriminate between expert and resident performance determined the construct validity of the simulator. Participants' performance was measured on procedure-specific checklists used for rating performance on the video-recorded simulation sessions.
For developing the procedure-specific checklists (Appendix), three authors (A.R.P., L.B.G. and L.K.V.) independently searched the literature to create technical performance checklists addressing the main steps of the simulation tasks: hysteroscopic navigation, direct biopsy, and foreign body removal [12, 13]. Items in the checklists addressed ergonomics, image visualization, safe navigation and handling the grasping forceps [10]. Items of the checklists represented the elements or steps identified in the three tasks that could be measured and potentially differentiate amongst levels of technical competence. Three-point scales were added to the scoring rubrics, where zero indicated an unskilled performance, as expected from a novice; one indicated a somewhat skilled performance, as expected from an in-training novice; and two indicated a skilled performance, as expected from an experienced surgeon [14].
Simulation sessions
Simulation sessions took place in private rooms equipped with a video endoscopy equipment consisting of a high definition monitor (Karl Storz, Germany), a cold light source XENON 300 (Karl Storz, Germany) and a video-camera IMAGE 1 HUB™ (Karl Storz, Germany). Also available in the simulation room were HOPKINS® Forward-Oblique 30°, 2.9-mm Telescope (Karl Storz, Germany), BETTOCCHI® inner and outer sheaths (Karl Storz, Germany), and semirigid 5 French (Fr) grasping forceps (Karl Storz, Germany), which were used to complete the tasks in the simulator. The simulator box was positioned on a ninety-centimeter high table for the training sessions. Participants stood in front of the simulator during the experiment, having the video equipment positioned anteriorly and to their left, allowing a full view of the screen. The same investigator (A.R.P.) conducted all simulation sessions, which involved only one participant at a time. Formative feedback on participant performance was allowed [11].
Simulation sessions consisted of a preparatory and a hands-on phase. The preparatory phase aimed to provide the participant with relevant theoretical and practical guidance about the tasks to be simulated. The preparatory phase started soon after the arrival of the participant at the simulation room. The instructor started by providing a structured twenty-minute presentation addressing the definition, the indications, the equipment, the technique and the complications of office-based hysteroscopy. Next, participants were informed about the technical aspects of the tasks to be simulated, their metrics and goals. Participants were further instructed about how to handle the scope and the grasping forceps. The preparatory phase of the experiment finished after the instructor presented the simulator, demonstrated the tasks and allowed the participants to manipulate the simulator, the video and the hysteroscopy equipment for approximately five minutes before starting the hands-on phase of the simulation session.
Participants performed three exercises (tasks) during the hands-on phase of the simulation session. The navigation exercise (task 1) consisted of identifying, visualizing and centering the nine targets and simulated tubal ostia inside the first model followed by obtaining a panoramic view of the interior uterine model, before withdrawing the hysteroscope. The aim of the direct biopsy simulation exercise (task 2) was to press the sensors positioned inside model 2 with the tip of a grasping forceps. The foreign body removal simulation exercises (task 3) consisted of using grasping forceps to clamp and pull the mini balloons inserted on the wall of the third model at least 3 cm into the simulator cavity. Participants repeated each exercise five times. Task completion times were counted from the moment of insertion to the moment of withdrawal of the hysteroscope from the simulator. Participant performance was digitally recorded (Canon EOS 60D 18 MP, Canon U.S.A., Inc., Huntington, NY, U.S.A) for further analyses. Three authors (A.R.P., L.B.G. and L.K.V.) rated the videos independently [15].
At the end of the simulation session, participants answered the demographic data questionnaire. Experts also completed the eight-item face and content validity questionnaire.
Statistical analysis
Statistical analysis was performed utilizing SPSS v 26.0 (SPSS Inc., Chicago, IL, U.S.A). Basic descriptive statistics were calculated for demographic data. All continuous variables were presented as mean (standard deviation) or median (range), and nominal variables were presented as frequency (percentage, %). Kolmogorov-Smirnov test was used to assess normality. Generalized linear mixed model (GLM mixed-model) ANOVA was used to compare scores and task completion times within and between groups. Median scores of the items used to assess the face and content validities were tested against the mathematical center of the scales by using one-sample Wilcoxon signed-rank tests.
Psychometric analyses of the technical performance checklists included exploratory factor analyses by the principal-component extraction method and varimax rotation with Kaiser Normalization to determine the factorial structure of the instruments and the estimation of Cronbach’s alpha coefficients to assess the internal consistency of the technical performance checklists.
Inter-rater agreement was evaluated by estimating intra-class correlation coefficients and their 95% confidence intervals among the scores attributed by raters 1, 2, and 3 to the items in the checklists and among the average scores, under the following assumptions: the same raters assessed all videotaped procedures; the selected raters were representatives of a larger sample of raters (experts in ambulatorial hysteroscopy); the reliability of the mean value of multiple raters was the measure of interest, and we searched for consistency of ratings. Based on the assumptions mentioned above, two-way random-effects models were used [16].
The estimated sample size was 44 participants for five attempts with a power of 80% at an α level of 0.05. Calculations were based on the effect size (ε2) equal to 0.057 observed in a previous study [10]. Fifty-five subjects were enrolled in the study to account for eventual losses.