Development of novel Mixed Reality-Based Simulator for Endoscopic Transnasal Surgery with HoloLens and multiple AR tracker

Virtual reality (VR) and mixed reality (MR) are now widely applied for preoperative simulation and intraoperative navigation. We developed an MR-based simulator for endoscopic transnasal surgery (ETNS) with a head-mounted display HoloLens and evaluated its usefulness. This simulator consisted of MR images of patients and an MR endoscope. HoloLens was used for projection of MR images and recognition of markers. The MR images were reconstructed from the preoperative images of patients and superimposed onto the endoscopic training model of the head. The MR endoscope was superimposed onto a three-dimensional (3D) printed replica of an endoscope. The MR endoscopic images from the replica of the endoscope were projected in the operator’s visual eld. The MR images followed the manipulation of the replica of the endoscope through a nasal cavity on the training model. To evaluate the developed simulator, the MR endoscopic images were compared with video-recorded actual operative endoscopic views. And face validity and content validity of the simulator were evaluated by senior residents, using a 5-point Likert scale.


Abstract Background
Virtual reality (VR) and mixed reality (MR) are now widely applied for preoperative simulation and intraoperative navigation.

Methods
We developed an MR-based simulator for endoscopic transnasal surgery (ETNS) with a head-mounted display HoloLens and evaluated its usefulness. This simulator consisted of MR images of patients and an MR endoscope. HoloLens was used for projection of MR images and recognition of markers. The MR images were reconstructed from the preoperative images of patients and superimposed onto the endoscopic training model of the head. The MR endoscope was superimposed onto a three-dimensional (3D) printed replica of an endoscope. The MR endoscopic images from the replica of the endoscope were projected in the operator's visual eld. The MR images followed the manipulation of the replica of the endoscope through a nasal cavity on the training model. To evaluate the developed simulator, the MR endoscopic images were compared with video-recorded actual operative endoscopic views. And face validity and content validity of the simulator were evaluated by senior residents, using a 5-point Likert scale.

Result
The 3D MR images through HoloLens correlated well with the actual intraoperative views. Although there was an innate learning curve with the simulator, the face validity and the content validity demonstrated effective simulation of an operative eld of view with real-time characteristics of the surgical procedure.

Conclusions
Our developed simulator for ETNS will contribute to learning the unique and the limited surgical elds through the narrow surgical corridor with endoscope, particularly for novice surgeons.

Background
The use of virtual reality (VR) and augmented mixed reality (MR) technology has burgeoned in the neurosurgical eld in the last two decades. Commercially available MR and VR technologies have been widely used for preoperative simulation and intraoperative navigation. Several papers have evaluated the usefulness of a VR simulator and demonstrated that as a training method, it could become an alternative to cadaveric training or hands-on practice. For endoscopic transnasal surgery (ETNS), a virtual endoscopy (VE), one of the VR simulators forms, was also described for training use. Additionally, the use of VE and MR endoscope has been reported for intraoperative navigation [1][2][3]. NeuroTouch-Endo (National Research Council Canada, Ottawa, ON, Canada) and PHANToM (SensAble Technologies, Woburn, MA, USA) series were used to form a haptic interface for the VE simulator. Rosseau et al. reported that the simulation software for NeuroTouch-Endo VR simulation of ETNS provides an opportunity to evaluate performance metrics for use during neurosurgical residency training [4]. It allows effective preoperative assessment of the patient anatomy, reducing the risk of injury to anatomic structures near the surgical target and helps novice surgeons overcome the steep learning curve associated with the endoscopic procedure [5]. However, these devices require excessive investment to establish them and to bring them into versatile use. For the purpose of developing a versatile simulator for ETNS, we focus on a see-through head-mounted display: HoloLens (HL: Microsoft, Redmond, WA, USA). HL provides the merging of real and virtual worlds, which is called mixed reality (MR), to produce new environments and visualizations where physical and digital objects coexist and interact in real time (Microsoft, https://docs.microsoft.com/en-us/windows/mixed-reality/mixed-reality). Some studies have shown the potential of pre-and intra-procedural navigation with HL for surgical elds [6-8]. Herein, we report development of a novel and versatile MR-based simulator for ETNS with HL and evaluate its validity and e cacy for training use.

Material And Methods
The endoscopic MR simulator as developed consists of models of the AR patient, an AR endoscope, and a three-dimensional (3D) printed replica of the endoscope. The concept behind our endoscopic MR simulator was to display the virtual endoscopic images that follow manipulation of a replica of the endoscope. We obtained patients' images as data formatted with digital imaging and communications in medicine (DICOM), including preoperative head computed tomography (CT), CT angiography, and enhanced magnetic resonance image.

Three-dimensional modeling
The images of bone, skin, soft tissues, arterio-venous systems and lesions were segmented and these three-dimensional (3D) models were reconstructed on a commercially available workstation (Plissimo Era; Konica Minolta Ltd., Tokyo, Japan). Then these data were exported as Standard Triangulated Language (STL) formatted les. These STL formatted polygon data were optimized for our simulator with MeshLab (ISTI-CNR; Pisa, Italy, www.meshlab.net) and converted to a FilmBox (FBX) formatted le with Blender, a professional free and open-source 3D computer graphics software package, for creation of MR models (Blender Foundation; Amsterdam, the Netherlands, www.blender.org) (Fig. 1A).

Application for the MR ETNS simulator
We used a free integrated development environment called Unity (Unity Technologies, San Francisco, CA, USA; https://unity.com) to create an application of the endoscopic simulator for HoloLens. Microsoft published the Mixed Reality Toolkit software development kit (MRTK SDK), which is a collection of scripts and components intended to accelerate the development of applications targeting the Microsoft HL. It enabled us to create MR applications easily for HL in Unity. Vuforia SDK was used for speci ed MR marker creation and tracking. By adapting MRTK and Vuforia SDK, the MR ETNS simulator was created.
Cameras, using one of the scripts of Unity, that capture and display the MR world to the operators are con gured on the tip of the MR endoscope in Unity. That enables the display the 3D VE images in the eld of view of the operator wearing the HL. Then the individual patient's 3D model (FBX formatted) was inserted into the created application. The 3D model was projected at real size. The individual application was deployed to the HL via Visual Studio (Microsoft, Redmond, WA, USA) ( Figure. 1B). The application of the simulator created assets, which become representations of several scripts and modi ed SDKs. To create another patient's simulator, one only exchanges the 3D model of the patient.

MR projection
A 3D model of the endoscope (EndArm 0°; Olympus, Tokyo, Japan) was created with Fusion 360 CAD software (Autodesk, Inc., San Rafael, CA, USA) ( Fig. 2A), and a replica of the endoscope with a cubic MR marker was printed with a 3D printer (Flashforge, Zhejiang, China) (Fig. 2B). The 3D model of the endoscope was imported into Unity and linked to the cubic marker by Vuforia. The MR patient's models were projected and xed to an endoscopic training model (transsphenoidal pituitary surgery training model; SurgTrainer, Ltd, Ibaraki, Japan) with a two-dimensional (2D) MR marker. The MR endoscope was projected onto the replica of the endoscope model with the cubic marker ( Fig. 2C, 2D). The HL tracked the cubic marker, and thereby the MR endoscopic images, so the manipulation of the replica of endoscope was displayed in the operator's eld of view (Fig. 2E). Then the 3D virtual endoscopic images from the tip of the MR endoscope were established through the HL (Fig. 2F).

Evaluation of MR-based simulator for ETNS
For evaluation of the e cacy of our simulator, patients who had undergone ETNS at our hospital between January 2016 and December 2017 were included in this study. Characteristics of the patients, including age, sex, histological diagnosis, deviation of the nasal septum, and Vidic type of sphenoid sinus [9], are provided in Table 2. Deviation of the nasal septum and Vidic type of pneumatization of the sphenoid sinus were evaluated from preoperative plain CT in each patient. These anatomical variations might complicate the understanding of optimal anatomical orientation for novice surgeons. The virtual endoscopic MR images were compared with a video-recorded actual operation eld. Additionally, the developed MR simulator with HL applied for each of eight patients was operated by three senior residents of our department. After the procedure, the participants answered a questionnaire about face validity and content validity of the simulator, using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree). Face validity means the appearance of MR anatomical structures, and content validity means effectiveness in teaching (the reproducibility of the procedure by the trainer).
Eleven anatomical structures related to face validities and four items related to content validity are shown in Table 1. The assessments were performed in the order of patient number as described in Table 2.   The assessments were performed in order of patient number.
The scores with under bar emphasized below the score of 3 (mean neutral).

Statistical analyses
Statistical analyses were performed with the JMP 14 statistical software (SAS Institute, Inc., Cary, NC, USA). The validity scores were statistically analyzed with a Mann-Whitney U-test. The differences among the scores for Vidic type of sphenoid sinus (presellar type and sellar type vs. postsellar type), deviation of the nasal septum (deviation vs. no deviation), and the assessment group (patient numbers 1-4 vs. 5-8) were assessed. A value of p < 0.05 was considered to be statistically signi cant.

MR projection
Three-dimensional modeling and application was deployed successfully in all eight patients. Segmentation of MR models of each case was done within about 2 hours and MR application was deployed to the HoloLens in about 5 minutes. Three-dimensional models were well registered to the endoscopic training model with the 2D MR tracking marker, which satis ed the condition of operative simulation. The MR endoscope was completely registered to the replica of the endoscope and followed its motion by tracking the cubic marker. The endoscopic view from the tip of the MR endoscope was displayed in the operator's eld of view with HL successfully, and the views were coordinated with manipulation of the replica of the endoscope (Supplemental video 1). These anatomical structures of the patient's nasal cavity, sphenoid sinus, and intracranial structure including a tumor were displayed in the view. The MR anatomical structures were correlated with the actual operative view (Fig. 3).

Evaluation of the simulator
Eight male patients were included in this study, with a mean age of 52 years (range 24-76). The histological diagnosis was a nonfunctioning pituitary adenoma in seven of the patients and a growthhormone-producing adenoma in one patient. Preoperative plain CT of the patients revealed that the Vidic type of sphenoid sinus was of the postsellar type in ve cases, sellar type in two cases, and presellar type in one case. Deviation of the nasal septum was revealed in three cases in the video-recorded actual operation eld. The mean scores on the 5-point Likert scale for each item is shown in Table 2. There was no signi cant difference between the groups of Vidic types in the score (p < 0.05) except for sella turcica, carotid protuberance, and opticocarotid recess. The scores were statistically signi cantly higher for patients who had a deviation of the nasal septum in teaching anatomy and in the surgical plan of content validity, compared within the patients without a deviation of the nasal septum. Between the earlier four cases and the later four cases, there were signi cant differences in the face validity of evaluations of inferior turbinate (p < 0.02), sphenoid ostium (p < 0.0001), sphenoid rostrum (p < 0.0001), planum sphenoidale (p < 0.0001), and sella turcica (p < 0.04). There were signi cant differences in content validity for all questioners (p < 0.02).

Discussion
The simulator developed and reported here is the rst described MR endoscopic simulator using the HL for ETNS. The concept of this simulator involves MR and VR being combined as one application. Actually, the view from the tip of the replica of the endoscope is a VR image, and the VR image projection combined with MR tracks creates the realistic feedback as an MR simulator. The HL is a stand-alone head-mounted display that contains four environmental cameras, a depth camera, and Windows OS system. Thus, this device enables one to immerse in the MR environment, with hands free and a wireless connection. Long Qian et al. evaluated which choice among HL, ODG R-7, and Epson Moverio BT-200 was the most accessible for medical use as a head-mounted display. They concluded the HL best performed among the three [10]. Among these head-mounted displays, the HL's advantage is its versatility.
One of the most important advantages of computer simulators for surgical training is the opportunity they afford for independent learning [10]. Most of the commercially available simulators so far have proved costly because of their complex computer hardware and software and are not easily accessible or convenient [11]. Our simulator needs only the HL, two MR markers, the replica of the endoscope, and the training model, which are all easily portable, so they can be used anywhere, at any time. In the process of creating our MR, all software was free and could be created inexpensively. Additionally, creating an application was easier with the Unity engine than with other programming systems, and after the application was created as an asset, functional options could be added to it. For example, in this study, MR markers and the replica of the endoscope were among the functional options.
European Association of Endoscopic Surgeons guidelines for validation of virtual reality surgical simulators specify that they must be able to mimic visual-spatial and real-time characteristics of the procedure, and preferably provide realistic haptic feedback [12]. A previous study of a VR simulator showed that the face validity was assessed in three separate domains: visual appearance (photorealism), haptic feedback, and user-friendliness [13]. The simulator developed and described in this report provides user-friendliness with versatility. We consider that this simulator achieved photorealism, which is demonstrated by the participants' improvement in understanding of anatomical structures. Regarding the haptic feedback, commercially available ETNS simulators created a sense of reality as judged by haptic feedback [14] [15,16]. On the other hand, our developed simulator has no haptic feedback function for surgical manipulation, such as mucosal dissection, drilling bony structures, and tumor removal. This is a disadvantage of our simulator compared with the commercially available simulators. However, in our simulator, the projection of the MR to the real model could create interactive feedback. The endoscopic training model created a limitation of motion and real contact resistance for the replica of the endoscope.
It is not necessary to use the endoscopic training model as we described; we could also apply whatever would reproduce the restriction of the nasal cavity. Additionally, for this study, we decided it was more important to simulate a limited operative eld of view with a narrow surgical corridor, rather than to reproduce surgical manipulation like a surgical trainer. There was an innate learning curve associated with VR itself for previous VE simulators, because of using the unnatural haptic device [11]. Because we would like to assess whether there is a learning curve with our simulator, we analyzed the scores of validities in the rst half term and the second half term of assessment. There was a statistically signi cant difference in each group. There was also an innate learning curve for manipulation of the replica of the endoscope and understanding the VR surgical elds.
The HL has problems with processing speed and the device's viewing angle. A stable MR in the HoloLens is expressed by 60 frames per second (FPS). The HL has a limit to the number of polygons that could display without FPS delay. If the 3D model has more than 100,000 polygons, the processing speed becomes slow, and there is FPS delay. This causes loss of orientation of the MR marker and a time lag between operation and display. To avoid these factors, a minimum segmentation of 3D model was required; however, this might take more time. Furthermore, the reconstructed 3D image of the paranasal sinus from CT parameters applicable in clinical routine data was ine cient [17]. However, cutting off polygons causes a loss of reproducibility of MR images and might result in the face validity being adversely affected. As with other wearable computers, there is some potential for nausea and vertigo based on the predisposition of the users. These points are disadvantages of our simulator.
MR simulator for the future Our concept for the MR simulator would apply intraoperative real-time navigation for the ETNS. However, the HL continuously uses spatial mapping to provide us with stable MR imaging. Spatial mapping means creation of a spatial surface mesh of surroundings and adaptation of MR images to that mesh. However, the spatial mesh is often too coarse to merge AR with real objects precisely. Vuforia supports the stability of MR, but it remains in need of improvement. Thus, at this point, the accuracy of registration and tracking with the HL was not good enough to use for neuro-navigation, especially in microscopic and endoscopic neurosurgery.
MR technology is making remarkable progress. In fact, we had used the HL since 2017, and in the beginning, the projection of MR and MR tracking were unstable, with delays and mismatch. The setup was found to be problematic for medical application. But the continuous update of software and SDK gradually improved these issues. Additionally, some developers published usage tips and pitfalls on the Internet. There is strength in using a versatile game engine and Microsoft production. The disadvantages that we have reported as limitations can be improved over time. This development and improvement may also provide MR navigation with the HL that satis es microscopic or endoscopic neurosurgeons.

Conclusion
The MR simulator developed and reported here for ETNS will contribute to learning in the unique surgical eld of ETNS, especially for novice surgeons, due to its versatility.  (2241)). Written informed consent was obtained from all participants in the study.
The written consent for publication has been obtained from the patients shown in the gures and supplementary les.

Consent to publish
Not applicable.
Availability of data and materials The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare that they have no competing interests.