Research and Development of an Augmented Reality Visual Image-Guided Radiotherapy Positioning Technology

: Objective: This study is conducted to explore the development of a noninvasive and nonradiative guided radiotherapy positioning system based on augmented reality (AR) technology to increase positioning accuracy and completely eradicate positioning errors. Method: The development system includes three monitors and supporting equipment. The 3D virtual model is obtained by 3D reconstruction of the radiotherapy plan data set and is superimposed on the correct treatment position of the real treatment environment by using augmented reality production software. This approach provides the standard for the patient's positioning and guides the patient. The position of the patient then coincides with the 3D virtual model through the visual image method, and positioning error of the radiotherapy position is reduced. Results: A total of nine cases are included in this study, namely, three patients who require head and neck radiotherapy and six patients who require chest and abdomen radiotherapy. According to experimental results, no statistical difference exists between AR-guided positioning and manual positioning for patients requiring head and neck radiotherapy. However, AR-guided positioning can keep errors of x-axis, y-axis and z-axis within small ranges, such as -0.092 – 0.072, -0.052 – 0.228, and -0.192 – 0.05 cm, respectively. For patients requiring chest and abdomen radiotherapy, AR-guided positioning is not statistically different from manual positioning on the x-axis, but statistical differences were found on the y-axis and z-axis. The errors of AR-guided positioning on y-axis and z-axis are much smaller than those of manual positioning. Conclusions: The AR visual image guided radiotherapy set-up system can increase radiotherapy positioning accuracy well, especially for patients requiring chest and abdomen radiotherapy.


Introduction
With the development of computer technologies, three-dimensional (3D) visualization technology plays an increasingly important role in people's daily life. As a 3D visualization technology, augmented reality (AR) technology is developing gradually.
AR technology is a branch of virtual reality (VR) technology. Different from VR, in which users are immersed completely in the virtual environment, AR integrates the virtual and real worlds. It generates a virtual environment with vivid visual, auditory force, and touching and motion sensations [1] . Users are immersed in the virtual environment through a sensing equipment, allowing direct natural interaction between users and the environment [2] . Mobile AR integrates intelligent display technology, registration tracking technology, virtual fusion technology [3] , and man-machine interaction technology, thereby realizing seamless connection between 3D virtual object and real scenes.
Mobile AR increases the perception range of people significantly. Currently, AR is widely used in various fields, such as education, navigation, games, and medicines. It possesses very promising application prospects in the medical field.
Although AR has been increasingly applied to the medical field, it is hardly used in radiotherapy around the whole world. In 2009, Talbot et al [4] proposed the application of AR to external radiotherapy for the first time. However, this system is only used as a feasibility study. The 3D information only includes the external contour of the body surface. The camera cannot be blocked. The placement position has strict requirements.
At present, in addition to the traditional cone-beam CT (CBCT) and Electronic Portal Imaging Device (EPID)-assisted positioning methods, several new navigation methods have been applied to radiotherapy. Calypso electromagnetic navigation technology is used in the radiotherapy of prostate cancer, liver cancer, lung cancer and other diseases. It detects the electromagnetic signal fed back by beacon balise [5] through electromagnetic array and measures the therapeutic target position in 3D space for the continuous tracking of the target area. However, it is an invasive operation [6] , and the balise is prone to offset or no response [7] . Surface guided radiation therapy (sgrt) has been used in the treatment of breast cancer in recent years. It obtains the patient surface by optical projection [8] [9] and compares it with the reference surface that is generated or captured from planned CT data [10] . Combined with deep breath holding (DIBH) technology, sgrt can significantly reduce the radiation dose of breast cancer treatment center and can effectively protect the heart. However, some main problems exist. When the Gantry rotates to block a camera, the patient's surface data are lost. Positioning accuracy is also affected when the patient's skin is black or damaged.
This study aimed to explore the feasibility of using AR to improve the positioning accuracy of radiotherapy. Moreover, a noninvasive and nonradiative guided radiotherapy positioning system was developed. This system applied the 3D visualization technology to solve positioning errors caused by the invisibility of organs and tumor tissues, to decrease positioning error, and to completely eradicate the occurrence of treatment mistakes.

AR system
The main process used in this study is shown in Fig. 1. The main process of this system is shown in Fig.1.
It implemented 3D modeling to outer appearance and normal tissues, PTV, and CTV based on patient CT data.
Meanwhile, a virtual cube with the same size as the cube calibration module was established in TPS and was located in the center of the LINAC. This virtual cube was modeled together with the patient data as a whole. The cube calibration module was aligned with the laser positioning line. The cube calibration module was identified by the AR object recognition technology, and the previously constructed 3D virtual models were loaded. By adjustment, the virtual cube in the 3D virtual model was in accordance with the cube calibration module in all directions. Later, the cube calibration module was removed, and patient positioning was performed. Monitors have in-built cameras for the real-time acquisition of images of patients on the treatment couch. When the patient was spatially anastomosed with the abovementioned 3D virtual model, especially the body surface contour. The treatment can be carried out only when the actual patient's body surface was highly compatible with the 3D virtual model .

Patients
In this study, nine patients who require radiotherapy were chosen as our research objects, including three patients requiring radiotherapy on the head and neck and six patients requiring radiotherapy on the chest and abdomen. Positioning errors of each patient in three conventional manual positioning performances and three AR-guided positioning performances were collected. Informed consent was obtained from the subjects in the human experiments, and the privacy rights of the subjects were respected.

Acquisition of CT data and 3D modeling
DICOM images were collected through the CT scanning of nine patients. CT data were input into the therapy planning system (TPS) to depict the profiles of target volume of radiotherapy and important protection of organs and tissues. Finally, the DICOM-RT dataset Fig1.Manufacturing process of the system output was obtained and then input into the medical image processing software (3D Slicer version 4.11, National Institutes of Health); Open-source toolkit was applied for radiation therapy research to construct the 3D medical models. These constructed 3D medical models were input into 3D MAX (2018 version, Autodesk Company, USA) to adjust the model format. The virtual field was established according to dicom-RT field information.

System setting
The hardware system included three monitors and supporting devices. The supporting devices were placed at the right, left, and end of the treatment couch (Fig. 2). The supporting devices were adjusted to the same height of the treatment couch, and three monitors were put in. The three monitors were connected online by using the multipeer-connectivity short-range communication technology to realize the real-time sharing of environmental information data.
Three devices were fixed in the position shown in Fig. 2, and the virtual model was operated interactively in one device. The position of the virtual model in the other two devices was updated in real time with the interactive operation, thereby reducing the error due to device movement. If the rack rotates and blocks a certain device, the other two devices would also send information on the location of the virtual model. There would be no loss of model tracking.

AR simulation
The processed 3D visual model was input into the AR manufacturing software, and an AR interactive system was constructed. The main process was able to reproduce the real scenes through 3D modeling, and the object was driven in real-time to move in the 3D environment, thereby realizing AR simulation on mobile terminals. The main technologies used include visual-inertial odometry (VIO), inertial measurement unit (IMU) and simultaneous localization and mapping (SLAM). The system used AR object recognition technology to guide radiotherapy positioning. First, the reference object was scanned and the spatial feature information of the reference object was recorded in advance. When the system was used for 3D object recognition again, the image information obtained by the camera was compared with the space feature information of the reference object. After a successful comparison, the established 3D virtual model was loaded. The main steps of object recognition included feature point extraction, feature point matching, camera pose estimation and feature point location estimation.

Feature point extraction and matching
Object recognition feature points refer to the points with large difference in brightness, color, and grayness in video images obtained through vio. To ensure better real-time performance, ORB feature points were used for visual navigation in this system. The main process involved fast corner extraction and brief descriptor creation. As shown in Fig. 3 (a), fast corner extraction was performed to select pixel p from the image. Its brightness is IP, and the threshold TP was set. Pixel P was the center, and all pixels on the circle with radius of 3 are selected. If the brightness of N consecutive points on the circle was greater than IP + TP or less than IP -TP, then these points were considered as feature points. To reduce the influence of scale change and rotation change on feature point detection, the Image Pyramid and Intensity Centroid method were respectively constructed to ensure scale and rotation invariance. After extracting the key points, each

Estimate camera pose
After accurately matching the feature points of two adjacent frames, the motion of the camera was estimated according to the feature point pair, and the translation matrix T and rotation matrix R of the motion from the first frame to the second frame were solved. As shown in Fig. 3 (b), the first frame image was I1, the second frame image was I2. O1 and O2 were the center of the camera, and p1 and p2 were the feature points. These two points can be obtained by feature matching; they connecting P O 1 1 and P O 2 2 in3D space; and they intersect at point p. The fundamental matrix F can be solved by simplifying the epipolar constraint formula: is the known camera internal parameter matrix, from which the camera rotation matrix R and translation matrix t can be decomposed to determine camera positioning. The world coordinate system of the system was obtained.

Estimating the spatial position of feature points
After acquiring the camera motion, the depth of feature points that needs to be estimated to further obtain the spatial position information of feature points was determined. The distance of the same point was determined by observing the angle between two points.
As shown in Fig. 3 (c), the images observed at two places were I1 and I2. The camera was located at O1 and O2. The feature point p1 was in I1 whereas the feature point p2 was in I2. Theoretically, the straight lines O1p1 and O2p2 intersected at a point p, which was the position in the 3D scene determined by the two feature points. The calculation formula was as follows: are the normalized coordinates of the two feature points, R is the rotation matrix, t is the translation vector, and s1 and s2 are the depths of the two feature points to be solved. After obtaining the s1 and s2 , the depth of the feature points under two frames can be obtained, indicating that the spatial coordinates are determined.

Isocenter calibration of 3D visual model
Finding a new method to calibrate the isocenter of the 3D virtual model is necessary to ensure that the 3D virtual model is at the isocenter of the LINAC. During the design of TPS, a virtual cube model with the same size as the cube calibration module (Fig. 4(a)) was built ( Fig. 4(b)). First, a square at the isocenter was created on the cross-section according to two isocentric lines. Then, the sagittal plane was adjusted to expand the upper and lower levels, ensuring that its center at the isocenter of the LINAC. The virtual cube model and model of patients were processed together and input into the 3D Slicer software. In a practical operation, interactive operation is conducted on the 3D virtual model loaded (Fig. 4 (c)) after object recognition to ensure that the virtual cube in the 3D virtual model was aligned with the cube calibration module in all directions (Fig. 4(c)). At this time, the 3D virtual model was accurately located at the isocenter .

Positioning of patients
After 3D virtual model calibration, the object recognition function of the system is turned off. At this time, the 3D virtual model is fixed at the isocenter. Meanwhile, transparency adjustment is performed to display the human organ model (Fig. 5 (a)). Then, the patient is placed. First, the cube calibration module was eliminated from the treatment couch ( Fig.5(b)), and the treatment couch was lowered for conventional positioning. Patients were instructed to grasp the inspiratory depth during positioning. A 3D virtual model was used as the reference object, and the treatment position of patients was adjusted (Fig.5(c)) to conform to the previous 3D virtual model on space, especially to the body profile (Fig.5(d)). The actual patient was highly consistent with the 3D virtual model, and the treatment was carried out.

Statistical processing
CBCT scanning results were used as the gold standard, and all relevant data were processed by SPSS 22.0 statistical processing software. Measurement data were expressed by mean ± standard deviation ( X ±S). Patients requiring radiotherapy on the head and neck were designated as Group A, whereas the patients requiring radiotherapy on the chest and abdomen were designated as Group B. A pairwise t-test was performed to determine the manual and AR-guided positioning errors of both groups. The test standard was α=0.05, and statistically significant differences were observed at P<0.05.

Results
The displacement errors of Groups A and B in the x, y, and z directions are listed in Table 1  T-test was carried out for manual and AR-guided positioning displacement errors in the x, y, and z directions in Groups A and B. The sample statistics of T-test are shown in Table 2, and the paired sample test of T-test is shown in Table 3.  According to Tables 2 and 3, for  For patients undergoing chest and abdomen radiotherapy, the effect of AR-guided positioning error reduction was significantly greater than that for patients undergoing head and neck radiotherapy, and the effect of AR-guided positioning error reduction was more obvious

Discussion
An AR visual image guided radiotherapy set-up system was proposed in this study. It helped realize noninvasive and nonradiative 3D virtual modeling for the radiotherapy plan. The 3D virtual model and real space were fused by AR technology, which provided certain guidance for positioning during radiotherapy. Through an experiment, the present study showed that the proposed AR system was feasible in radiotherapy positioning. It decreased positioning error effectively and completely eradicated the occurrence of treatment errors. The AR visual image-guided radiotherapy set-up system is a type of surface guided radiotherapy. The experimental results obtained in the present study were similar to those obtained with SGRT. The positioning error of the proposed system was approximately 3 mm [11] .
For patients requiring radiotherapy on the head and neck, AR-guided positioning error was not decreased significantly compared with manual positioning error. This finding might be related to the presence of few soft tissues in the head and neck, which are not influenced by breathing movements and are only slightly influenced by small changes in the body [12] . However, the AR system can keep the positioning error within a small range.
For patients requiring radiotherapy on the chest and abdomen, AR-guided positioning errors on the y-axis and z-axis were significantly lower than manual positioning errors. Patients needing radiotherapy on the chest and abdomen often need their position fixed by using thermoplastic film and vacuum pad. The thermoplastic film fixation often makes radiotherapy markers on the thermoplastic film [13] , whereas the vacuum pad fixation needs radiotherapy markers on the body of patients and the vacuum pad. Marking lines serve as the main references for manual positioning. However, manual positioning based on these marking lines is inaccurate [14] . First, the positioning marking errors on the body of patients are high due to the relatively loose skin of some patients or to their inaccurate lying position during treatment [15] [16] . Second, the width of marking lines reaches 0.3-0.5 mm after repeated tracing, and a certain error is present in the manual alignment with marking lines. Moreover, marking lines on patients requiring radiotherapy on the chest and abdomen can have certain positioning errors, because of breathing activity and body changes that occur during radiotherapy, especially for overweight patients. AR-guided radiotherapy positioning does not require a marking line on patients.
However, the proposed AR-guided radiotherapy positioning system still needs improvement. First, To ensure the isocentric placement of the 3D virtual model, the cube calibration module is used for isocentric calibration with the virtual cube. However, in the calibration process, the virtual cube loaded by object recognition technology cannot be automatically matched to the cube calibration module. It relies on the AR interactive system for operation, but this system has low efficiency and needs improvement in terms of accuracy.
To solve this problem, we will introduce ICP algorithm for correction [17] and complete automatic matching of 3D virtual model to isocenter [18] [19] . Second, the system we proposed is used in radiotherapy patient positioning, and other functions, such as real-time monitoring during radiotherapy, are not considered. In the future, we will introduce real-time monitoring function into the system.
Future studies will address the abovementioned problems to realize the intelligence and multiple functions of the AR-guided radiotherapy positioning system. We will integrate deep learning and neural network into the AR-guided radiotherapy positioning system to conduct automatic matching between the 3D virtual model and the real model to increase positioning efficiency. In addition, a matching algorithm will be integrated to remind clinicians and researchers of the distance error between the practical body position of patients and 3D virtual model by real data.

Conclusions
A statistically significant difference exists between AR visual image-guided radiotherapy set-up system and manual positioning errors. AR visual image-guided radiotherapy setup error is smaller than the manual positioning error, especially for patients requiring chest and abdomen radiotherapy. The AR-guided radiotherapy positioning error can be kept within a small range for patients requiring head and neck radiotherapy.

Ethics approval and consent to participate
The present study was approved by the Clinical Ethics Committee of Second People's Hospital of Changzhou of Nanjing Medical University (Changzhou, China).