To quantitatively investigate the working performance of the visual odometry method, we conducted comparing experiments on the evaluated robot displacement and orientation with those measured with an external 3D laser scanner FARO Focus 3D x130. The 3D scanner can perform high accuracy 3D reconstruction with High Dynamic Range (HDR) camera, realizing the reconstructing accuracy within 2 mm. We considered this 3D scanner could provide reliable and accurate measurement of the current robot pose, and serve as the reference object in the comparison. Concretely speaking, in the specific motion, we used the modified visual odometry method to perform localization, and recorded the evaluated robot displacement and orientation. We installed four special-made spheres on each corner of the robot, to be recognized in the reconstructed results. Thus, we used the position of the spheres to calculate the position and orientation of the robot, and compared with the evaluated results from visual odometry.
As for the evaluating criterion, we utilized the localizing error rate applied by PMORPH2 in previous investigations, error within 100 mm with 1500 mm displacement. Since 100 mm is the width of a single grating lattice, and 1500 mm is the regular distance between multiple investigating points.
6.1 Basic experiments with simplified environments
In the beginning phase, we used a simplified environment by splicing several grating blocks to form the grating floor. We allowed the robot to perform various types of motion, including translation, rotation, and a combination of them. Figure 8 shows one of the generated maps with the camera trajectory in purple lines. Table.2 shows the comparing results between the evaluated displacement and the measured displacement.
Table.2. The comparing results of visual odometry and 3D scanner under a simplified short-ranged route
Result type | x-direction displacement (mm) | y-direction displacement (mm) | orientation (degree) |
Visual Odometry | 738.0 | 1247.9 | 95.5 |
3D scanner measuring | 728.3 | 1256.0 | 97.7 |
Thus, the comparing results showed that the y-direction error rate was 9.7 mm/1500 mm, and the x-direction error rate was 20 mm/1500 mm. We considered the accuracy met the requirement in such a simplified condition. Besides, the grating lattices in the reconstructed map appear quite regular, which made it possible to count the lattices’ numbers to estimate the robot position.
6.2 Simulated experiments in JAEA Naraha Center for Remote Control Technology Development
To further investigate the performance of the visual odometry method, we utilized the experimental instrument water tank in Naraha Center for Remote Control Technology Development, provided by Japan Atomic Energy Agency (JAEA). Figure 9 shows the structure of the three-floored water tank. At its center, a water tank with a 4690 mm diameter was utilized as the simulated basement body. The top floor served as supporting platform of the investigating robot. Across the water tank, we prepared a wooden bridge with a small grating part, whose lattice size was same as that of the No.1 reactor.
Concretely speaking, we would like to simulate the investigating route as the red line in the right part of Fig. 9. Since there is only a small grating part on the wooden bridge, to simulate the grating floor, we took images of grating lattices and printed them to covered the floor.
Thus, we repeated the simulated routes three times and the comparing results are shown in Table 3. Figure 10 shows one of the generated maps by visual odometry method.
Table.3. The comparing results of visual odometry and 3D scanner under simulated routes
Result type | x-direction displacement (mm) | y-direction displacement (mm) | Orientation (degree) |
Visual Odometry-1 | 3043 | 3022 | -117.2 |
3D scanner measuring-1 | 3025 | 3180 | -126.9 |
Visual Odometry-2 | 3134 | 3122 | -127.2 |
3D scanner measuring-2 | 3146 | 3226 | -129.1 |
Visual Odometry-3 | 3249 | 3130 | -133.9 |
3D scanner measuring-3 | 3192 | 3041 | -134.2 |
Thus, we calculated the error rate from these datasets, and the averaged error rate is 35 mm/1500 mm, and the maximum error rate is 54 mm/1500 mm, which satisfied the required error rate. Besides, we compared the reconstructed objects like the cross-shaped metal supporters under the small grating part on the wooden bridge with the realistic one, and confirmed the reconstructed correctness.
6.3 Camera parameters ranges according to the required accuracy
Through the experiments in the simulated experiments, we could confirm that under the specific experimental conditions, the visual odometry method was feasible to provide the required localizing accuracy. However, in the on-site investigations, there might be more restrictions according to the environmental conditions. Like the size limitations to pass through the access penestration, limited camera performace in the extreme situation. In order to provide useful information for the further development of the practical model, in the following sections, we would like to focus on the parameters of the camera pose and program setting and use an experimental method to investigate the available ranges to keep the error rate within 100 mm/1500 mm. Table.4 shows the concerned factors, their default values in the previous experiments, and the adjusting ranges. Since we mounted the camera system on a metal frame, the adjustable ranges of the camera pose might be limited.
Table.4. The default values of the concerned parameters
Parameters | Default values | Adjusting ranges |
Camera height (mm) | 400 | 250 ~ 600 |
Camera inclined angle (degree) | 50 | 30 ~ 70 |
Frame per second (frames/s) | 30 | 12 ~ 60 |
Minimum number of feature points | 250 | 50 ~ 350 |
Thus, we would like to investigate the changing tendency of the localization accuracy corresponding to each parameter within this adjustable ranges individually, by keeping the other parameters constant. Still, we utilized the 3D scanner as the referring device. We focused on two types of simplified motion modes, translation, and rotation. We controlled the robot to perform repetitive motions under different parameter setting and recorded the estimated displacement or orientation by visual odometry.
Thus, to quantize the localizing performance, we denoted the desired displacement as (xd, yd, θd), which were adjusted according to the predetermined motion modes including translation and rotation. Correspondingly, we denoted the estimated displacement as (xet, yet, θet) for the translation motion, and (xer, yer, θer) for the rotation motion. In the predetermined translation mode, the robot moved along the forwarding direction for 1000 mm on the grating floor. Thus, we denoted the error rate as ERt, the proportion of the difference between the main displacement index and the estimated one. In the translation mode, the main displacement index should be the y-direction displacement. While, SEt represented the shift error occurred in the displacement estimating process of translation, defined by the x-direction displacement as the transverse shift error. Eq. (3) and Eq. (4) show the concrete definitions of these indexes. In the predetermined translation motion, (xd, yd, θd) should be set as (0,1000,0).
For the predetermined rotation mode, the robot rotated around the center for 45 degrees clock wisely. Thus, we defined the error rate ERr as the proportion of the orientation difference over the desired orientation. We defined the robot center displacement as the shift error SEr. Eqs. (5) and (6) shows the relationships. In the predetermined rotation motion, (xd, yd, θd) should be set as (0,0,45).
Thus, we would use the error rate within 100 mm/1500 mm in translation motion, and the same proportion 3degree/45degree in a rotation motion as the requirement. By maintaining the error rate within the range, we adjusted the parameter to acquire the adjustable ranges. As the results, we presented the changing tendency of both error rates and the shift errors with the changing parameters in Fig. 11, Fig. 12. The vertical axis indicated the error rates with the line chart, the horizontal axis showed the changing parameter. We also included the shift error value on the figure with the bottom bar chart, as an alternative index of the accuracy.
Table.5. The parameters range meeting the error rate requirement
Parameters | Available ranges |
Camera height (mm) | > 300 mm |
Camera inclined angle (degree) | 40 ~ 60 degree |
Frame per second (frames/s) | > 24 fps |
Minimum number of feature points | > 150 |
Reflecting on the overall tendency of both the error rates and the shift errors, we concluded that the shift error would also be decreasing when the error rate was low. The ranges of the parameters were listed in Table 5.
Generally, we regarded that the camera height had little influence on the localization accuracy of translation mode. A similar situation occurred in the rotation mode; however, when the height was below 0.3 m, error appeared to arise in both the angle and position estimating process, as the error rate and shift error were quite significant. We considered that this was due to the limited view range and illumination in low height. For the camera inclined angle, in the translation mode, we noticed the relatively significant errors when the angle was below 40 degrees. And the tendency in the rotation mode indicated the adjustable ranges within 40 ~ 60 degrees. We considered that the view transforming process limited the angle ranges, while the horizontal camera configuration of PMORPH might not be directly applied. When it comes to the program setting, the results appeared to be similar that both the parameters had a minimum limit to maintain the normal working performance. We concluded that the fps should be more than 24 frames/s and the minimum number of feature points should be larger than 150.