A Study on the Evaluation Method of Autonomous Vehicle for Fixed Targets

Recently, the automobile industry aims to commercialize autonomous vehicles, standardization and research and development are actively underway based on the specifications of automotive electronic control systems and the verification of functional errors. Based on this, autonomous driving technology is becoming more advanced, and it is preparing for an era of full autonomous driving through V2X technology convergence. It is also, expanding the models of ADAS applied vehicles such as FCW, BSD, LCA. Based on this, research is also actively underway to secure safety of vehicles and pedestrians, such as V2X, Localization, Fail Safe, to supplement the limitations of sensor-based autonomous driving. In this regard, this study proposes a theoretical formula for longitudinal relative distance computation for autonomous vehicles evaluation method, and uses test device such as DGPS to collects and verify data. In addition, a scenario was proposed for the fixed target, based on this, four types of test were formed to conduct the actual test, and the relative distances were compared, analyzed and verified. Comparative analysis results, in the second test of the first test scenario, the avoidance test of the fixed target in driving own lane, the minimum error rate was 0.5%, and in the second test of fourth test scenario, the avoidance test of the fixed target in driving own lane, the maximum error rate was 7.4%. The main cause of error is, it was judged as an error due to sensor recognition, depending on the scenario progress method, the condition of the test path, and the weather such as sunlight. In the future, we plan to conduct an evaluation on moving target.


Introduction
The automobile industry recently commercializes autonomous driving, Vehicle electronic control system specifications, functional error verification, etc. Standards and research and development are being actively carried out to analyze potential risk situations of autonomous driving and establish countermeasures. Based on this, the quality of autonomous driving technology is being enhanced, and autonomous driving technology is being developed through Wireless Access in Vehicle Environment (WAVE) communication-based Vehicle to Everything(V2X) technology fusion.
Also, due to the advancement of autonomous driving technology, as a way to consumer's acceptance expansion plan, as the spread of vehicles equipped with advanced driver assist system(ADAS) which a step-by-South Korea as of 2019 increased 0.2% year-on-year to 51,709,098, and vehicle registration increased 1.9% to 27,500,403 units. But casualties in traffic accidents decreased by 11.4% from the previous year, from 3,781 in 2018 to 3,349 in 2019. It is confirmed as a result of the increase in the installation of advanced driver assistance devices, which are functions level 0~2 of autonomous driving of vehicles.
Currently, the automotive industry applies ADAS in vehicle such as Forward Collision Warning(FCW), Autonomous Emergency Braking(AEB), Blind Spot Detection(BSD), Lane Change Assist(LCA), etc. Based on this, research is actively underway to ensure vehicles and pedestrians' safety, such as V2X, Localization, Fail Safe, to compensate for the limitations of sensor-based autonomous driving [3,4].
Regarding autonomous driving technology, the Society of Automotive Engineers(SAE) defines autonomous driving technology in 6 steps from level 0 to level 5. In September 2017, ADS 2.0(Autonomous Drive System : A Vision for Safety 2.0) was released, a safety guideline for development/verification of autonomous driving technology in the Department of Transportation(DOT) and the National High way Traffic Safety Administration(NHTSA). And guidelines suggest 12 safety design elements such as System Safety, Vehicle Cyber Security, Fallback, etc. Meanwhile, the Ministry of land in South Korea, Infrastructure and Transport is pushing for safety verification of autonomous driving commercialization in January 2020 by establishing safety standards for three step of Conditional Automation or Limited Autonomous Driving Automation [5].
Looking at this research trend for safety evaluation of autonomous driving, Christopher et al. conducted a study on generating abstract simulation scenarios for verification of autonomous vehicles [6], Mansur et al. proposed an accelerated approach for testing considering the safety and efficiency of autonomous vehicles on public roads [7]. Tim et al. proposed an open-source graphical user interface for generating multi-vehicle scenarios [8]. Lima et al. discussed global scenarios for self-driving cars focused on new test track proposals, Brazilian transportation systems and certification and approval processes [9]. Galen et al. conducted a study on the automatic generation of various and challenging scenarios for testing and evaluating of self-driving cars [10], and Chai et al. proposed a basic simulation environment for dynamic scenarios of Connected and Autonomous Vehicle [11]. Stark et al. conducted a study on the generation of verification scenarios for autonomous vehicles [12], and Park et al. studied approaches to generating multievent-based simulation scenarios for sensors and devices in autonomous vehicles [13], and Whlschke et al. conducted a study with the aim of deriving test cases [14]. Nabhan et al. conducted a study on scenario range optimization for driving scenarios for autonomous vehicles in simulations [15], and Wen et al. conducted a study on the performance analysis of NDT-based Graph SLAM for autonomous vehicles based on various driving scenarios in Hong Kong [16]. Zheng et al. conducted research on optimization model proposals, simulation tests, and rapid generation of difficult scenarios based on safety, softness, clarity, and smartness to evaluate the autonomy of autonomous vehicle [17], and Wang et al. reviewed automation levels according to the role of automation systems in the selfdriving process, and statistically analyzed autonomous vehicle accidents on public roads [18]. Tuncail et al. studied how to automatically identify simulation-based test scenarios for self-driving cars using machine learning components [19], and Huang et al. conducted a study on the synthesis of test approaches for various autonomous vehicles. [20], Reitz et al. provided a case study on repeated tests and evaluation conducted for many years to support the development of autonomous vehicle systems [21].
Until recently, the research trends are most simulationbased autonomous vehicle evaluation and scenario generation. And study on test case derivation and sensor and device development. Study on safety evaluation theory and autonomous vehicle evaluation method based on actual vehicle test is insufficient. Therefore, in this study, propose a longitudinal relative distance theoretical formula for evaluating the safety of autonomous vehicle and an integrated evaluation scenario for fixed target, and propose a theoretical evaluation method for autonomous vehicle by comparing and verifying them with actual vehicle test.

Test evaluation method 2.1 Relative distance theory formula
In a previous study, the Adaptive Cruise Control(ACC) and AEB systems' evaluation criteria were proposed as target distance values and verified [22,23]. The explanation is as follows.
The target distance theory formula for ACC is proposed using vehicle speed and distance information measured using Radar sensors to control the speed and distance and the deceleration&acceleration required for the Stop&Go function.
Where, d des is target relative distance, 0 is the distance between cars at stop, is the time difference, is the speed of the target vehicle, is the target acceleration, 1 , 2 is the gain, is the speed of the control vehicle AEB is generally controlled by Time to Collision(TTC). Assuming that Enhanced Time to Collision(ETTC), Which strengthened TTC, is equivalent to TTC, proposed AEB's target distance theory formula.
Where, TTC is time to collision, a TV , a SV is acceleration of fixed target vehicles and autonomous vehicles, v TV , v SV is average velocity of fixed target and autonomous vehicles, d vehicle is relative distance Therefore, by combined equations (1) and (2), propose a safety evaluation theoretical formula using the relative longitudinal distance of autonomous vehicles as shown in equation (3).
is a formula that is identifies relative longitudinal distances when the autonomous vehicle follows the lead vehicle. And is a formula that can identify relative distances to obstacles in front of them when braking.

Test Scenario
A scenario for the fixed target of autonomous vehicle proposed as follows. The maximum speed of the fixed targets scenario was set equally to 40km/h, the actual test limit speed for an autonomous vehicle made by company such as Google.
The scenario was configured to proceed continuously.   proceed on straight and curved roads, respectively. Fig.  2(b) is the emergency braking malfunction evaluation scenario, recognizes a vehicle stopped(parked) in the side lane and determines whether it is malfunctioning. Fig. 2(c) is a crossroad(intersection) scenario(left turn, right turn) and a scenario in which a signal is read and safely passed. Fig. 2(d) is a braking & avoidance scenario in which an obstacle is recognized at a certain distance and then slowed down to avoid or stop, rather than emergency braking. Fig. 2(e) is a lane change scenario in which a lane is changed through a set path on a straight road. Fig 2(f) is a road sign recognition scenario. IT constructed and proposed that a 30km/h sign is recognized and deceleration from driving speed of 40km/h to a speed limit of sign.

Actual vehicle test 3.1 Actual vehicle test vehicle
In this study, the actual vehicle test was validated using the proposed longitudinal distance equation and the fixed target evaluation scenario. There are a total of 3 vehicles used in the test, as in Fig. 3 (a) the Avante ADbased autonomous vehicle of H company, and fixed targets are shown in Fig. 3(b) using IONIQ of H company and QM6 of R company.
The specifications of the autonomous vehicle are shown in Table 1, and the specifications of vehicles used as fixed targets are shown in Table 2  The autonomous vehicle used in the actual vehicle test is shown in Fig. 4 of sensors such as DGPS, camera, lidar, actuator, and CAN communication equipment, and specifications are shown in Table 3.

Actual vehicle test location
The actual vehicle test was shown in Fig. 5 by utilizing the vehicle-road connection test intersection during the driving test of the KIAPI(Korea Intelligent Automotive Parts Promotion Institute). The driving test road used STANLEY LONDON's 'skid-resistance' to determine that the mean friction coefficient on the road surface was 1.08. And the actual road friction coefficient, road width, etc. were applied, so it was judged suitable for the test site.

Actual vehicle test conditions
For objective data acquisition, test scenarios must be reproduced equally repeatedly. Accordingly, the number of people conducting the actual test and the test equipment and control coding remained the same and were repeated three times under the same environmental conditions. Table 4 were carried out, and there were no changes in the weather, such as showers.

Comparative analysis of test results 4.1 Actual vehicle test results
The actual vehicle test results are shown in Fig. 6~9. Based on the proposed fixed target scenario, four types of actual vehicle test scenario were each repeated three times.  The first test was to exclude road sign recognition from the scenario of Fig. 1. And the result is Fig. 6. Referring to Fig. 6, it can be seen accelerate to an own lane follow the section on about 0-25 seconds. The about 25-35 seconds section have been identified passes without malfunction as an emergency braking malfunction scenario section. And about 30-45 seconds, the right turn section was decelerated, and the right turn was performed. About 45-55 seconds, the left turn was made to enter the second lane. And about 55-75 seconds, after the left turn was completed the U-turn proceeded. About 75-100 seconds, were after recognizing the fixed target from a distance, it slowed down slowly and returned to the main lane after evading. After acceleration, about 100-115 seconds, the lane change scenario proceeds and decelerate for the end of the scenario. And after about 115 seconds, the scenario was finished.

Figure 8 Test results(Third Test Form) (a) speed, (b) steering angle (c) longitudinal acceleration (d) lateral acceleration (e) left lane distance (f) right lane distance (g) longitudinal relative distance (h) lateral relative distance (i) yaw rate
The second test is to avoid obstacles by braking slower than the first scenario and the result is Fig. 7.
Referring to Fig. 7, it can be seen that the tendency similar to the results of the first test (form) in Fig. 6. But, about 75 to 100 seconds, after the process of recognizing the fixed target, after deceleration and avoidance, it can check difference at returning process to the main lane, in the braking velocity and longitudinal deceleration and avoiding relative distance, etc.
The third test is to recognize and brake a fixed target on straight road, and the results is Fig. 8. Fig. 8, about 0-25 seconds, it was accelerated on a straight road after the fixed target was recognized from a distance and then slowly decelerated and braked.

Referring to
The fourth test is the scenario for Fig. 1, and the result is Fig. 9.
Referring to Fig.9, it can be seen that it has a similar trend to the first test result in Fig. 6. But, due to the difference in the presence or absence of road signs, at between about 110-120 seconds can check the difference in velocity, longitudinal deceleration, avoiding relative distance, etc.
All four tests were repeated three times, and the results of each test were similar. Still the difference was judged due to the difference in time between acceleration at the start of the scenario and the autonomous vehicle's U-turn.
The relative distance at the start of the avoidance of selfdriving cars for fixed targets in the first, second and fourth tests and the actual braking completion relative distance of the third test are organized in Table 5.  Table 6 shows the error rate comparing the calculated value for the relative distance when avoiding the fixed target and the actual vehicle test result using the theoretical formula.

Comparative analysis of theoretical formula and actual vehicle test results
The error rate of the actual vehicle test result value compared to the theoretical relative distance at the time of avoidance is 0.5% in the fixed targets avoidance test in the second of the first test. The avoidance of the fixed target in the second(test) of the fourth test in the test, the maximum error rate was 7.4%.   The distance to the lane was judged to be diverged because there was no line at the crossroad. And the measurement error was judged by problems such as lane recognition by the camera and lanes reflected by sunlight. It was also judged that the change in the value at the crossroad of the third test without diverging was because the velocity at the time of passing through the crossroad was higher than that of the other(first, second, and fourth) tests.  This error was judged as an error due to sensor recognition due to the scenario progress method, the test path's condition, and the weather such as sunlight, even though the same scenario was performed.

Conclusion
In this study, a test scenario and a theoretical formula of longitudinal distance for the evaluation of autonomous vehicles for fixed targets were proposed. However, by comparing and analyzing the error rate between the theoretical value and the measured value through the actual vehicle test, it evaluates the applicability of autonomous vehicle of the longitudinal distance theoretical formula for the proposed scenario.
(1) Autonomous vehicle safety assessment scenarios for fixed target are proposed by integrated scenarios such as own lane follow, emergency braking malfunction assessment, crossroad(left turn, right turn), braking & avoidance, lane change, road sign recognition, etc.
(2) A theoretical evaluation formula of the longitudinal distance of an autonomous vehicle for distance from fixed targets was proposed.
(3) To verify the proposed scenario and the theory formula, the same device was used and repeated (6) The error of 0.5%-7.4% was the error according to the sensor's recognition, the test method of the scenario, the condition of the test road path, the weather such as sunlight, etc. and it was judged that the reliability of the proposed theory was within 8%.
Therefore, an expensive device such as rider, radar, DGPS, etc, used in the evaluating of autonomous vehicles and the need for experts have a large disadvantage of cost and time burden. But using the proposed theoretical formula, it is possible to grasp the tendency toward the longitudinal direction in the development stage. It is judged that it can be used as a safety evaluation method in an environment where actual vehicle testing is impossible.
In this study, a scenario for a fixed target scenario was proposed through theoretical formulas and actual vehicle tests for safety evaluation of autonomous vehicles and compared and verified them for verification. Testing of Moving Target will be conducted in the future.     Vehicle-road connection test intersection of Korea Intelligent Automotive Parts Promotion Institute  Test results(Fourth Test Form) (a) speed, (b) steering angle (c) longitudinal acceleration (d) lateral acceleration (e) left lane distance (f) right lane distance (g) longitudinal relative distance (h) lateral relative distance (i) yaw rate