A Novel Lane Line Detection Based on Multi-feature Fusion and Windows Searching

Lane detection embedded in intelligent vehicles can greatly improve the security of automatic driving. This work offers a new approach towards lane detection in the video in real-time combining multi-feature fusion and window searching. As the pre-procession, polygon filling is adopted to locate ROI (Region of Interest) in the video frames, which contain the lane-lines to be detected. To remove the backgrounds in the ROIs, we extract and fuse the features of color, histogram, and gradient of line lanes. Based on the density distribution of the pixels in the line lanes, the initial location is found by homography transformation. Then all candidate pixel points in the whole lane-line are extracted in the way of window searching. Finally, On the basis of the obtained lane-mark coordinates, a curve model is defined, and the model parameters are obtained by Least Square Estimation (LSE). Experimental results show the robustness and instantaneity of the proposed algorithm with the accuracy of 96% and the detecting time of only 20.7ms. In addition, lane-lines with misleading backdrops can also be detected such as yellow lane lines on the ground, shadow, bright light, lane-line defects and traffic light


Introduction
The unceasing progress in China's economy and urbanization have enhanced internal expressway mileage and automobiles, and resulted in great traffic jams. Besides, with the requirements of comfortable driving, Advanced Driver Assistance Systems (ADAS) and autonomous driving techniques have attracted widespread attention from domestic and foreign scholars. In recent years, the major companies such as Baidu, General Motors, Google and Uber have delved into Autonomous Vehicles (AVs), with higher mobility, fewer faults and smarter choices. As a key of autonomous driving technology, lane detection analyses the position of the vehicle on the road by extracting lane information from the image. It provides a decision basis for the steering and lane change of the vehicle. At present, there are two main methods for lane detection: Deep Learning (DL) [1][2][3] and traditional feature extraction. DL mainly uses a multi-layer convolutional neural network and a large number of samples are used to complete lane line recognition through continuous learning, without the need of manually designed features. The method has the advantages of strong robustness and high accuracy, but the biggest defect of high computational complexity and large time overhead, which requires GPU for supporting. In contrast, traditional methods focus on the characteristics of the lane line itself, such as the color, texture, brightness and the type of lane line to separate lane lines from the complex background [4][5]. For example, Nima et al. [6] covert RGB color space into YCbCr space and HSV space, and implement lane detection by the gradient detection operator. Jamel Baili et al. [7] use a horizontal difference filter to simplify the edge detection process and group the detected edge points into a straight line by the improved Hough transformation. Qian Kidd et al. [8] utilize the Sobel edge detection operator to detect lane edge information, and detect possible lane line by the modified Hough transform on the region of interest. However, the above three lane detection methods are susceptible to illumination, road shadow, and lane line defects. Under such defective conditions, the lane curves are hardly detected and the time overhead is large. In this paper, we propose a novel lane detection method by combining multi-feature fusion and window searching. The main contribution of the proposed method is summarized as follows: 1) the color, gradient and grayscale features of lane line are considered. Then, the multi-feature fusion algorithm is proposed to fuse these features, creating a binary image and enhancing the robustness of lane detection in complex environment. compared with the deep learning features. Compared with deep features, the time complexity of extracting these three features is greatly reduced, and the accuracy is higher; 2) If the left and right lane are detected in the previous frame of the video image, The LFPF (Line Fits by Previous Fits) method is proposed to search lane-line pixels. It uses the relationship between the front and back frames, and the lane-line pixels in the current frame are searched in the vicinity of the left and right lane-line boundary equations fitted in the previous frame, which can effectively reduce the time complexity of the algorithm and improve stability of lane detection.

Proposed Method
The algorithm mainly includes Image Pre-procession, Multi-Feature Fusion, Lane-line Fitting and Lane-line area visualization. The overall architecture of the proposed method is shown in Fig. 1. In Image Pre-procession, the video frames are resized firstly. Different angles of the chessboard images are used for camera distortion coefficients. The correspondence between 3D word and 2D image points? for a bunch of images is stored and applied to the actual camera. Finally, the polygon-filling is used to obtain a trapezoidal ROI (Region of interest), which contains lane-lines to be detected so as to avoid the unnecessary calculation. In the module of Multi-Feature Fusion, color space conversion, histogram equalization, and gradient operator filtering are performed on the pre-processed images. Then the logical OR operation is used on the pixels of the image. At last, a binary image is obtained by morphological filtering and threshold process. After the above steps, the lane-line is separated from the complex background.

Multi-feature Fusion
The white lane-line and the road surface gray value are too close to extracting the lane information after the grayscale processing when the image captured by the camera is too bright or too dark. Furthermore, the street tree will generate a shadow under the illumination of the sun, so that the information of the lane-line is lost when it is covered on the white lane. Finally, the significance of the yellow lane is reduced and cannot be separated from the road surface background after the image is grayed. In order to address the above problems, the multi-feature fusion algorithm is used to process the image.

Color Transforms
When the light is relatively stronger, the yellow lane lines in the RGB color space appear less obvious than the road surface after graying. Therefore, A. Ajmal [9] applied H-V or S-V channels to improve the quality of object saliency detection. According to [9], the color space of the preprocessed image is converted from RGB to HSV in this paper. The HSV model focuses on color representation. By setting a certain range of hue (H), saturation (S), and lightness (V). HSV model performs better on the segmentation of specified colors. The setting formula is as follows: Where , represents the range of Hue values, , , , represent the ranges of Saturation and Value. Fig. 2 shows the extracted yellow lane line. Fig. 2(a) is the pre-processed image, and Fig. 2(b) is the result after threshold procession in HSV color space. It can be seen that the algorithm can effectively detect the yellow lane lines, but the white lane lines are completely ignored.

Histogram Equalization
When the image is dark overall, the white lane in the image is not significant with respect to the road surface after the image is grayed out. Furthermore, the brightness of lane-line is also darkened when the edges are worn. For better segmentation, the image contrast should be enhanced. Generally, methods for improving image contrast include direct contrast enhancement [10] and indirect contrast enhancement [11]. In this paper, histogram equalization, an indirect method is used to enhance the contrast of the image to detect white lane-line. Considering that the lane lines are mostly white, the brightness of the lane lines concentrates is within a certain range. The following formula is used to find a binary image of the white lane: In the formula, and respectively represent the minimum and maximum gray values after the histogram equalization. By testing on thousands of frames under different lighting conditions, and are set to 200 and 250 empirically. Through histogram equalization, the white lane line is effectively detected.

Gradient Filtering
In order to better detect the boundary of the lane-line and facilitate subsequent procession, the Sobel operator is used to filter the image to get the edge information of the lane. Different from the traditional Sobel edge detection algorithm, the image is converted to grayscale. The horizontal and vertical Sobel operators is used to filter the image separately to obtain the images S_h and S_v. (4) In the above functions, ℎ is horizontal Sobel operator and is vertical Sobel operator. Empirically, ℎ in (5) is added to obtain the expected edge detection, in which S_h and S_v are the images filtered by ℎ and .
() is the maximum pixel value. Fig. 3(a) is the pre-processed image, Fig. 3(b) is the filtered image by Sobel operator. It can be seen that the lane line boundaries in the ROI have been precisely detected.

Image Fusion
Correctly creating the binary image from the input frame is the very first step of the whole lane detection. It plays an important role in creating the binary image according to the designed features for detection. Image Fusion uses a specific algorithm to combine two or more images into a new image. The fusion result can use the correlation of two (or more) images in space-time and the complementarity in information, and make the image obtained after fusion have a more comprehensive and clear description of the lane line. In order to detect the white lane, the histogram of the input frame is equalized to highlight the actual lane-line before threshold process. For the yellow lane, thresholds on the channels of H, S and V is employed in HSV color space. Furthermore, the input frame is convolved with Sobel kernel to get an estimate of the gradients of the lines. Then, the logical OR operation of pixels is performed to fuse the features of color, histogram and gradient. Finally, the morphological closure is used to fill the gaps in the binary image. The image obtained after pixel-level image fusion has more detailed information, such as the extraction of line-lane edges and textures, which is conducive to further analysis, processing, and understanding of the image, and can also expose potential targets, which is conducive to judgment and identification of potential target pixels Point operation, this method can save as much information in the source image as possible so that the content and details of the merged image are increased. The result of the multi-feature fusion algorithm is shown in Fig. 3. Compared with figures 2(b) and 3(b), both yellow and white lane lines are detected in Fig. 4. It can be seen that the Multi-feature fusion algorithm can effectively eliminate noise interference and has good robustness to yellow and defect lane condition

Line Detection
At present, straight-line models are mostly used in lane detection algorithms to fit lane-lines. Considering that only color [12][13] or edge [14][15] features are extracted, the model is of low complexity, good real-time performance, and robustness to straight lane-lines. However, this method is limited to the inaccuracy of fitting results, i.e., lanelines can hardly be fitted in the form of curves in practical applications. In this paper, we are seeking a way to fit more precise curves for lane-line detection.

Lane Pixel Search
After the Multi-Feature fusion processing, high-quality binary image of the lane line can be obtained. Then, the wider-range and multi-lane feature data is obtained by converting the binary image into the top-view space via homography transformation. Furthermore, a lane line detection method based on pixel density distribution is proposed in the lane line recognition process, and lane window pixels are extracted using window searching. In order to identify which pixels of a given binary image belonged to lane-lines, there are two possibilities at least. The first one is never identified where the lane-lines are in a new brand frame, the LFSW (Line Fits by Sliding windows) method is proposed to perform an exhaustive search on the frame. This method starts from the bottom of the image, precisely from the peak location of the histogram of the binary image, two windows is slid towards the upper side of the image, decided which pixels belonged to which lane-line. This method has high accuracy, but high time complexity The second one is confidently identified lane-lines on the previous frame in a video, lane-line pixel search can limit in the neighborhood of the lane-lines detected before, since all the video going at 20fps, the lines won't be so far. The LFPF (Line Fits by Previous Fits) method is proposed to keep track of detected lines across successive frames, which helped faster the lane-line search. The specific steps of the algorithm are as follows: Step1. The homography transformation is performed on the binary image processed by the multi-feature fusion algorithm since it can map points in one plane to another, which is convenient for subsequent lane-line pixels search with feature extraction windows. Another advantage is lane-lines can be identified even when it is defective. Though this step, the image of homography transformation is obtained, as shown in Fig. 5. White areas are lane line pixels. Figure 5 The Image of homography transformation.
Step 2. The density distribution of pixel is counted on the image of homography transformation, as shown in Fig. 6, where the abscissa is the length of

Lane Line Fitting
In order to better adapted to the situation where the lane-line is a straight or a curve line, the quadratic polynomial equation is used to improve the robustness of the lane detection, the fitted quadratic polynomial is described as: where and is respectively the abscissa and ordinate values of all lane-line pixels obtained by the feature extraction windows, , and are unknown parameters. Assuming that a lane had acquired the coordinates of N pixels, the set of pixels could be expressed as ( , ), = 1,2, … , . The error function of the least square is identified.
The least square approach is applied to get the unknowns parameters b a, and c in equation. Finally, the boundary equation of the lane-line can be determined, as shown in Fig. 8.

Experimental Results and Analysis
In this section, a series of experiments are presented to verify the proposed algorithm. The goal is to improve the robustness and efficiency of lane line detection in complex environments and visualize lane line areas. The video frames set for the experiment are collected from the data recorded by the driving recorder in a car, with the size of 768×432 a frame. All the simulations are run on Intel (R) Core (TM) CPU I5 8500 @ 3.00Ghz with a 16 GB RAM.

Lane Line Detection Result
When During the test, the accuracy of the algorithm is evaluated by comparing the difference between the detection result and the ground truth. The correct rate of the lane line detection is the ratio of the correctly detected lane number of frames and the total number of frames. The experimental results are shown in Table 1 and Figure 9. When the car is driving, the algorithm is less affected by noise, marking interference, yellow lane lines, shadow interference and bright light. It has certain accuracy and robustness.

Comparison with other Lane Line Detection Method
The next experiments are to demonstrate the prominent robustness and instantaneity of LFPE by comparing with Canny [16], Otsu [17], Optical Flow [18] and LFSW. The test data uses the same sample (4230 frames of effective video on a normal road). The LFSW and LFPF is more accurate, because the binary image of the lane line obtained by Multi-Feature fusion algorithm is clear, and pixel points found by window searching are more accurate. The Canny algorithm is more time-consuming because of the repetitive adoption of the Hough transform for line calculation. Therefore, LFSW and LFPE outperforms all the other alternative methods.
Through statistical comparison of the experimental results in Table 2, the LFSW are superior to the Canny, Optical Flow and Otsu, and has highest accuracy with 96.68%. Since the multi-feature fusion algorithm is used to obtain a high-quality binary image of lane-lines, and the feature extraction window is performed a refined lane-line pixel search. Moreover, the results from Table 2 show that the accuracy of the LFPF is 2.33% lower than that of the LFSW, but the speed is improved by 14.2ms. Because the lane-line pixel search is limited in the neighborhood of the lane-lines detected before.

Conclusion
In In addition, the algorithm is compared with three typical algorithms -The Canny, Otsu, and Optical flow algorithms in terms of lane detection results. Several sections of freeways selected for testing, and statistics show that the algorithm proposed in this paper is better than the three conventional algorithms in terms of detection rate and reliability, the accuracy of the proposed algorithm is 96.68%. At the same time, it is less time-consuming, has good timeliness, the average time for the proposed method to process a single frame of image is 20.7ms, and satisfies the motion planning for unmanned vehicle systems. Although the algorithm proposed in this paper can achieve better results, some problems remain to be addressed. For instance, when a road surface takes an undulating form and fusion declines. In certain particular sections such as long-distance tunnels and others, the lane detection and tracking are weakened. In addition, the proposed method has a problem in detecting lane in sharp curves and in the presence of adjacent vehicles and pedestrians. In future research, multiple road features will be further combined with more sensor information to detect the sharply curved lanes amidst the presence of vehicles and enhance the flexibility and robustness of the proposed algorithm.

Availability of data and materials
We can provide the data.