Optimal region gradated fusion based automatic image stitching for industrial systems

In this paper, an improved image stitching method based on image fusion is presented, and the purpose is to use multi-equipments and image stitching algorithm to obtain the complete image of a large-scale production line for industrial systems. Focus on the problems in the existing image stitching algorithm, an improved fusion algorithm is proposed. Firstly, the optimal seam is found, and then the fusion range is limited. Finally, the gradated in and out fusion algorithm is used to perform fusion calculation within the limited fusion range to complete image stitching. Compared with the existing fusion algorithms, the experimental results based on different image fusion indexes show that the method in this paper solves the problem of unnatural image stitching effect, enhances the image stitching result, and has the great fusion effect.


I. INTRODUCTION
Monitoring the parts of large-scale systems can simplify the management of large-scale facilities and ensure safety. At present, detection technology is developing towards online detection and non-contact detection [1]. Image detection technology is widely used in industrial production. However, the captured image can only be taken at a limited angle due to the limitation of the field of view of a single camera. There is often large equipment in the factory, which makes it difficult to use only one camera to record the complete production line. Thus, using multi-equipment and image stitching algorithm to obtain the complete image of a large-scale production line becomes particularly important.
Image stitching refers to a series of combining a series of small-view and low-resolution images with overlapping areas to generate a new panoramic image with high-resolution and large-view [2]. This panoramic image contains the information of all previous series of images. Therefore, we can use image stitching to obtain a panoramic view of the production environment, and use the panorama for the computer vision processing part to detect.
There are two key steps in image stitching: image registration and image fusion.
Image registration is a process of spatially aligning the image with the overlapping area. In 1998, Capel proposed Random Sample Consensus (RANSAC) [3] to purify the matching points in image registration. In 2004, Lowe proposed the Scale Invariant Feature Transformation algorithm(SIFT) [4]. SIFT has scale and rotation invariance and is robust to light intensity and noise, which is still widely used in image stitching technology. In 2007, Lowe used the global homography matrix [5] to complete image registration. However, the global method can't get excellent registration accuracy in the case of big parallax. As Projective As Possible(APAP) [6] is proposed to align the image with partial blocks of the image to improve the accuracy of image registration. But due to the local homography matrix transformation in the overlapping area, the image caused significant projective distortion in the non-overlapping area.
In the overlapping area, there are more stitching dislocations, resulting in obvious ghosting. Image fusion [7] is used to reduce the stitching seam of overlapping areas of images to achieve natural visual effects, ito achieve natural visual effects, including multi-band fusion and gradient template weighted fusion [8]. Reference [9] uses the difference image to calculate the weight. Although the virtual shadow problem is eliminated to a certain extent, it also makes the stitched image produce a certain deformation. A fast color fusion algorithm is proposed in [10], which can eliminate all color inconsistencies in prealigned images in certain circumstances. Image stitching technology [11] [12] directly determines the quality of the panorama. In this paper, the principle of image stitching is followed, performing sharpening of the image in the registration stage, using transition smoothing on the selected area around the optimal stitching seam to complete image fusion.
The remainder of this paper is organized as follows: The process of the new image fusion algorithm of the image stitching method is introduced in section II. In Section III, the experimental results and analysis are expressed. In Section IV, the conclusion is expressed.

II. PANORAMIC IMAGE STITCHING METHOD
A. Stitching process The basic process of image stitching can be divided into three parts: Image acquisition, image registration, and image fusion. After the images are obtained by the camera, first preprocess multiple groups of images, then register the images, and finally fuse the image overlapping areas to make the generated panorama looks more natural. Among them, image registration and image fusion are the most critical steps.
The purpose of image registration is to establish the geometric correspondence between images. It is a registration process to align the overlapping regions of two images by extracting and matching the feature points in the images. The common feature extraction algorithms include Harris corner detection algorithm, sift algorithm, surf algorithm, and orb algorithm. Among them, sift algorithm maintains certain stability for image rotation, scale scaling, brightness change, and projective change.
The purpose of image fusion is to improve the stitching effect. After the multiple images with overlapping areas are stitched together, the overlapping areas of the stitching will produce blur and ghosting. The purpose of image fusion is to achieve a smooth transition of the stitched overlap areas while keeping the information of the original image as much as possible to avoid information loss.

B. Perspective transformation image distortion processing
The stitching algorithm proposed in this paper uses the sift algorithm to extract features of the target image and the reference image respectively, and establish a set of feature description points. The feature descriptor subset is obtained from the target image and the reference image, including the location, direction, and size of the feature points. Then use the KNN matching algorithm to perform feature point matching to obtain the set C, and use the matching pair filtered by Lowe's algorithm on the set C as the result of the initial matching. The RANSAC algorithm is used to remove the external points and retain the internal points for accurate filtering. Then, estimation of homography and warping is applied respectively to mosaic the images. The registered image after the warping has caused the original image to be stretched and other changes. The image is distorted and looks blurred visually.
In view of this, this article sharpens the transformed view image. According to the Laplace operator, the template coef- This matrix can add the negative number of the value of the four neighboring points of the center pixel, plus five times the value of the center pixel, which can highlight the location of the sudden change in the brightness value to make the edges, contours, and details of the image clear. The effect of enhancing the image. After the sharpening process of the convolution operation is performed on the image, it can be seen from the projective that the clarity of the processed view image is highlighted, and the blurred details are enhanced, ultimately improving the quality of the stitched image.

C. Optimal region gradated image fusion
Efros and Freeman [13] proposed to use dynamic programming to solve the optimal seam in the image overlapping area. The optimal seam based on the idea of dynamic programming is an optimization algorithm, which can reduce the stitching phenomenon caused by the gray difference between images. The stitching line should meet the following two points in terms of color intensity and geometric structure [14] : • From the point of color intensity, the difference in color between the two images is the smallest.
• From the point of view of geometric structure, the structural difference between the two images is the smallest. The calculation criterion of the optimal seam is expressed as follows: Among them, E color (x, y) is the value of image color difference, and E geometry (x, y) is the intensity value of image structure difference; The solution criterion of E geometry (x, y) is as follows: among them, Where S x and S y denotes the template of 3 × 3 Sobel operators in X and Y directions respectively. In the overlapping region, the first row of pixels in the image is taken as the starting point to establish multiple stitching and the one with the least color intensity and structural strength is found as the optimal seam. Fig. 1 shows the process of the optimal seam search based on dynamic programming. Fig. 1(a) is a set of initial stitches and each pixel in the first row is taken as the seam starting point. Fig. 1(b) shows the situation when the seam starts to expand downward, considering the points of each next row adjacent to it. The point with the minimum criterion value is the extension point. As shown in Fig. 1(d), finally, among the seam obtained as shown in Fig. 1(c), the seam with the minimum criterion value is selected as the optimal seam.
In the process of image acquisition by different devices, due to the influence of many external factors, it is easy to cause differences in brightness and tone between images, which will directly affect the quality of the final generated panorama. Therefore, after the calculation of the optimal suture line, we also need to combine the image fusion technology to adjust the image coincidence area to reduce the color difference after the merging of the two areas. In this paper, the optimal seam obtained by the above method is used to fuse two images by selecting regions on both sides of the optimal seam according to the weight. The gradated in and out image algorithm is used in this paper to calculate the fusion on both sides of the seam, which can eliminate the mosaic effect caused by the difference of color brightness to a certain extent. The gradated in and out algorithm uses the weighted coefficient function, and the weight is gradually changed in the process of fusion to obtain smooth fusion results. The calculation formula of fused image I (x, y) is as follows: Among them, ω 1 and ω 2 are the corresponding weighted coefficients. The calculation of ω 1 , ω 2 follows equation: where x i is the abscissa of the current pixel, and x l is the left boundary of the overlapping region, x r is the right boundary of the overlapping region. Figure 2 shows the range of the fusion region determined within the overlapping region of the image to be spliced. Specifically, when calculating the pixels, the fusion range is determined as follows: • The left boundary start and right boundary end with a minimum circumscribed matrix of the optimal seam are calculated and the limited fusion region D is determined according to the position of the optimal seam.
• Take the current each line seams point x, compare the left and right distance between the points and boundaries. Select the shortest distance between the current point of the seam and the two boundaries as the limited range of current line fusion, and the limited boundary is [x-d, x+d].
• By traversing, the range of fusion region D is determined according to the d of each row. Basically, it is an irregular region.
In the limited fusion range, the gradated in and out fusion algorithm is used to calculate as shown in Algorithm 1. Specifically, in the determined fusion region D, the pixel values of the images to be fused A and B are set as I 1 (x, y) and end if 7: end for 8: zone D={x(i)-d(i),x(i)+d(i)}; 9: for each I(i,j) ∈ D do I(i,j)=ω 1 I 1 (i, j)+ω 2 I 2 (i, j); 10: end for 11: return panorama III. EXPERIMENTAL RESULTS AND ANALYSIS The experimental environment is as follows: Intel core i7-7700/3.60GHz CPU, 16GB memory, 64-bit Windows 10 operating system, the programming language is Python, running in PyCharm 2020.

A. Image data source
The image stitching dataset in the experiment to verify the effectiveness of the algorithm is provided by [15], taken by a drone (KC1600) in Hejiang East Village, China. Figure 3(a) shows the result of direct stitching of remote sensing images. It can be seen that due to the difference in gray scale of remote sensing images, there is an obvious stitching seam in directly stitching images (marked with red circles in the figure). Figure 3(b), Figure 3(c) and Figure 3(d) show that directly based on the optimal seam stitching algorithm, the directly gradated in and out fusion algorithm and Laplace algorithm can solve the color difference between the overlapping areas around the stitching seam. However, it can also be seen that the stitched images finally obtained by the three existing algorithms have black areas on the edges, and there are also black lines that impact the visual effect (marked with red circles in the figure). Because the image stitching is carried out directly using the optimal seam and Laplace fusion algorithm, the pixel-level fusion processing of the reconverged area is not carried out, resulting in the black area of the edge increasing, impacting the visual effect and also affecting the image information. In view of this, an image inpainting process is used to recover image information from adjacent areas on the edges.

B. Assessment methods of different fusion algorithms
Reducing the impact of distortion problems caused by image transformation by sharpening processing. The final stitching result is shown in Figure 3 (e). It can be seen that the fusion image is not only clearer but also more uniform in transition, contains more image information, and can achieve a good visual effect.
In order to evaluate the quality of the fusion image more objectively, the experimental results of this paper are based on different fusion image quality evaluation indicators to evaluate and compare the fusion results, and then verify the effectiveness of the algorithm in this paper. The evaluation indicators include standard deviation, spatial frequency, average gradient, entropy, mutual information. Among them, standard deviation, spatial frequency, and average gradient are indicators based on statistical evaluation, information entropy and mutual information are indicators based on information entropy.
The standard deviation (STD) reflects the degree of dispersion between the image pixels value and the mean value, and it's an objective evaluation index to measure the richness of image information. The larger value scattered the gray level distribution of the image, the more information image carries the better the quality fusion image gets. The calculation formula is as follows: The spatial frequency (SF) reflects the image gray level change rate. The larger the value of spatial frequency, the clearer image, and the better the quality fused image. The calculation formula is as follows: among them, The average gradient (AG) is used to measure the clarity of the fusion image. The larger the value of average gradient, the better image clarity, and better fusion quality. The calculation formula is as follows: among them, Entropy (EN) is an objective evaluation index that measures how much information an image contains. The larger the value of entropy, the richer information and better the quality of the fused image. The calculation formula is as follows: Where a is the gray value, and P A (a) is the gray probability distribution.
Mutual Information (MI) can measure by the degree of similarity between the fused image and the original image, that is, how much information the fused image has obtained from the original image. The larger the value of mutual information, the more fused image retains source image information and better quality. Mutual information is calculated based on the information entropy H(A) and joint information entropy  H(A, B) of the image. The formula for calculating mutual information is as follows:

C. Experimental results of different fusion algorithm
The remote sensing image dataset taken by the drone (KC1600) in Hejiangdong village is stitched and uses the five above-mentioned indicators to evaluate image quality to compare the effects of different fusion algorithms. In order to better distinguish and analyze multiple sets of data, firstly, according to the different content contained in the dataset, that is, whether the remote sensing image dataset contains houses, roads, rivers, or only woods, all dataset is divided into 4 categories. Among them, there are 87 result images for houses, 34 result images for roads, 15 result images for rivers, and 36 result images for forests, for a total of 172 result images. These results are obtained by different fusion algorithms, including the directly based on the optimal seam algorithm, the directly gradated in and out fusion algorithm, the Laplace algorithm, and an algorithm based on the optimal seam gradated in and out of fusion proposed in this paper. Based on these stitched images, calculate the values of the five above-mentioned indicators for objective comparison. Finally, obtained the data index values are classified according to data set categories, and each category takes the average value of the corresponding data for analysis and comparison, as shown in Table I. In order to compare the fusion effects on different algorithms more intuitively, the data in Table I is drawn as a histogram as shown in Figure 4.
From the distribution of standard deviation in Figure 4, the standard deviation value of the image with house and road information is relatively high, that is, considering the degree of deviation of the pixel gray from the mean value, the image content is richer. In this paper, some processing is done to solve the problem of image distortion based on the optimal seam and gradated in and out algorithm after optimization. Convolution is performed on the image, which makes the pixel value fluctuate more than the average value, indicating that the image contains richer information. Spatial frequency (SF) and average gradient (AG), as evaluation indicators based on statistical characteristics, can be used to measure the clarity of an image. According to the distribution of spatial frequency and average gradient shown in Figure 4, the spatial frequency and average gradient of the proposed fusion algorithm in this paper are four times those of the other three existing fusion algorithms. From the calculation formula of spatial frequency and average gradient given above, we can know that both indicators are calculated based on the difference of adjacent gray values in the image. Figure 5 shows the partial gray values of a certain group of data in the dataset processed by four different fusion algorithms in the vicinity of the stitching area. It can be seen that the difference between adjacent gray values of the fusion result obtained by the algorithm in this paper fluctuates the most. Therefore, the spatial frequency and average gradient index values of the resulting image are much larger than the index values of the existing fusion algorithm that have not solved the image distortion problem, resulting in the histogram distribution of the spatial frequency and average gradient as shown in Figure 4. This shows that the sharpness of the stitched image obtained by the algorithm in this paper has been improved, and the blurred part of the image details is enhanced, which is more conducive to subsequent image processing.
Judging from the distribution of entropy shown in Figure 4, the entropy value of the image containing forest and road information is the highest in comparison, that is, considering the gray distribution of image pixels, the image content is richer. At the same time, it can be seen from the figure that the entropy values of the results of the existing three fusion algorithms are relatively close, and the entropy value of the splicing processing fusion algorithm proposed in this paper is higher than other three algorithms, indicating that the pixels value of the image obtained by the algorithm of this paper have a wide distribution of gray levels, and the information in the image is more abundant.
Since entropy has nothing to do with the source image, it can only explain the size of the information of the evaluated image, and cannot fully prove the degree of information retention of the stitched image compared to the source image. Qu proposed to use mutual information (MI) to evaluate the quality of image fusion [16]. In the calculation, only the fusion result map and the source image A and the source image B are separately calculated for mutual information, and then the average weights of the two are added together to obtain the mutual information index values shown in Table I. Observed from the distribution of the value of mutual information shown in Figure 4, due to the large overlapping area of adjacent remote sensing data, each fusion algorithm retains the information of the source image better, and the mutual information index values are very close. Image fusion algorithm is widely studied by many scholars. An improved fusion algorithm is proposed in [17]. At the same time, it is also objectively quantified and evaluated with the mosaic image under the existing fusion algorithm mentioned above, and the calculated index results are shown in Table II.
We can calculate from the data in the Table I and Table II. Based on the index of entropy, the fusion algorithm in [17] is improved by 0.55% compared to the gradated in and out fusion  algorithm, and 1.4% compared with the Laplace algorithm. Our algorithm is 10.6% higher than the gradated in and out fusion algorithm, and 11.64% higher than the Laplace algorithm. Based on Avg Gradient, the fusion algorithm in [17] is 10.81% higher than the gradated in and out fusion algorithm, and 19.43% higher than the Laplace algorithm. Our algorithm is 72.18% higher than the gradated in and out fusion algorithm, and 74.15% higher than the Laplace algorithm. Based on STD, the fusion algorithm in [17] is 0.04% higher than the gradated in and out fusion algorithm, and 3.76% higher than the Laplace algorithm. Our algorithm is 21.26% higher than the gradated in and out fusion algorithm and 19.9% higher than the Laplace algorithm.
For the fusion algorithm proposed in this paper, we considered the image distortion and the loss of image edge information due to image transformation. Through image convolution processing and image inpainting process make clarity of the resulting image is improved. The degree of retention of the source image information has been greatly improved while preserving high-frequency details. And an improved fusion algorithm based on the optimal seam in this paper can improve the quality of the image stitching result.  IV. CONCLUSION In this paper, based on the research of fusion method in image stitching technology, in order to solve the problem of stitching seam caused by difference of gray level of images, an improved fusion algorithm is proposed, which realizes the smooth transition of image overlapping area and effectively eliminates the stitching gap. The key technology of image stitch technology is studied in this paper. Analyze and compare related algorithms, and make subjective comparisons and objective evaluations with different fusion algorithms. Through the experimental result, it is proved that the algorithm proposed in this paper has obvious advantages compared with the directly based on the optimal seam algorithm, the directly gradated in and out fusion algorithm, and the Laplace algorithm. It can effectively eliminate the seam and make the fused image clearer and have a better visual effect.
The algorithm proposed in this paper can be applied to factory visual inspection. In the process of the actual production line, the position of the workpiece may not be determined. The panorama created by the image stitching algorithm can determine the position of the workpiece and extract more detailed information. As shown in Figure 6(a), in one image, the workpiece is not completely photographed, which will affect the subsequent visual detection. Therefore, the image feature points are extracted and matched for the two images. The resulting image of feature point matching is shown in Figure 6(b), and the resulting image not fused is shown in Figure 6(c). After using the image fusion algorithm in this paper, a complete and clear panorama is obtained as shown in Figure 6(d). Instead of a small view image, a wide-angle clear panorama can be processed by visual detection.