A high-dynamic-range visual sensing method for feature extraction of welding pool based on adaptive image fusion

The high dynamic range existing in arc welding with high energy density challenges most of the industrial cameras, causing badly exposed pixels in the captured images and bringing difficulty to the feature detection from internal weld pool. This paper proposes a novel monitoring method called adaptive image fusion, which increases the amount of information contained in the welding images and can be realized on the common industrial camera with low cost. It combines original images captured rapidly by the camera into one fused image, and the setting of these images is based on the real-time analysis of realistic scene irradiance during the welding process. Experiments are carried out to find out the operating window for the adaptive image fusion method, providing the rules for getting a fused image with as much as information as possible. The comparison between the imaging with or without the proposed method proves that the fused image has a wider dynamic range and includes more useful features from the weld pool. The improvement is also verified by extracting both the internal and external features of weld pool within a same fused image with proposed method. The results show that the proposed method can adaptively expand the dynamic range of visual monitoring system with low cost, which benefits the feature extraction from the internal weld pool.


Introduction
High energy density welding has a wide application in the fields of industrial manufacturing because of its high efficiency [1]. The high heat input of the welding torch enables the application of a higher travelling speed on thicker weldment during welding process. As a variant of the traditional TIG welding, keyhole TIG welding achieves a high energy input with large current by using torch cooling system. It features with its singlepass welding of medium or high thickness metal weldments [2]. However, the arc with highly concentrated energy releases a huge number of photons in the welding scene. The welding region under the arc has high brightness because of the heat radiance and the reflection of arc light from the metal surface [3]. It brings challenges in visual monitoring of the molten pool, causing over-exposed or under-exposed pixels in the images. In order to solve this problem, some researchers applied advanced digital cameras to obtain well-exposed images. Xia et al. used a Xiris XVC-1000e camera to get clear images of the weld pool from the back side. But the head and tail of the weld pool could not be captured clearly at the same time [4]. Wang et al. applied NIT MC1003-1VB camera into getting a high-dynamic-range image of welding process [5]. Bakir et al. put an s-CMOS camera on a focusing optical head of a laser for a better texture recognition of the welding [6]. De, A. et al. located the weld seam ahead of the welding torch with images sensed by a simple CCD monochrome camera [7]. However, these cameras contain expensive photosensitive elements which increase the cost of the visual system setting up. Besides, the existing circuit settings in the imaging element determine that its imaging strategy cannot be changed. It results in a fixed camera dynamic range which is unable to be adjusted with the brightness changing in the scene. Some researchers used optical systems composed of extra light source with specific frequency and narrow filters. Luo M et al. used a laser and a green light to light up the weld region and took pictures through a filter for width detection of molten pool [8]. Shao et al. used three colors of laser to capture and process a single image so that the three dimensional positions of the weld seam can be obtained simultaneously [9]. But this method requires series of previous experiments to decide the choice of light sources and filters, and would need repeating adjustments once the welding scene changes with the operating parameters. In addition, the inner features of the weld pool are hardly recorded with the methods above due to the extremely high weld pool brightness [10,11]. As a result, the visual monitoring and detection in keyhole TIG welding become difficult because there is a high contrast in brightness as well as high oscillation within the weld pool. The welding quality is closely related to the internal and external geometry of the weld pool, so obtaining a panoramic view of the weld pool is crucial.
In this paper, a novel method called adaptive image fusion is proposed to solve the monitoring and detecting problem in keyhole TIG welding. It enables the low-cost camera to adaptively capture clear images of high-dynamic-range welding scene, realizing the complete features extraction of the weld pool. Section 2 analyzes the dynamic range of keyhole TIG welding by scene irradiance measurement, explaining the essential reason for the existing monitoring problem. Section 3.1 designs an adaptive imaging strategy based on the real-time relative irradiance detection. Based on this, Section 3.2 introduces a novel image fusion algorithm to obtain images of keyhole TIG welding of high dynamic range. Besides, tests are also carried out to determine the operating window for the parameter configuration in image fusion. Then, Section 4 realizes the detection of the overall geometry of weld pool including its external profile and the size of the keyhole entrance inside the weld pool. Finally, Section 5 concludes the performance of the proposed monitoring method, showing that the image fusion algorithm can be adaptive to high-dynamic-range keyhole TIG welding scene and can be performed on economic cameras.

Dynamic range of the keyhole TIG welding scene
The plasma arc during keyhole TIG welding process produces strong light and high heat input. Therefore, the weld pool under the arc emits great amount of light due to its heat radiance as well as the arc light reflected from the pool surface [12,13]. But the brightness of the light decreases rapidly in the direction away from the arc. It brings high contrast between different parts of the weld pool, causing the difficulty in clear image capturing during welding process. Therefore, having a complete knowledge of the brightness range of the keyhole TIG welding scene does help to design suitable strategy for visual monitoring.
In this section, dynamic range is used to describe the brightness range between the darkest and brightest parts within a realistic scene or an image. It shows the irradiance range of a scene or the ability of an imaging system to record the realistic scene [14]. Experiments are carried out on the keyhole TIG welding with different kinds of CMOS cameras. The dynamic ranges of both the welding scene and the cameras are obtained for comparison. Figure 1 illustrates the process of the measurement of the dynamic range of the welding scene. A method of calibration was used to reconstruct response function of the camera by taking weld pool images with various exposure times. First, the calibration for the response property of the camera is taken. It is described as a formula showing the relationship among the pixel value Z, exposure time Δt, and the irradiance L in reference [15]: where g(Z) is the response function of the camera and A represents the proportionality factor, which is a constant and determined by the camera installation and the use of the filter. Then several welding images of the same keyhole TIG welding scene are taken with various exposure times. The next step is to get the relative irradiance map from each image captured before via formula (1). Every relative irradiance map records a part of the brightness from the welding scene. Finally, by combining these maps, a complete relative irradiance map of the keyhole TIG welding scene can be obtained. From the map, the maximum and minimum relative irradiances are taken to calculate the dynamic range (DR) of the scene through the following formula: where L max and L min stand for the maximum and minimum irradiance of the welding scene respectively. The dynamic ranges of the keyhole TIG welding scene with different currents are listed in Figure 2. The figure also includes the dynamic range of the commonly used CMOS cameras in welding monitoring as well as the camera used in this study [16]. The result shows that for the commonly used CMOS camera, their dynamic range is not wide enough to cover the whole keyhole TIG welding scene. That is the reason for why the common cameras could not record complete information from the welding scene. Besides, the result also shows that the dynamic range of welding scene will change as the welding parameters change while the dynamic range of the camera is fixed. As a result, a camera with a much larger dynamic range must be used to ensure image quality, which increases unnecessary cost. In order to solve this problem, an adaptive method is needed to improve dynamic range of the image during keyhole TIG welding monitoring with low cost.

Adaptive image fusion in welding monitoring
In this section, an adaptive image fusion algorithm is proposed to improve the dynamic range of the images of keyhole TIG welding. First, a set of images will be captured periodically with various exposure times following an irradiance-based imaging rule. Then, an image fusion algorithm is used to combine these images to obtain a new image which covers the whole dynamic range of the welding scene. Finally, the influence of the camera characteristic on the performance of the proposed algorithm is analyzed through experiments. The operating window for choosing the optimal parameters of the camera is also concluded. In this way, the dynamic range of the visual system can be widened without updating the camera electronics. The quality of the new image can also be guaranteed by adjusting all the exposure time of the images in real time.

Irradiance-based imaging of weld pool
According to the irradiance map shown in Figure 1, different parts of the weld pool have different dynamic ranges. To be specific, the keyhole inside the head of the weld pool has a highest brightness, while there is a decline in irradiance from  The process of the measurement of the dynamic range of the welding scene the head to the tail of the weld pool. The automatic gain control (AGC) strategy is commonly used in camera imaging, which depends on the statistic analysis of the pixel histogram instead of the irradiance distribution [17]. It is not suitable to record the details inside the weld pool because it has high irradiance. In addition, the way that camera transfers the irradiance to the pixel value causes problems. The response function is a logarithmic curve describing this transforming, as shown in Figure 3. On one hand, the bright region of the weld pool corresponds to the blue part of the curve, which is very flat. It means that only a small portion pixel inside the weld pool can be well-exposed. On the other hand, the dark region of the weld pool corresponds to the green part of the curve, which is steep. This means that a small increase in the pixel value will cause a large change in radiance. It will lead to error of calculation. As a result, the propriate portion of the curve for mapping is the middle one that is not too flat nor steep. In this paper, this portion of curve is called the optical range for imaging (ORI), as shown in red window in Figure 3. The maximum pixel value within ORI is represented as Z top , while the minimum one is represented as Z bot . Their corresponding relative irradiances are L top and L bot , respectively. The dynamic range of the ORI is referred as optical dynamic range (ODR), represented as η. It can be got through formula (2). ODR is a key parameter showing the ability of a camera to capture high-dynamic-range images. ODR differs for various kinds of cameras. The overexposing and low-exposing can be avoided if the whole dynamic range is divided into several parts and each part is mapped within ORI. In this way, a complete information of weld pool can be obtained by the camera. This can be realized by the irradiance-based imaging algorithm proposed in this paper. It takes images of the welding scene by changing exposure time. The exposure time can be represented as a set {Δt 1 , …, Δt N }. Every exposure time Δt k from the set is used to capture one image respectively. The order will be from Δt 1 to Δt N , then loop. k ∈ [1, N], and N is the number of the exposure times. The set of exposure time {Δt 1 , …, Δt N } is decided in advance based on the welding scene irradiance. It starts by finding the first exposure time Δt 1 : (a) Setting the exposure time ΔT (k) = Δt min . Δt min is the minimum exposure time that the camera can set. At first, k = 1. can determined by the following: The determination of the rest exposure time in {Δt 1 , …, Δt N } obeys the following rules. As mentioned above, the exposure time {Δt 1 , …, Δt N } is used to capture images, and every image records a part of the welding information within the ORI of the camera, as shown in Figure 5. As a result, every image has its own range of relative irradiance L k which covers Fig. 3 Analysis of the different portions of the response function curve of the camera a certain region of the whole welding scene irradiance. For example, assuming that the whole welding scene irradiance is from L 2 bot to L 1 bot , then only two images are needed to cover the whole range: the first image is captured with an exposure time of Δt 1 , and its corresponding irradiance is from L 1 bot to L 1 top . The second image shows the scene irradiance range is within L 2 bot to L 2 top , while its exposure time is Δt 2 . Then while combining these two images, the summed up irradiance is equal to that of welding scene. In fact, according to the analysis in Section 2, the edge range of irradiance within one image is not suitable for using. As a result, if more images are used and their irradiance ranges are combined together by overlapping with each other, then the summed up dynamic range can be as wide as that of the welding scene. The overlapping law satisfies that the bottom limit of the first region is set to be equal to the top limit of the next region: Where L k bot is the minimum irradiance of the ORI of the k th image while L kþ1 top is the maximum irradiance of the ORI of the (k + 1) th image. Therefore, every image is captured within the ORI of the camera and avoids the use of the range out of ORI. For the k th image and the (k + 1) th image, there exists: By combining formulas (4) and (5), the relationship is established between two adjacent exposure times, Δt k and Δt k + 1 : By applying formulas (3) to (6), {Δt 1 , …, Δt K } can be obtained, as shown in Figure 6. Because Δt 1 makes sure that the brightest area in the image is well-exposed and Δt 1 < Δt 2 … < Δt K , the whole region avoids over-exposed. By using this method, the camera can be adaptive to the dynamic range change of the welding scene, avoiding the imaging problem countered with AGC strategy.

Welding image fusion
Although a wider dynamic range is obtained by taking a set of images with irradiance-based imaging algorithm, the features of the weld pool seem discontinued from the images. It means parts of a feature were shown in different brightness in different images. It brings difficulties in welding monitoring as well as the feature automatic detection. In this part, an image fusion method is used to reconstruct a high-dynamic-range image, avoiding disconnected and unnatural artifacts. The process of the image fusion is shown in Figure 7. For the k th image captured by irradiance-based imaging algorithm, the pixel value at the position of (i, j) is referred to Z(i, j). i is the x axis coordinate and j is y axis coordinate in the image coordinate. First, the gradient map of the image is obtained by the following: Where G k (i, j) represents the total gradient vector at the position of (i, j). G k x i; j ð Þ and G k y i; j ð Þ are the gradient vectors in the direction of x axis and y axis respectively. Because the pixel value matrix can be seen as a discrete domain, the moduli of G k x i; j ð Þ and G k y i; j ð Þ can be given by the following: Every G k (i, j) shows the information of the edges in the images, so the higher it is, the more features are around this pixel. Therefore, at every pixel (i, j), the maximum G k (i, j) is selected from k images and referred as G f (i, j): where A k is the weight corresponding to the index of image. It makes the pixels with higher brightness have a greater weight while reconstructing the final gradient map G f . In this study, A 1 = 1.2, A 2 = 1.1, A 3 = 1.0, A 4 = 1.0. G f becomes a new gradient map which contains the maximum feature information. It can be seen as the gradient map of a new image Z f , which is the pixel value of the fused image. Then the function between the new gradient and the pixel value at every point of the welding scene is described as follows: In order to solve for Z f , the formula (10) is changed into a form of Poisson equation: It aims to avoid the calculation problem caused by the unconservative of the field of gradient [19]. In this paper, a method based on the Haar wavelets [20] is used which relates the Haar wavelet decomposition coefficients of Z f and the fused gradient G f . Finally, an iteration form to get the fused image can be described as follows [21]: where n is the number of the iteration and ⨂ is the convolution operator.
By setting a factor, the iteration will stop when pixel value mean of the fused image is closed to this factor. The lowest image in Figure 7 is the final image got by image fusion. It contains more information of the weld pool, including the internal feature like keyhole and the external profile of the pool.

Operating window for optimal image fusion
The adaptive image fusion can greatly increase the dynamic range of the images captured by the camera. However, the performance of the algorithm depends on two factors. One is the ODR of the camera. According to Section 3.1, the final dynamic range of the new image is a multiple of ODR, which means the higher ODR of the camera, the better performance of the algorithm will have. The other one is the minimum exposure time Δt min that the camera can set. During the image capturing, the total time cost for a fused time is equal to the sum of all the exposure times. The next expose time is several times of the former one. Therefore, the value of the Δt min will affect the fluency and image quality of the video stream. Experiments and derivation are carried out in this section to determine the operating window for optimal image fusion. Assuming that Δt 1 = t min , so that the total time cost will be as low as possible. Let η represents the value of ODR. The Δt k and the total time cost T total can be described as follows: where K is the number of images used for fusion. In order to ensure the fluency of the video stream, there is a frequency threshold f v , otherwise the video will be choppy: In this study, f v is set to be 10 Hz, and the camera has a η of 2.6 dB and a t min of 38 μs. By adding the number of images used for image fusion to up to its limit (K max = 4), the total dynamic range of the fused image increases linearly and reaches its maximum, as shown as the third curve in Figure 8a. In order to capture the keyhole TIG welding process, the dynamic range of fused image has to be higher than that of welding scene. According to the measuring method mentioned in reference [22], the welding dynamic range is found to be 8.3 dB. Therefore, the law of choosing the propriate number of images is K = K max .
According to formula (15), the changes of t min and η will have impact on the total dynamic range and K max . Experiments were taken to find out an operating window for the setting of these two factors. When the t min is set to be constant, η is adjusted to see the changes, as shown in Figure 8a. As the η increases, the slope of the curve increases as well, making it easier to get a wider dynamic range with a smaller number of images. But if the η is too large or too small, the total dynamic range decreases. As a result, η should be chosen within a range. When the t min is changed, the  Figure 8b. On one hand, if the t min is too large, it is impossible to capture keyhole TIG welding because the curve is under the welding scene line. On the other hand, if the t min is small enough, the area under the curve and above the welding scene line will be appropriate for the optimal parameter configuration. In conclusion, the number of the images used for image fusion should be as many as possible to widen the dynamic range. Besides, the rise of η can increase the K max , but too large η will do harm to the dynamic range widening. There is an optimal choice of η which maximizes the dynamic range. Finally, the choice of the t min should be small in order to get a high dynamic range.

Internal and external feature detection of the weld pool
The detection of the features from the welding pool is usually uncompleted because of the narrow-dynamic range of the monitoring system. Once a part of the features is obtained, the other parts will be badly exposed in the image. It causes the lack of the information when the welding monitoring is carried out, resulting in errors in features detection. Therefore, the simultaneous detection of both the internal and external feature is valuable. It enables the geometry extraction of the whole weld pool instead of only a portion of it. A complete detection of features of the weld pool can be realized based on Fig. 9 Extraction of the internal and external features of the weld pool based on adaptive image fusion the images obtained by the adaptive image fusion algorithm. In this section, the width of the keyhole entry is detected, and the edge fitting of the weld pool head and its tail is also realized. It does help to the complete information detection of the weld pool during the keyhole TIG welding, solving the difficulty in the internal monitoring, as shown in Figure 9.
The first step is to detect the profile of the whole weld pool. By applying the Sobel operator [23], the image is turned into gradient field. Then the interested region is cropped out before applying binarization based on statistic of the gradience value. Geometric processing is used to delete the isolated region and preserve the useful edge. Finally, the edges are refined to get the edges. The second step is to detect three pairs of typical point for the pool head fitting. The first pair of points are the two pixels which the edges intersect with the top boundary of the image. The second pair of points is chosen as the interaction of the edge of the head of pool and the tail of pool. The third pair of points is on the edge of the tail of pool, and the horizontal distance between these two reaches its maximum. The first two pairs of points are used to detect the boundary of the head of pool, while the first and third pairs of points are used to detect the boundary of the tail of pool. After that, the boundary of the head and tail of pool is fitted. The third step is the detection of the keyhole entry by applying image processing algorithm within the head of pool. A region from the gradient map is cropped out and this region contains the features of the weld pool head. Based on this, the edge extraction and curve fitting process can be realized. After the process mentioned above, the key features from the weld pool are completely extracted, including the profile of the whole weld pool, outline of the weld pool head, and the size of the keyhole entry. These can be realized through an image on a single visual sensor, improving the accuracy and efficiency of the detection of the weld pool features.

Results and analysis
Comparison of the imaging quality is made on the CMOS camera between those with and without adaptive image fusion. Experiments are carried out twice with the same welding parameters, which ensure the weld pool geometry can be assumed to be the same. The camera is used to capture images of keyhole TIG welding process. One is with the proposed algorithm and the other is without. The distribution of the brightness of the pixels in both images is analyzed in Figure 10. The image without algorithm records the welding pool with less information. A lot of pixels exceed its brightness value (reaching 255) inside the weld pool head. In addition, the weld pool tail keeps low brightness, resulting in difficulty in the weld pool tail edge recognition, as shown in Figure 10a. By comparison, the image with fusion algorithm shows more diversity of the brightness, having more different values within the weld pool head while the weld pool tail remains a high level of brightness, which helps to detect the edges, as shown in Figure 10b. In order to further discuss the performance of the proposed algorithm, three lines are chosen from the image. They are 0 mm, 15 mm, and 26 mm away from the position of the torch tip. All the pixels on these three lines correspond to three different positions on the weld pool. Because the camera is calibrated for its position, the same position of lines can be selected from the image from two experiments with or without proposed image fusion algorithm, as shown in Figure 11a, b. The pixel values on these three lines are extracted to see the changes of pixel value along the row. For the line under the torch tip, it goes through the region of the keyhole inside the weld pool, as shown in Figure 11c. From the image without the algorithm, many pixels reaches the maximum pixel value, unable to show value change in the image. By contrast, the fused image which is processed with the algorithm contains more step changes in terms of the pixel value, which shows the details Fig. 10 The comparison of the image pixel brightness histogram. a Image without the algorithm. b Image with the algorithm from the keyhole. For the line of 15 mm away from the torch tip, both two curve show irregular ups and downs in pixel value, as shown in Figure 11d. It could be owed to the reflection from the arc light on the weld pool surface. For the line of 26 mm from the torch tip, as shown in Figure 11e, it goes through the tail of the weld pool. The curve in original image is relatively flat and hardly to show the edge of weld pool tail. By contrast, the curve in fused image has two peaks in the middle, corresponding to the left and right boundaries of the weld pool tail. To sum up, the image captured with the adaptive image fusion can be adaptive to the brightness of the keyhole region, avoiding the overexposure of the pixel in the keyhole area. Meanwhile, the boundary of the weld pool tail can also be recorded clearly. In addition, the edges information is reserved which does help to the feature extraction in the monitoring process.
The results of the size measurement of the features in weld pool region are analyzed below. The width of the weld pool and the width of the weld pool head are detected. In addition, due to the use of the adaptive image fusion, the keyhole inside the weld pool can be extracted. The highest coefficient of the fitted parabola is considered to be the curvature of the keyhole entry geometry. The results are shown in Figure 12. At the beginning of the welding process, the arc pressure is applied to the weldment but the penetration is not realized. Therefore, the heat input is transferred to the weldment, causing a large amount of metal melted and form the weld pool. As a result, the width of the whole weld pool and its head become wider. As the welding process becomes stable, the keyhole is formed and the torch travels for a certain distance. A portion of the input heat is leaking from the keyhole exit at the bottom and transferred to the other parts of the weldment. Therefore, the width of the weld pool becomes smaller and reaches a stable value which is close to the width of the weld pool head. During the whole process, the arc pressure digs out a cavity in the liquid metal no matter whether the keyhole is formed or not, so the curvature changes due to the oscillation of the molten metal but remains within a certain range. The result shows that the complete information from the weld pool can be an extraction via the adaptive image fusion, such as the internal and external features of the weld pool. Besides, the accurate data can be obtained by the measurement of the width or curvature, which has potential to reflect the welding process information like quality. The proposed method can be a new way to analyze the process of the keyhole TIG welding.

Conclusion
The high dynamic range of the keyhole TIG welding scene is measured by experiments. It is found to be the reason for bringing difficulties to the clear monitoring of the weld pool region as well as the detection of the internal features of the weld pool. In this paper, an adaptive image fusion algorithm is proposed to obtain a new image of the weld scene. It can not only extend the dynamic range of the welding images, but can also have the potential to keep good imaging quality when fusing the images captured under different welding parameters. Based on this, the extraction of the features of keyhole and the weld pool is realized. The main contributions of this study are concluded as follows: 1. The reason for the visual monitoring problem encountered in keyhole TIG welding is found to be the dynamic range of the scene is greater than the dynamic range of the camera. 2. An adaptive image fusion algorithm is proposed to widen the dynamic range of the welding image and ensure that the weld pool with high contrast can be well exposed when the welding situation changes. 3. The operation window for adaptive image fusion algorithm is got, including the choice of the optical range for imaging and the minimum exposure time of the camera. 4. The simultaneous feature extraction of the whole welding zone is realized, including the internal and external regions of the weld pool.
Author contribution (1) The dynamic range of the keyhole welding scene was measured, which is considered to the reason for overexposing in camera monitoring.
(2) A high-dynamic-range visual sensing method was proposed to solve the over-exposing problem caused by the strong arc light.
(3) Achieve the simultaneous extraction of the features of both weld pool and keyhole of K-TIG welding.
(4) The optimal operating window for proposed sensing algorithm was found for industrial application.

Declaration
Ethics approval and consent to participate All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee. I believe the participant is giving informed consent to participate in this study.