Stable 3D seam tracking for thick plate GMAW manufacturing with T-joints using Kalman filter and machine learning

In this paper, a robust stable three-dimensional (3D) seam tracking method is investigate based on the Kalman filter (KF) and machine learning during the multipass gas metal arc welding process with a T-joint of 60 mm thickness. The laser vision sensor is used to profile the weld seam, and with the reference image captured before arcing a scheme is proposed to extract the variable weld seam profiles (WSPs) using scale-invariant feature transform and the clustering algorithm. An effective slope mutation detection method is presented to identify the feature points of the extracted WSP, namely the candidate welding positions. In order to lower the impact of fake welding positions on seam tracking, a Bayesian Network model is first built to implement fault detection and diagnosis for the visual feature measurement process using the involved process parameters and the trigger rule. A KF, as an estimator, is then established to further stabilize the tracking process combing with a self determination algorithm of the measurement result. With the visual calibration technology, 3D seam tracking is realized. Seam tracking results show that the proposed method overcomes the tremor of the tracking position and multiple fake candidate welding positions on tracking accuracy, and the tracking accuracy is 0.6 mm. This method provides potential industrial application value for industrial manufacturing with large-scale components.


Introduction
Welding of the thick steel plates of more than 10 mm thickness is a major production step in many industries such as marine engineering, shipbuilding, the construction of the power plant and pipeline manufacturing. Mainly this is done with the gas metal arc welding (GMAW) process or the submerged arc welding process [1]. Advancements of computer vision, control theory, robotics, machine learning and artificial intelligent [2], are applied to the complicated welding process by many researchers to understand the welding input and weld formation [3] and improve welding quality and efficiency [4,5] etc. However, it is the multipass welding process that makes thick plate GMAW manufacturing full of challenges because for this automated welding manufacturing process, weldment quality highly depends on stable seam tracking for each sampling time and each pass. More robust seam tracking methods are the guarantee of welding quality during the multipass arc welding process. *Correspondence: lejian_ncu@163.com 1 School of Information Engineering, Nanchang University, Nanchang, 330031, China Effective acquisition of the tracking position is the key to realize the automated welding process similar to welders [6][7][8]. Visual sensing technologies are the most commonly used method for weld seam detection and tracking because it has many advantages over other sensors, such as huge information content, noncontact detection, on-line processing and high precision [9]. The most popular technology used for weld seam detection and seam tracking is based on the principle of optical triangulation [10]. A structured light is projected on the workpiece surface in front of the laser focus, and the reflected scattered light is imaged back to a camera [11]. The controller uses the visual feature (e.g., a designated feature point of the weld seam profile (WSP)) to deal with seam tracking. Based on visual sensing, the specific challenges of seam tracking during the thick plate GMAW process include the acquisition the variable tracking positions, more disturbances, the irregular fluctuation of the tracking position because of the vibration of the welding torch and effective decision making of the tracking position from multiple fake tracking positions. These adverse factors extremely influence the accuracy of seam tracking and final manufacturing quality.
The current seam tracking methods based on visual ·2· sensing are classified into two categories [12]. The first one involves sending the welding position to the robot controller and using the robot interpolation algorithm to control the robot's movement. In this case, how to obtain the accurate welding point in images is the key, such as the continuous convolution operator tracker [5], the Gaussian kernelized correlation filter [13] and the error smoothing filter [14]. In [15], a controller based on the kinematic and dynamic model of the mobile welding robot was designed to deal with the partial uncertainty and the disturbance of the welding process. The second one uses control algorithms to eliminate the deviation of the welding torch to implement seam tracking, such as the self-tuning fuzzy controller [16], the self-adaptive PID controller [17], the fuzzy-PID controller [18] and feedforward compensation and predictive tracking [19]. As the Kalman filter (KF) can use the real-time state estimation and the measurement from sensors to predict the optimized estimation state of the observation object, it is an effective technology for object tracking in machine vision [20][21][22]. Some researchers apply the KF to seam tracking based on visual sensing. In [10], through embedding an Elman neural network into an adaptive KF, the error estimator was used to compensate for the filtering errors during the high-power fiber laser welding process with the stainless steel plate. In [23], a KF and a radial basis function neural network were applied to tracking the weld center position by a magneto optical sensor. In [24], the WSP in images was searched in a small area that was estimated by a KF to avoid some disturbance. In addition, the KF was applied to seam tracking to eliminate the influence of noises upon the seam tracking precision in [25].
Although considerable seam tracking technologies have been presented in the literature, the stable automated seam tracking method to acquire satisfactory welding quality is not fully investigated particularly during thick plate GMAW using active visual sensing. This is not only because of the challenges mentioned above but also because of process parameters that are generally used in the corresponding steps [5,14]. Fault detection and diagnosis (FDD) of the visual feature acquisition process is necessary to stabilize each tracking process in automated welding manufacturing. Bayesian Networks (BNs) have emerged from research into artificial intelligence as a formal means of analyzing decision strategies under uncertain conditions [26]. BNs allow to achieve desirable inferences using bidirectional reasoning, which makes them a widely used tool in system reliability [27], fault diagnosis [28] and risk analysis etc.
In order to stabilize the seam tracking process and acquire the high-accuracy manufacturing result, this paper presents a stable 3D seam tracking method based on the KF and machine learning during the GMAW manufacturing process with a T-joint of 60 mm thickness. A laser vision sensor is used to detect WSPs, and a novel scheme is proposed to extract the variable WSPs based on scale-invariant feature transform (SIFT) and the nearest neighbor clustering algorithm. The boundary points of the existing weld beads are identified with slope mutation segmentation as the candidate welding positions. Compared with the methods that are used by the related companies in the market, this paper presents two methods to stabilize the tracking process rather than just use a designated tracking point to implement the tracking process. The first step is using a BN to implement FDD of the welding position acquisition process to improve the measurement accuracy of the tracking position, and the second one is proposing a KF to optimize the tracking position with a self determination algorithm of the measurement result after the welding position is specified. Seam tracking experiments show that the proposed method overcomes the vibration of the welding torch and various fake tracking positions to implement stable tracking processes with the tracking accuracy of 0.6 mm.

Feature point identification of the WSP
As the identified feature points of the WSP will be used to extract WSPs, the feature point identification (FPI) process precedes the WSP extraction section.
The effective FPI result is the basic premise of stable seam tracking. There is currently no standard definition of the feature points of the WSP. However, most of studies that have been reported use slope variation characteristics to detect these points. They basically are the boundary points of the weld beads and the groove of the joint. Hence how to effectively compute the slopes of these data points and measure variation characteristics is the crucial step.
The novel FPI process mainly includes four steps (Figure 1) after the extracted WSP has been processed to be linear. First, pairwise linear interpolation is used to complete the extracted WSP (some segments of the extracted WSP may lost during the extraction process) and guarantee the actual position of the feature point as  ( , ) ii xy is the coordinate of the start point of the segment that is next to the previous one. K is linearly filtered with a one-dimensional window 1×9 to further reduce the fake fluctuations on monotonic interval detection. Third, searching monotonic intervals in the slope vector K , as monotone interval acquisition 1, is followed with the definitions where i b and i e are the endpoints of these monotonic intervals in the vector K . In order to reduce fake monotonic intervals, the fourth step is using the Otsu algorithm to segment the lengths of these monotonic intervals to acquire a threshold T . Only these monotonic intervals whose lengths

Figures 2 and 3 show the effectiveness of the proposed FPI method in this paper through different FPI processes.
Tests show that the FPI result mainly depends on the size of the filtering window, and from 1×3 to 1×19, the larger the filtering window size is, the fewer the fake feature points are. However, bigger sizes will result in bigger errors between the identified feature points and their actual positions. The FPI error produced by this process parameter is about ±4 pixels. It is the first process parameter used in the proposed algorithm and initialized to 1×9. Tests also show that the desired tracking position for each pass is from or can be referred to the identified feature points, and few fake feature points may surround the designated tracking position.

WSP extraction with SIFT and clustering algorithm
The FPI result in this paper is sensitive to the remaining interference data points around the extracted WSP. It is a challenge to the WSP extraction scheme facing the large-scale spatial distribution of the laser stripe. Raw images (Figures 2 and 3) show that the different WSPs have notable directional characteristics. They partly vary ·4· with the increase of the weld beads. This study presents a novel scheme (Figure 4) to extract the WSP based on SIFT, orientation feature detection and the nearest neighbor clustering algorithm to adapt to this variety [29]. This scheme uses a reference image that is captured before arcing (the welding torch and the IAPS are in service). This action does not delay much time. However, the reference images provide lots of useful information as follows. First, the regions that are used to emphasize the multidirectional characteristics of the WSP are automatically divided using the identified feature points of the reference image. Second, the region of interest (ROI) is determined to save the computational cost using the extracted WSP of the reference image (the upper and lower boundaries are 80 pixels respectively from the center of the extracted WSP of the reference image). Third, one of the feature points of the reference image is the benchmark that is used to optimize the designated tracking point with the proposed KF. Finally, the reference image is used to locate the WSP with SIFT during the seam tracking process.

Gabor Filtering
Gabor filters can effectively highlight the detection object through designating specific filtering angles as well as restrain interference background [30,31]. We present a Gabor filter as

Local Thresholding
Local thresholding is carried out twice during the WSP extraction process (Figure4), which is defined as where j represents the jth column of the image, 2: 2, 2: 2 i i j j F − + − + represents the average intensity in each 5× 5 pixels sliding window (the average thickness of the WSP in images is about 5 pixels) and j LT is the threshold for binarizing the region from column j-2 to j+2. The step of j is 2.

Locating WSPs with SIFT
The reference image and the ROI of the orientation feature map are the input images of the SIFT algorithm. Using the matching points produced by the SIFT algorithm (Figure 6a), the WSP is located when the orientation feature map is binarized, and the data points are clustered with the nearest neighbor clustering algorithm (Figure 6b). The clusters that are the nearest to the matching points are considered as the segments of the WSP (Figure 6c). This extraction process adapts to the variable WSPs because the SIFT algorithm, to a certain extent, keeps invariant to image rotation, translation and scaling, and partially invariant to illumination changes and affine or 3D projection. In addition, Denoising is implemented as where i and j respectively denote the coordinates of the data point in images in the row/column direction and represents the relationship of and. Dilation is conducted as where  is a three-order unit matrix, F is the image and represents the relationship of or. After the WSP has been extracted, it is judged whether it is divided. When the first located cluster lies on the right of the right-most feature point of the reference image, the extracted WSP is not divided, and the first located cluster is just the extracted WSP. Otherwise, only the cluster with the larger length is kept when two located clusters overlap in the horizontal direction. Thinning means calculating the average ordinates of the data points of the extracted WSP in the vertical direction. Using the proposed WSP extraction scheme and the FPI procedure, the percentage of the effective FPI result is over 93%.

BN Model for Monitoring the FPI Process
Offline FPI tests show that the process parameters such as Left filtering angle (LFA) for out of the groove, Middle filtering angle (MFA) for Groove, right filtering angle (RFA) for Filling region and Future filling region used in Gabor filtering and Smoothing window size (SWS) account for the ineffective FPI result, 7%. In order to improve tracking accuracy and stability, a FDD mechanism is proposed in this paper using a BN model to optimize the corresponding process parameters in real time.
A BN is a pair ( , ) is a directed acyclic graph, and  is a set of parameters. V is a set of nodes, and E is a set of arcs (or edges) connecting these nodes. An arc between two nodes in V denotes a direct probabilistic relationship. A parameter on node vV  is a probability ( ( )) is the parent set of v [32]. If there is no parent node of node v , then the parameter on v is () Pv . A BN model interactions between variables describing a system using representative datasets and statistics founded on Bayes' theorem of conditional probability [33]. It represents the joint probability distribution of a set of random variables [34]. Bayes' theorem is represented as where () PA is the prior distribution of parameter A , () P B A is the likelihood function of the probability of new data B given existing data A and () P A B is the posterior probability of A given B .
Once the conditional probability tables of all variables have been determined from the historical data or experience, the posterior probabilities of the target variables can be yielded with inference algorithms such as the Adaptive Importance Sampling (AIS) algorithm [35] and the Estimated Posterior Importance Sampling (EPIS) algorithm [36] etc. Decision making is conducted using the posterior probabilities and the specific rule.  Table 1.  Figures 2 and 3. thicker cluster means that the maximum thickness of a cluster is over the average one, 5 pixels. big gap means that the minimum Euclidean distance between two adjacent clusters in the horizontal direction is over 10 pixels. In addition, Left denotes the second feature point from left to right is designated as the tracking point. Middle denotes that the feature point between the second and the right most one from left to right is the designated as the tracking point. Right means that the right most point is designated as the tracking point. Bigger means that the related process parameter should be set bigger. Normal denotes that the number of the identified feature points is equal to the one of the reference image. Maintain means that the current process parameter is suitable.
In the following, we use k ij a to denote the conditional probability on Structure learning refers to [37][38][39]. This process is carried out by initializing k ij a , ij c and k i d and followed by optimizing them through various FPI tests and the experience in setting process parameters as follows.
Within the corresponding angle ranges, the bigger filtering angles for Out of the groove, Filling region and Future filling region will result in the extracted WSP that is more complete. The middle filtering angle for Groove is better than other angles. Moreover, the larger the smoothing window size, the less the number of fake feature points of the extracted WSP. The learning process continues offline until the conditional probabilities of ( 1, 2, 4) , as the main factors, have been adjusted to correct the ineffective FPI results. Meanwhile, we stipulate that if the posterior probability of the target node is over 60% with an appointed inference algorithm, this target node needs adjusting. Based on this stipulation, the result of each parameter decision should accord closely with the judgment by our experience.
The trigger of starting the BN-based FDD process is that the specified tracking point satisfies the requirement: the Euclidean distance between the specified feature point and the last optimized result (provided by the subsequent KF) is bigger than 3 pixels.
After some process parameters are diagnosed with adjustments, their current values should be increased or reduced by 10 for the Bigger or Smaller, respectively.
To vividly show the proposed BN model, we use the GeNIe Modeler to illustrate this model and give an example of the FDD process shown in Figure 8a-c. Seven classic inference algorithms are tested to show the effectiveness of diagnosing the given fault. The EPIS algorithm is adopted in this study because it uses (9) and (10) to calculate an approximation of the optimized importance function with the highest correct decision-making rate rather than learning it (Figure 8d).
where i X is a variable in a polytree,  is the set of  The effective FPI rate is up to about 98% with this FDD process. It is implemented only once for the FPI process because of the high computational cost. Actually, the vibration of the welding torch during the arm moves also causes the measurement result to deviate from the actual position. The KF proposed in this paper is responsible for the rest 2%.

Proposed KF Model
In this paper, the KF model is used to further stabilize the tracking process. After the optimized tracking point in images has been determined, it is converted into the coordinate in the world coordinate system through the visual calibration technology [40]. The feature point of the reference image is used to initialize the proposed KF through designating one of the identified feature point online as the tracking position.
We suppose that the coordinate of the tracking point in images is independent, and only the state function that estimates the coordinate of the tracking point in the horizontal direction is given (the implementation of the proposed KF in the other direction is the same) as where ( )~(0, ) k N Q  is the zero-mean Gaussian noise and Q is the process covariance, which reflects the change of the WSP at two consecutive sampling times.
Here, (0)  is initialized with the coordinate of the designated tracking point in the reference image. The measurement function is where () Mk is the coordinate of the identified feature point corresponding to the designated tracking point in the row directions, is the zero-mean Gaussian noise and R is the measurement covariance and reflects the accuracy of identifying the feature points with the proposed method here. In order to effectively determine the measurement result, a self determination algorithm of the measurement result is proposed as follows. The feature point(s) that deviate(s) from the last estimation position by less than 5 pixels are acquired first. Then, if the number is zero, the optimized result at the last sampling time is consider as the current measurement result; if the number of these points (this point) is only one, it is considered as the measurement result, and if the number is bigger than one, the one that is the nearest to the last estimated position is used as the measurement.

·8·
In this paper, we suppose that Q is a constant 0.1, and R alters with the tracking position.
R is also a constant for each welding process. As Q is very small, the estimation function is thus written as where ( 1) kk  − is the prediction result at the current sampling time based on the last optimized result, where ( 1 1 where () Kg k is the Kalman gain: The covariance at the kth sampling time is updated as The unit of Q , R and () P  is the pixel. The inputs used in the KF include (0)  , (0) P and R . (0)  is determined with the designated feature point of the reference image. This optimization process overcomes the instability from this seam tracking form of designating a feature point as the tracking position just using the number. This technology is usually provided by some companies such as Servo-Robot.

Experimental results
Experiments are conducted in the automated welding system shown in Figure 9. Before implementing these related algorithms, we updated the dedicated welding system to meet the computational requirement. Three tests of seam tracking are given with different welds to show the performance of the proposed method in this paper. For the first welding test, the parameters used in the proposed KF are given in Table 2. Before welding, the second feature point is selected as the tracking position (see Figure 2a). Because of too many samples collected from the long-distance welding process, only the first 300 samples are used to show the tracking result. The welding process parameters include the wire extension 25 mm, the wire feeding speed 150 mm/s and the welding speed 5 mm/s. The welding result and the tracking result with the proposed KF are given in Figure 10. The tracking accuracy is 0.48 mm in the x-direction, 0.58 mm in the y-direction and 0.60 mm in the z-direction. Without the proposed KF, the corresponding tracking accuracy is 1.23 mm, 1.43 mm and 1.58 mm.   Figure 3, the inputs used in the proposed KF are given in Table 3. In order to validate the flexibility of the proposed seam tracking method in this paper, an offset position based on the fourth feature point is specified as the tracking position. The position lies between the third and fourth feature point, and deviates from the fourth one by 8 pixels. The welding process parameters are the wire extension 20 mm, the wire feeding speed 141.6 mm/s and the welding speed 6 mm/s. The tracking results with the proposed KF are that the accuracies in the x-, y-and z-directions are 0.49 mm, 0.58 mm and 0.57 mm respectively whereas the corresponding accuracies are 0.91 mm, 2.01mm and 1.49 mm respectively without the proposed KF. The tracking and welding results are given in Figure 11.   Figure 3, the inputs used in the proposed KF are given in Table 4. The welding process parameters are the wire extension 20 mm, the wire feeding speed 133.3 mm/s and the welding speed 8 mm/s. The tracking results with the proposed KF are that the tracking accuracies in the x-, y-and z-directions are 0.53 mm, 0.54 mm and 0.59 mm respectively whereas the corresponding accuracies are 1.48 mm, 1.51mm and 1.63 mm without the proposed KF. The tracking and welding results are given in Figure 12.  Welding result Ⅲ Using the proposed tracking method, the welding manufacturing result with a T-joint of 60 mm thickness is given in Figure 13, which shows that this method can be applied to the thick plate GMAW welding process once the tracking position is specified in advance.

Conclusions
Aiming at eliminating the effects of the vibration of the welding torch and the fake welding positions that easily occur in the visual-sensing-based arc welding process on seam tracking, this paper investigated a robust seam tracking method based on a KF and machine learning for thick plate T-joint GMAW manufacturing. Several conclusions include (1) The proposed visual feature extraction method can identify the candidate tracking position required during the multipass welding process. This method provides a valuable reference for thick plate GMAW with different joints. (2) Using a BN model to diagnose the visual feature acquisition process improves the accuracy and robustness of the visual feature measurement. (3) Combing with the proposed self determination algorithm of the visual feature measurement, the proposed KF can eliminate the effect of the vibration of the welding torch and fake tracking positions on the specified tracking point with the tracking accuracy of 0.6 mm. This seam tracking technology can be applied to industrial manufacturing with large-scale components.