Visual sensing and controlling of the keyhole in robotic plasma arc welding

With an advantage of “single-sided welding and double-sided forming,” plasma arc welding (PAW) has a great application potential in modern industrial production. The welding quality can be guaranteed by sensing and controlling of the keyhole. However, it is difficult to make an on-line observation on the back of base metal, and realize a dynamic registration of the visual sensor and welding torch. In this study, it has investigated the relationship between the welding condition and image feature of keyhole. Image processing is designed to obtain the weld pool image and conduct a template matching of the keyhole. The target feature of weld zone will be extracted and processed in real time. Besides, it has designed a digital controller for the welding robot and power source in this study and discussed control method to stabilize the keyhole and achieve good welding quality. Eventually, experiments are conducted to inspect the comprehensive performance of the welding control system with varying disturbance. This study is of important significance for the visual sensing and controlling of the keyhole in PAW. It will provide technical support for the weld quality control, and promote the development of welding technology based on machine vision in intelligent manufacturing field.


Introduction
With the advantages of large energy density, good arc directionality, and strong penetration ability, the PAW can penetrate 8-10-mm medium-thick stainless steel at one time even without groove and no filler wire. Wu et al. have proposed a keyhole effect to achieve "single-sided welding, doublesided forming" [1]. The dynamic behavior of keyhole can determine the weld quality, and it is significant to maintain the keyhole to obtain a strong metal welded joint [2,3]. Keyhole in the PAW depends on the welding condition, such as welding current, pilot gas, and welding speed [4,5]. Li et al. have simulated the influence of welding parameters in PAW. It is sensitive to the change of parameters and may become unstable due to the behavior of molten metal [6]. Prasad et al. have pointed that the uncontrolled variation may lead to the keyhole closure or collapse of the weld pool [7]. As a result, it is difficult to maintain a stable keyhole by optimizing the process parameter window in PAW.
Because of high energy density and concentration of the plasma arc, welding position is just below the electrode hiding in the torch center. Due to the compression effect and high ionization degree of the plasma arc, PAW is available to obtain a uniform weld bead with high welding quality, and it has broad application prospect in heavy machinery, storage and transportation tanks, pressure vessel pipelines, aerospace shipbuilding, and so on [3,8]. However, the PAW system is relatively complex. The process parameters are numerous and coupled, which need the skilled technicians and high requirements for the control system. Due to the large size and complex structure of the welding torch, the observability of the heat-affected area is poor [9,10].
In order to observe and control the keyhole behavior directly and accurately, some researchers have utilized visual sensor in their studies. Wu et al. studied the dynamic variation of keyhole exit and inclination extent. The extent was indicating the penetration and key-holing ability of plasma arc [11,12]. A practical method based on a circular shape model of the weld crosssection area has been proposed to evaluate the weld quality [13]. A synchronous visual sensing system was built by Jia et al. with triple CCD cameras. Image registration was conducted and showed keyhole entrance and topside weld pool behaved closely related to the keyhole exit evolution process [14]. Yamane et al. applied convolutional neural network (CNN) to identify a possible gap in the image of weld pool [15]. Zhang et al. proposed a controlled-pulse strategy to improve the stability and dynamics of keyhole [16]. The relationship between process parameter and image features of keyhole has been investigated in these studies. However, there is still a lack of effective adaptive control method using closed-loop feedback approach to keep the keyhole stable and realize seam tracking in the meantime.
The authors have carried out the prior study on machine vision, image processing and adaptive control in PAW, which can be divided into three parts. First, the weld zone was monitored by utilizing visual sensor before the welding torch. Through the boundary detection of gouging domain of the keyhole, the movement and height control of plasma torch was realized [17]. In the subsequent study, the visual sensor was installed behind the welding torch. By detecting the triangle groove or a reserved gap in the welding direction, the study achieved the seam tracking during the welding process [18]. In addition, authors performed pattern matching and boundary detection of the keyhole. The study accomplished the adaptive control of weld pool behavior [19].
In this study, fundamental experiment is conducted to investigate the relationship between the process parameters and weld pool image with the keyhole by utilizing high-speed video camera. Target feature from shield cap to the edge of keyhole entrance in the image is found to vary along with the flowrate of plasma gas. According to the phenomenon, the control method to stabilize the keyhole has been discussed. We try to design a digital controller to conduct an adaptive control of keyhole by regulating the gas flowrate. This study will be of important significance in PAW to analyze the welding process, design of visual control system, and so on.

Visual PAW system
The visual PAW system built in this study is shown in Fig. 1a. Main components consist of a six-axis industrial robot, a visual sensing system with CMOS camera, computers for welding control and image processing, a digital controller (dsPIC) for the power source characteristic, and the data acquisition device collecting process parameters. Moreover, the welding robot with its movement controller connects to the control PC through the Ethernet to realize the robot position setting and human-computer interaction. The welding conditions of the power source are set on the DSPIC, and data is transmitted through I/O serial communication. It prevents the electric noise of welding power from entering the control port by utilizing isolation amp.
In the visual sensing system shown in Fig. 1b, there is a CMOS camera equipping with 95-nm filter intercepting all the visible light, especially the arc light, down to near infrared region (NIR). It focuses on the optical characteristics of the heat-affected zone of the base metal. In order to observe the keyhole entrance, the camera and welding torch are installed on the same side of the base metal. They are fixed on the same jig and move together in the welding. Furthermore, the camera is installed at a vertical angle of 47° in this system to make a clearest observation through the keyhole during the welding process. The distance from the camera lens to the observation area under the electrode is 150 mm. The waveform of shutter is synchronized with the welding current, and the shutter opens for 3 ms at each current trough. It is controlled by the dsPIC to eliminate the effect of strong arc light on the weld pool image.
The resolution of CMOS camera is 1280 × 1024 pixels and the field of view (FOV) is 18 × 15 mm 2 for each frame of image. The pixel accuracy is 14.3 m∕pixel with 1 mm corresponding to about 70 pixels in the taken image. The groove width is 12 mm narrower than the 18 mm of horizontal FOV. In this case, the CMOS camera can concentrate on the keyhole entrance during the PAW. Moreover, in order to correct lens distortion and make an accurate measurement of the keyhole size, camera calibration is adopting before the welding experiment to obtain intrinsic matrix and extrinsic matrix. Image processing software called Halcon13 containing over 2000 library operators has been utilized.

Basic experiment investigating the effect of welding conditions
In this study, the base metal is SS400 mild steel with V-type groove. Both plasma gas (PG) and shield gas (SG) adopt the pure argon. Welding condition conducted in the basic experiments is shown in Table 1. In order to investigate the effect of welding conditions on the image feature of weld pool, it conducts comparison experiments due to the change of the current value and PG flowrate. High-speed video camera utilized in this study can take 5000 frames per second. The images captured by the camera are shown in Fig. 2a.
According to the experimental results, it is found that the target feature from the shield cap to the keyhole entrance varies with the PG flowrate. While welding current is below 250 A, such as 235 A and 225 A in Fig. 2b, it shows an approximate linear variation along with the PG flowrate. However, when welding current approaches 245 A, the linear relationship will disappear and the keyhole will become unstable. It is considered that the pressure of plasma arc on molten metal becomes small, if heat input continues to increase. At this moment, PG flowrate has less influence on the size of keyhole, and the balance of arc pressure, gravity,  and surface tension of weld pool will be broken. Once welding current exceeds 250 A, vibration of weld pool will happen. The weld bead will become intermittent and humping. Relationship between PG flowrate and average value of target feature approximately calculated in steady state is shown in Fig. 3. In Fig. 4, welding current effect on the keyhole is investigated at the same PG flowrate. The effect of current changes on the weld pool is compared by utilizing high-speed video camera with the PG flowrate of 3.0 L/min. The width of the heat-affected zone increased with welding current, but target feature keeps approximately constant regardless of the welding current's variation. This condition is consistent with the PG flowrate of 2.0 and 4.0 L/min. Geometric model of the groove and sectional view of the weld bead after PAW is shown in Fig. 5a and b. By measuring the height and width of weld bead on both sides, sectional area of weld bead is approximately calculated. The relationship between the welding current and section of the weld bead is shown in Fig. 5c. As a result, the sectional area increased linearly with the welding current. And this condition is consistent with the flowrate of 2.0 and 4.0 L/min.
Based on the experiment, welding results show that the amount of molten metal in the weld pool increases with the heat input. However, the variation of welding current has less influence on the size and appearance of keyhole entrance in steady state. On the other hand, the variation of PG flowrate has an obvious influence on the keyhole feature. It is easy to obtain the image features of keyhole to ensure the stability of PAW process.
The size of keyhole is limited by the arc pressure, and the surface tension of molten metal. If target feature continues to increase with the arc pressure under high welding current, it cannot afford to build the bridge between the iron plates of butt-weld. Once the distance increases to the critical value, it is difficult to maintain the keyhole's shape and probably results in burning through. Therefore, the target feature serves as a significant factor to ensure the stability of keyhole.
In the actual welding process, it has utilized CMOS camera to observe the weld zone, filter the strong arc light, and realize a real-time image processing. In order to investigate the flowrate's influence on the keyhole, step change (from 2.0 to 4.0 L/min) of flowrate is taken with a constant current (235 A). The appearance of weld bead and keyhole before and after the step change is shown in Fig. 6a. It is confirmed that the target feature increases with PG flowrate, as well as keyhole size. The increase in the height of back bead is accompanied by a decrease in the face bead, which appears to be a concave shape. Penetration of the base metal has been raised by PG flowrate.
Target feature calculated by image processing is shown in Fig. 6b. As mentioned before, 1 mm is corresponding to about 70 pixels. It shows a transient response of the target feature following the step change of PG flowrate. According to the experiment result, it can be approximated by a first-order system with time delay. Therefore, it is available to realize the maintenance of keyhole and weld bead through real-time control of PG gas flow in PAW.

Image processing for keyhole maintenance and seam tracking
In the PAW, the weld pool image is a complex result making a coupling of the intense light of plasma arc and the heat radiation of heat-affected zone with high-temperature. It is significant to design an image processing to effectively extract the keyhole feature from the image containing the arc state, brightness level of the weld zone, and some other information hidden in the weld pool. In the built system,  welding torch starts to move after the keyhole is formed. In order to design the image processing program, the first thing is to make sure whether the keyhole is formed. It will indicate that the welding has entered an initial stable state. According to the grayscale characteristics at different stages of the welding process, binarization pre-processing is performed on the weld pool image. Considering that the weld pool has not yet formed at the beginning of the welding. The length of weld pool will be shorter than normal and the bottom of the image is solid area with low brightness. So, we set a detection area at the bottom of the image to determine the welding stage, shown in Fig. 7a. If the average brightness of detection area is below the threshold, it will consider that the welding has not entered a stable state.
Cycle detection will be adopted until the welding reaches a stable state, and then the subsequent image processing will be carried out. By this means, the weld pool images taken at the beginning and end of the welding can be excluded to improve the image processing efficiency.
Because the electrode is in the geometric center of the welding torch and the plasma arc generated by the electrode is highly concentrated, the underside of the electrode can be considered the welding position. Therefore, we take the weld pool images with a shield cap on the top to calibrate the welding position. By detecting the geometric center of the shield cap, we can determine the welding position. It has set another detection area on the top of the taken image to identify the shield cap, shown in Fig. 7b. And the lowest coordinates of the welding torch will be calculated to determine the electrode position, i.e., welding position in the horizontal direction.
Since the keyhole and possible gap have low brightness and obvious shape characteristics, template matching can be used to identify the keyhole and gap quickly and improve the image processing efficiency. When the keyhole and weld pool are basically formed, the template matching will start to work. In order to reduce the effect of singularity with high  Step response of target feature brightness and enhance the contrast of keyhole and gap, the current image is combined with the previous two to generate a composite image with lowest brightness of each pixel. Representative samples of the image under several welding conditions will be captured through basic experiments for template matching in advance. Then, the region of interest (ROI) including a steady-state keyhole and a possible gap will be obtained as the template models. Through similarity calculation with the models, template matching is adopted to separate and extract the ROI from the weld pool images acquired during welding experiments. And the subsequent image processing will be performed only on the ROI rather than the entire image, which can save the processing time.
Template matching has adopted normalized cross correlation (NCC) in this study. It will calculate the similarity between the template model and target image. The adaptive threshold algorithm called Otsu is used to obtain the best global threshold. And if the similarity degree meets the threshold requirement, the similar region will be extracted. Similarity degree R NCC is given by In the equation above, the pixels of the template image are M × N . T(i, j) and I(i, j) stand for the pixel values of the template model and target image at the point (i, j) . The brightness of each pixel from the template model to target image serves as a vector element. The similarity degree will approach 100%, when the difference of the vectors' angle become smaller. And it is able to calculate the inner product by the transformation of vector and trigonometric function. Since the gain variations and vector length in inner product expression is regardless of the brightness change, template matching is shown to be appropriate once again in the image with confusion of various brightness.
The template matching of keyhole is shown in Fig. 8a. The keyhole has a relatively specific shape and can be easily recognized in the steady state without multi-stage processing. However, welding defects such as a gap or some other disturbance may exist and cause a deformation of keyhole. So, we have utilized several template models of unstable keyhole in the basic experiments to identify the keyhole accurately. It will reduce the image processing errors and robustness can be expected.
The image filtering and enhancement will be performed on the ROI. The image will be discretized and a Gaussian filter window is established to remove the image noise. By utilizing the histogram equalization to maximize the contrast of grayscale area, we can obtain continuous and clear edge information through Canny edge detection. In order to extract the basic outline of the keyhole and weld pool, the image smoothing and morphological image processing with structuring elements will be adopted. The keyhole and weld pool contour is obtained by calculating the difference with the original image, and finally, the shape characteristic parameters are extracted according to the contour curve.
The gap may exist in the weld zone due to the welding defects or placement of the base metal. A gap with large size will cause the deformation of the keyhole and destroy the force balance in the weld pool. Since the welding control in this study is based on image processing, it is necessary to avoid the processing mistake caused by a possible gap. Hence, a detection area traversing the image is set below the shield

Fig. 7
Identification of welding process through weld pool. a PAW has not entered a steady state. b PAW in a steady state cap, shown in Fig. 8b. The regions with low brightness can be obtained by binarization. By comparing the brightness of these regions with the central column in the image, the area with smallest difference and intense brightness change will be temporarily identified as a gap. A different similarity standard robust to the linear average brightness change is required, and the study has taken a double check of the gap. It will carry out template matching of the gap in the detected area, shown in Fig. 8c. The matching score above the threshold obtained by Otsu method will be recorded. Image processing program will calculate the transformation matrix based on the vector difference between template model and target image. The template model is represented through translation and linear transformation, and affine transformation. Template matching will be taken at least three times with some other prepared models, and the biggest matching score will be utilized to determine whether the target image contains a gap.
In addition, by detecting the horizontal center of the gap, or extract the shape feature of the weld pool, the way of obtaining the weld line has been studied before and introduced in the introduction. By calculating the deviation between the welding position and weld line, seam tracking has been realized. The target feature obtained by image processing is shown in Fig. 8c. It starts from the lowest coordinates of welding torch to the lower boundary of keyhole in the vertical direction. The shape of keyhole and length of the target feature remain steady under normal conditions. But if the welding condition is not adjusted properly, it will be difficult to maintain a stable keyhole and lead to the keyhole closure. As a result, it is significant to maintain the stability of keyhole shape feature, i.e., the size and target feature by adjusting the welding conditions. Then, the image processing results will be extracted and sent to the robot control PC and digital controller for power source, respectively. The keyhole maintenance and seam tracking can be achieved simultaneously in real time.

Controller design for welding robot and power source
In the joint space of the welding robot, path planning can ensure the end effector, i.e., plasma torch, passing through the start and end point of the weld line. The trajectory between the two points may be unknown; it will depend on the weld line and the robot kinematics characteristic.
In the visual robotic system, path planning of the operating space is carried out by visual sensor in the Cartesian coordinates, and the trajectory of plasma torch can be represented by a series of nodes. To find the vector of joint variables which can achieve the desired movement of the plasma torch, it needs to solve the inverse kinematics and path planning problem of welding robot. In this study, the control PC will calculate the coordinate conversion relationship from the image point to the space point. Then, based on the target feature obtained by image processing, the controller will estimate the desired pose of end effector. In this way, it will perform inverse kinematics calculation and transmit the pulse signals based on position information to the robot. Shan and Guo has designed the state feedback controllers and studied the relationship between movement pulse signal and end effector location in the robot kinematics [20,21]. Based on the position information, the required rotation angle of joint variables is given to each robot motor, and a desired trajectory of plasma torch will be obtained. In the practical operation, there are external forces and torques like inertial force and gravity affecting the dynamic performance of the welding robot. Even if the deviation from the target position is correctly calculated, it is difficult to obtain an ideal operation. Therefore, we have designed a digital controller in this study.
The block diagram of digital controller designed in this study is shown in Fig. 9.

Digital controller for welding robot
Control of the PAW robot is performed by feeding a movement pulse signal from the robot control PC to the manipulator. Since strong inertia force exists in each joint axis of the welding robot, plasma torch is unable to reach the target position immediately when the controller receives a movement pulse. Therefore, it is necessary to consider the time delay of the robot motion in the PAW system.
Based on the robot kinematics experiment, the characteristic curve of the system response has been summarized in this study. The 1st and 2nd-order delay indicial response to the movement pulse in the torch motion is shown in Fig. 10. Since the second-order response curve is closer to the step response, the robot control system will be approximated by a second-order delay system. The pulse signal sent to the control port will be integrated in the robot controller and the movement of plasma torch will base on this integrated value. Hence, the dynamic behavior of the welding robot expressed in the form of transfer function G t (s) is shown as follows: where X t (s) is the current position with respect to the weld line, and U t (s) is the operating value of robot controller. Due to Fig. 10, the damping ratio = 0.8 and the undamped natural frequency n = 40 rad/s are approximated in the designed robot welding system.
(2) can be further organized by By adopting the z-transform for the transfer function G t (s) , it is able to process the discrete-time signal and acquire the following equation: (2)  The desired response for the robot movement is illustrated in Fig. 11, and X t [z] to the reference value of the weld line is determined in a 10-sampling period and is shown as follows: In order to limit the setting response and control the weld line independently, the digital controller for seam tracking has been designed with its characteristic D t [z] by substituting the deviations of Eq. (5) and the desired response of Eq. (6) through where: And the a 0 ∼ a 12 and b 0 ∼ b 12 are the coefficients of the deviation and operating value.
The reference value R t [z] is the z-transform of the step function and it is recorded by The manipulating value u t [kT] of the welding robot at the kth sampling period is found by transforming Eq. (7) to the discrete-time system where T is sampling period with respect to the interval of image processing, and k is the number of iterations. Due to Eq. (9), the manipulating values are calculated from the weighted average of the deviations of previous 12 periods. Therefore, even if there exists an image processing error, it will have a less influence on the results. Hence, the seam tracking has been realized through the digital controller for the welding robot.

Digital controller for power source
On the other hand, due to the influence of molten metal viscosity and forces balance of weld pool, time delay exists in the characteristics variation of weld pool with the change of PG flowrate. According to the result of previous section, transient response of the target feature is following the step change of PG flowrate. Through the Z-transform, the controller characteristic D s [z] for power source has been approximated by a first-order delay system in this study In Eq. (10)

Experiment result and discussion
This study establishes a model between keyhole feature and welding condition parameters. Through accurately obtaining the feature information and effectively adjusting the condition parameters, it is available to stabilize the keyhole and achieve a quality control. Based on the welding quality through the designed experiments, it has evaluated the established model, image processing, and the control system.
The change of the welding condition is easy to cause a weld defect, such as incomplete fusion or burn through in the welding process. It has utilized the base metal with a 0 − 2mm gap in this study, shown in Fig. 12. Therefore, the plasma gas is easy to escape from the possible gap affecting the arc pressure and force balance in the weld pool. In this way, it is available to inspect the comprehensive performance of control system with a varying disturbance. The research goal is to verify whether the built system can maintain the stability of the keyhole feature to guarantee the welding quality.
Welding speed is 2.5 mm/s, and the length of base metal adopted for the one-pass welding experiment is 30 cm. The image feature of keyhole has been obtained to analyze the experiment result. The gap feature obtained by image processing is shown in Fig. 13. As a result, the detected gap width is smaller than the actual reserved under 2 mm. This is due to the influence of the fluidity and inertia of molten metal on both sides of the gap. When the gap width is less than about 1 mm, molten metal can fill into the gap so that it cannot be detected during this period.
Plasma arc could not penetrate the base metal immediately after the welding started, and the target feature was not detected by the visual sensor at this moment, because the keyhole had not formed. The welding torch started to move 4 s after the plasma arc generated a relatively stable keyhole. Since the flow of molten metal in the weld pool had not been stabilized after the torch started moving, target feature obtained by the sensor sampling became a fluctuating state, shown in Fig. 14. Then, the image processing program was about to work based on the template matching of keyhole. The length of the target feature varied to an ideal threshold determined by the basic experiments. And under the control of PG flow, it remained relatively stable. Therefore, it can be proved that by adjusting the PG flow, the keyhole feature can be effectively controlled as shown in the figure.
Since the plasma gas is easy to escape from the gap with its size increasing. It may easily cause a deformation of the keyhole and destroy the force balance of arc pressure, gravity, and surface tension in the weld pool. On the other hand, the diameter of heat-affected zone remains almost invariant due to a constant welding current, so the gap will lead to a decrease of the molten metal in the weld pool. In order to keep the target feature constant, i.e., maintain the shape of the keyhole, the flowrate of plasma gas has gradually decreased since the gap size became larger.
The welding result of weld bead is shown in Fig. 15. By inspecting the appearance of base metal on the both sides, there are no obvious welding defects such as incomplete fusion or burn through. By utilizing a flaw detector to inspect the weld zone after the experiments, no blowhole is found inside the weld bead. It has utilized laser cutting machine to obtain the cross section of the weld beads in order to analyze the welding quality, shown in Fig. 16. It can be concluded that the PAW in this study has achieved a good welding result. By comparing the cross sections of weld bead at each stage of the welding process, it is found that the plasma arc always penetrates the base metal, and the cross-sectional area as well as the height of the weld bead decreases due to the increase of the gap width.

Conclusions
In this study, it has faced the requirements of the observability of welding process and the controllability of welding quality in the PAW. And we have investigated the evolution process and response rule of keyhole behavior and weld pool feature, when welding condition dynamically changed. As a result, the built PAW system can effectively maintain the keyhole and guarantee the welding quality. The main results of this study are summarized as follows: