A low-cost vision-based weld-line detection and measurement technique for robotic welding

doi:10.21203/rs.3.rs-2110063/v1

Download PDF

Research Article

A low-cost vision-based weld-line detection and measurement technique for robotic welding

https://doi.org/10.21203/rs.3.rs-2110063/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

To make robotic welding more flexible and intelligent, artificial intelligence-based systems are one of the most important developments. With the help of computer vision, an algorithm for detection of weld path, and measurement of weld gap, and weld length is developed and its implementation in robotic welding is explored in this paper. A novel method that makes use of various image processing algorithms and mathematical operations is proposed for subtraction of boundary edges and determining the weld length and weld gap at every seam point. An evaluation of the robustness of the developed algorithm was performed on butt joints of various shapes (e.g., straight, zigzag, and curve) and sizes (e.g., length of the joint, gaps between abutting edges). The obtained length and gap values are validated by physical measurements, and the absolute errors were found to be within ± 1 mm, and ± 0.1 mm, for length and gap measurements, respectively, The model possesses 99.5% average precision in detecting atypical weld region. Weld seam information can be extracted effectively with the proposed algorithm, which proves its viability for industrial applications.

Weld-line detection

vision-based

low-cost

weld gap

weld length

robotic welding

There has been a tremendous impact of welding on the manufacturing industries worldwide. Metalwork is indispensable to the automotive, aviation, and other process industries. The physical demands of welding are very high. It has become increasingly common for robots to be deployed in fabrication industries to meet industrial demands for fast and accurate welding by reducing human intervene. Different manufacturing sectors use robotic welding for its precision and accuracy. In addition, it reduces labor costs, avoids welders working in hazardous environments, and increases the productivity and quality of welds. Previously, and even now, the majority of robotic welding relied on operator’s instructions. In spite of this, these methods do not take into account the uncertainties associated with welding. Programming is done well in advance of welding on teach-and-play robots, where the weld paths must be predefined, and the robot must be reprogrammed to perform welding in different conditions. It drains an ample amount of time and increases the cost. This reduces process efficiency, constrains welding shapes, and makes it inflexible to meet the needs of the industry. Real-world problems involving unpredictable environmental conditions cannot be solved by it. For the purpose of addressing these challenges, intelligent robotic technology has been boosted by incorporating computer vision into robotic systems.

The concept of weld seam tracking was evolved through years. Researchers used several techniques for achieving accurate seam tracking. Chronologically, weld seam tracking can be divided into three generations [1]. A two-pass weld movement was used in the first generation, with the first pass obtaining seam geometry, and the second pass welding. During this period, contact type sensors predominated. Touch sensors and tactile sensors are examples of sensors used for seam tracking. Tactile signals obtained from sensors are very useful for acquiring location features [2] at different portions of the area of interest. Knowledgebase containing possible features for weld line was created for segregating weld line features from other locations. Collected features are generally matched with the knowledge frame for accurate seam tracking. These sensors had been used for both two- and three-dimensional edge tracking [3, 4]. These contact type sensors seem to be useful for electronic industries, where minute welds are being performed on circuits. These types of sensors are not suitable for medium or large welding, because they require searching a large area for feature acquisition, which is time-consuming.

Welding in the second generation includes real-time tracking of seams and welding together. A variety of non-contact sensors started being used, such as arc sensors, ultrasonic sensors, and electromagnetic sensors. Arc sensor [1, 5] has been used mostly among mentioned ones. Measured voltage changes, when distance between torch-tip and workpiece changes. Based on this principle, stable tracking has been achieved [6]. Nonetheless, arc sensors are used for larger diameter pipe welding in shipbuilding industries, but they are only effective for up to 0.5 mm thick plates [7]. For high thickness workpieces like 10–12 mm, ultrasonic sensors are used [8–11], which measure the distance from any sensed obstacle. The major disadvantage of using ultrasonic sensor is that the speed of sound is dependent on temperature, which limits its application in real-time sensing. In addition, any unwanted obstacle can also be detected by ultrasonic sensor, which can lead to wrong estimation of weld line information [1].

As a result of the limitations mentioned above, third generation weld seam tracking evolved relying on vision-based technology. In the past, highly expensive infrared and laser-assisted sensors have been used for flawless seam tracking [12, 13]. Radiation from the weld zone is captured by the infrared sensors, which gives thermal distribution of that zone as an image data. Those are used for tracking of weld seams in real time by image processing algorithms. Additional benefits of this kind of imaging are that, it can detect weld gap as well as the depth of penetration. However, this method is not widely used due to its high cost. Also, very less gap in weld lines can cause insignificant difference between thermal distributions at the edges. In order to deal with these challenges, high quality vision sensors like CCD and CMOS cameras have been started to be used [14–16]. But due to environmental noise, appropriate weld edge detections were tough to achieve. That is why laser assisted vision sensing was introduced. A laser vision system has inbuilt CCD sensor and laser producer, which is projected on weld seam [16–19]. CCD sensor scans the projected laser for concentrated seam tracking. Though, it was an effective setup for successful seam tracking, but cost wise it was not economical. This has led to the use of normal optical cameras and improvisation of algorithms for tracking.

Machine vision has revolutionized the industry by making it faster, cheaper and simpler to use. In the robotic welding, the robotic arm has 6 degrees of freedom and moves on a 3-dimensional axis. The robotic arm’s end consists of the welding torch that enables the welding process. In a reported work, the workpiece was assumed to be at the center to simplify the image processing methods [20]. A search window was used to segment the foreground from the background by thresholding the intensity values in RGB space. To detect the start and end points of the straight path, stereo matching had been performed but points on the template should be known beforehand. Vision sensor was also used for seam tracking in GMAW process [15]. Two focused windows were introduced that covered the weld pool and seam. An improved Canny detection algorithm was developed to predict the threshold values automatically and used edged fitting to detect the edges of the weld lines. However, these algorithms are just limited to detecting straight weld seams. As an alternative approach proposed, the boundary subtraction method [21] compares the intensity value on either side of the line detected after Hough transformation. If the pixel intensity difference was greater than the threshold, these were considered as boundary lines and eliminated in further processing steps. To a certain extent, the algorithm works, but it fails in images of low contrast. An attempt has been made to track the weld path in an enclosed region of interest [22]. It focused on ensuring the most efficient seam path line based on parametric shapes like straight, zig-zag lines, and semi-circle. But the shape of the weld lines should be known prior to performing the welding. An improved canny edge detection method was used to detect the weld path for straight butt joints by performing edge fitting using a focused window [15]. In an automatic welding for lap joints, a method for finding the initial and endpoints of a lap joint was described and an algorithm was developed to link the broken weld lines and recognize the T-junctions in weld seam through image processing [23]. The lap joints are in high contrast with respect to the background, making the edge detection more accessible and not satisfying all the weld assemblies set up. In another study, as alternative to use traditional image processing methods, image segmentation, CNN and features point extraction were performed to process the noisy V groove weld images using laser sensors, which were used to extract the center point of the groove [24]. Since image processing techniques depend on their surroundings, the Fuzzy Decision-making theory helped to determine edge lines and applied linear regression to obtain the coordinates of weld seam contours for straight, kinked and curved lines [25]. Though this method works well at different light conditions and performs seam tracking, it does not provide us any information on the weld parameters like its gap and length from the captured image. The coordination of process parameters like current and voltage with the weld gap is essential for producing a sound weld. In the above papers, researchers have considered welding with the assumption of a constant weld gap. However, depending on the seam shapes, the seam gap may change at every seam points. Hence to control the weld penetration, the variation of weld gap must be taken into account. In various edge detection techniques reported [26], using the vision toolbox in LabView, edge fitting technique was applied to obtain the edge lines of the weld seam. The midpoint and seam gap of the corresponding edge points were measured by the caliper of the vision assistant toolbox. In the proposed technique expensive CCD camera with frame grabber was used for image acquisition and LabView software platform for processing. Moreover, no detailed information was mentioned on the applicability of the method on finding corresponding points for gap measurement in curve shaped weld profile or at the corners. The weld seam and weld pool in the images had been extracted using a series of image processing techniques and fitted the edges of the weld seam using the least square method (LSM) [27]. The difference between the upper and lower lines yields the weld gap. However, the set-up used to capture the images during welding is expensive and not industrially economic.

A predefined region was introduced in most research work to lower the computational cost. The weldment is considered at the center, which is not feasible in all cases, or the algorithm is not flexible enough to detect any arbitrary shape of the weld seam. In spite of the fact that these parameters affect weld quality, scanty research exists on how to automatically determine the gap and length of the weld. This paper’s significance lies in the developed algorithm which automatically detects the weld assembly (RoI) without any prior knowledge about the weld seam and reduces the irrelevant information. A predefined region is not required to subtract the weldment from the background. The paper describes performing seam tracking on any given weld shape and finding the gap and length of the weld seam by applying image processing methods. Moreover, the camera used in the present work for image acquisition is a low-cost webcam. In addition, both the image acquisition and processing are completed within a minimal wait-time (2 seconds) of the robot arm at the ‘Home’ position of the robot. Therefore, the proposed process of weld path detection and measurement does not impact the process downtime. In contrast, the real-time vision-based seam tracking systems need expensive CCD or CMOS camera with various types of optical filters and dimmer glasses to capture the image near weld pool. However, a certain type of filter cannot accommodate different lighting conditions at varying arc intensity [28]. In an alternative approach, the seam information was approximated from the center of the weld pool captured by a CCD camera [29]. Nevertheless, due to disturbances in arc light or radiation from the weld pool, the false predicted points were eliminated by setting a threshold value from user experience, which makes the method unsuitable for very curvy paths or sharp corners. As example, if a weaving path is considered, dynamic torch movement will affect the seam tracking. Also, gradual change in weld gap measurement is not possible for weld seam tracking in arc welding if the torch is moving fast or in a highly non-linear path. Present solution also overcome this issue by giving prior knowledge about weld gap. The presented algorithm in this paper can meet the requirements of industrial production with good reliability. The developed model is cost-effective, robust and industrially deployable.

Human may quickly identify the welding seam profile from a contaminated welding picture because they understand its fundamental structure rather than relying on numerical measurements. It is critical for a welding robot to extract and express the intrinsic structure of the welding seam profile in the same way that a human does. As a result, we need to reduce the search region to the area surrounding the seam to eliminate a lot of frames/pixels which are unnecessary in the image. This also increases the efficiency of feature point extraction. It is required to obtain the basic information of the joints to track different types of joints automatically. To detect the weld joints automatically, the area of object detection has been explored.

As shown in Fig. 1, present study involves three basic steps for fulfilment of the objective, which are camera calibration, weld path detection and weld length and gap measurement. Camera calibration is a common step, which is used for conversion of image pixel to world coordinate points. Other two steps consist of the methodologies for visible weld gap and zero gap welds (abutting edges are in contact). Later sections describe these steps in details. The complete algorithm with object detection, image pre-processing, and processing takes ~ 1.3 seconds to run in a computer having configuration of an Intel i7 Hexa core processor and 16GB of memory.

2.1 Weld path detection for visible gap welds

2.1.1 Region of Interest (RoI)

For capturing the image of the workpiece, a low-cost webcam (Logitech C270) was attached with the robot manipulator. The initial images are captured from a fixed location (‘Home’ position) of robot arm, so that the camera can cover the entire welding area on the fixture (as shown in Fig. 2) The captured images are converted into grayscale images. The main reason for the conversion is that the deep learning model used was not able to detect the weld zone as the region of interest (RoI) for variations in colour features with variations in materials. Thus, it was required to train the deep model with several combination of workpieces. Gray scale conversion eliminated this problem. For locating and cropping the ROI, a state-of-the-art deep learning model YOLOv5 (You Only Look Once) was used.

YOLOv5 is one of the top object detection algorithms easily retrained on any custom dataset. YOLO is a one-stage detector that performs feature extraction, image localization, and classification simultaneously. It consists of mainly three architectures: (a) Backbone, (b) Pyramid, and (c) Head. Cross Stage Partial (CSP) darknet53 is used as the backbone architecture for feature extraction. CSP works on the principle that the input feature map is divided into two sections. The first section passes through the dense blocks, and the other one is passed through the next layer without any convolution. Due to this, CSP gets the advantage of reusing its features repeatedly, and the vanishing gradient problem is diminished. The neck architecture makes use of the Path Aggregation Network (PAN). Due to the down-sampling of the images, many fine-grained features are lost during the convolution, hence preserving these features. They have used the bottom-up and top-down approaches to concatenate the large-scaled features to the low-level features and vice versa. The Head is the final prediction which comprises a vector that consists of predicted bounding box coordinates (center, height, width), the prediction confidence score, and the probability classes [30].

The leaky Relu activation function is used in the middle layers to implement non-linearity and it helps to prevent the shortcomings of ReLU (dead neuron problem). The Sigmoid function is used to detect the final layer stochastic gradient descent as the optimization function as it helps the model to generate nonlinear separable boundary. Irrespective of the input, the sigmoid layer outputs value between (0,1) making it easy to calculate the gradients during backpropagation and binary cross-entropy for calculating the loss of class probability. Depending upon the size and accuracy, Yolov5 is divided into Yolov5n, Yolo 5s, Yolo V5m, Yolo 5l, Yolo 5x. Yolo 5x has been chosen for weld seam detection from the above five, because of its Mean Average Precision of 0.65 on the COCO dataset, which is higher than the other versions, though a compromise of few milliseconds in object detection time because of its larger size.

An initial set of 100 images were captured of butt joints with aluminum and steel plates of thickness ranging from 0.5 mm to 6 mm in similar and dissimilar combinations. The joints consist of different shapes like straight, zig-zag, curvilinear and multiple straight lines. Augmentation techniques like flip, rotation, brightness, grayscale were applied to increase the dataset to 800 images on which the model was trained. The image set is split into training and validation sets. The training images contained 90% of the dataset, whereas validation set consists of remaining images. After training the model on custom images, a Mean Average Precision of 0.995 at a confidence score greater than 0.5 was achieved. Figure 2 shows the ROI location within the captured images. These RoI has been cropped for reducing unwanted features around the weld zone.

2.1.2 Bilateral filtering and edge detection

The Gaussian filter computes the weighted average of neighborhood pixels for image smoothing to average the noise and preserve the signals. However, there are high spatial variations at the edges, and image smoothing suppresses these contours, which generally hold information of the weld seam. Therefore, a bilateral filter (extension of Gaussian filter) is used as an edge-preserving technique that averages out the smooth regions in the image while preventing the weighted averaging operation at the edges. Unlike the Gaussian filter, the bilateral filter considers spatial and photometric distance.

The canny edge detector is the most popular among other operations due to its accurate edge detection and anti-noise properties. The stages included in this edge detection operator are gradient calculation, non-maximum suppression, double thresholding and edge tracking by hysteresis. Figures 3 and 4 illustrate the results for the noise reduction and edge detection using canny filter.

2.1.3 Boundary subtraction algorithm

Despite applying various filters and morphologies, the borders of the workpiece are still prevalent. To extract the weld path, it is necessary to eliminate the boundary edges. A border subtraction method is proposed, which is suitable for this purpose. The boundaries at the ends of the weld path in the cropped RoI are straight lines and are located at the extremum of the image. To detect these border lines, Probabilistic Hough transform (PHT) is considered as an excellent tool. PHT gives the freedom to choose a threshold, minimum line length and minimum gap between points considered as a line. The parameters are set to a lower value to detect the finer noises. Still, there is a possibility that some weldment region will be detected. A conditional logic is applied to prevent such misdetections. For identifying the horizontal borders, Y coordinate of the start and endpoint of detected lines should lie at extremity of the image and slope ranges between 0° to 15°. Similarly, for locating vertical borders, the X coordinate of start and endpoint of lines should lie at extremity of the image and slope ranges between 75° to 90°. After filtering out the lines through the given condition, all the pixels situated on the filtered lines are set to zero.

2.1.4 Connected component algorithm

The connected component algorithm helps to analyze the connectivity, and blobs in the binary image that can be filtered and segmented. The classical connected component algorithms make two passes over the image. The first pass scans throughout the image linearly and assigns temporal labels to the current pixel. It stores the above and left pixel label values. If they are not foreground pixels, a new label is assigned to the current pixel. If the above and left pixels have different labels, it is known that the current pixel connects the two adjacent pixels, so these two labels are stored as equivalent. Then, a second scan is run over the image to replace each temporary label with its corresponding equivalence class’s identifier [31]. Brighter corresponding pixels whose surface area is less than the given user input threshold is discarded. Using this operation, two binary images of the weld seams are obtained, and the weld line, weld gap can be extracted from these images. If there are two weld lines present in an image, it will be processed later for weld gap measurement, otherwise it will be fed to zero gap weld line measurement.

2.1.5 Determining weld gap

To determine the gap between two abutting edges at any particular point, two corresponding points on each edge are matched. The distance between those two points yields the weld gap and the series of the mid-points between all the corresponding points is treated as the weld path. The steps followed in order to execute this operation are discussed below:

(a) All the bright pixel coordinates of any one of the weld lines are stored in an array.

(b) Any arbitrary point (x₁, y₁) and its consecutive 10th point (x₂, y₂) are selected. The slope of the line connecting these two points (m₁) and its corresponding perpendicular slope m₂ = -1/m₁ are calculated.

(c) A straight-line equation is formulated from the obtained slope (m₂) and the midpoint (\(\frac{{x}_{1}+{x}_{2}}{2}, \frac{{y}_{1}+{y}_{2}}{2})\).

(d) The point on the other weld line having least distance and satisfying the above line equation is determined, and taken as the corresponding point of first edge.

(e) The distance between these two points is calculated to obtain the weld gap and also the midpoint to get the average weld line.

Figure 7 and 8 show the results of detected gap and weld line, respectively.

2.1.6 Extrapolation

Due to image processing steps, the fine granular information at the end of weldments is eliminated along with the noise. This leads to incomplete detection of the welding line. An extrapolation is performed to find the approximate missing coordinates of the weld seam. This involves generating points outside of a set of known data points. Complete path detection is achieved by extrapolating between the weldment endpoints and the boundaries. Figure 9 shows the final result of weld gap and weld path measurement overlayed on the initially captured image.

2.2 Weld path detection for zero weld gap

2.2.1 Edge detection using Difference of Gaussian

The bilateral filter was previously used to remove noise as a technique. Nevertheless, this filter has the disadvantage of sweeping away fine textures in an image. Edges of the weld seam with no gap (~ 0) are not detected well using canny though it is used only to segregate whether there is gap between weld lines or not, and the difference of Gaussian (DoG) is used to avoid losing fine textures of weld seams. A condition is imposed in the above methodology to automate the welding process for any gap measure. In the case that the connected component algorithm fails to give us two binary weld seams, it can be assumed that the weld seam is very narrow (weld gap is almost equal to 0), and then all the steps should be repeated with some minor changes as referenced in Fig. 1. This additional condition decreases human intervention and makes the algorithm more flexible for all gap measures.

DoG filter has been applied in place of the bilateral technique. It is an edge enhancement algorithm consisting of subtraction of two Gaussian Kernels of different standard deviation (σ) values. By using Gaussian filters at different scales, edges can be subtracted of an image by computing the difference between these two filtered images. The filter with a larger σ removes the high-frequency components (noises), making the image smoother and subtracting helps suppress the low-frequency regions. The remaining frequency components are considered as edges. It is a low pass filtering technique, and applying DoG will help to highlight specific high-frequency regions in the image.

2.2.2 Boundary subtraction and area Morphology

Detected weld seam images still contain some unwanted clusters of pixels after performing the DoG filter. Area opening is a morphological operator that removes the brighter connected pixels whose surface area is less than the given user input threshold. This process is based on an image’s maximum tree representation. The area property keeps track of a node’s total number of pixels (i.e., the size of a granule or region), and reconstructs the image with the tree nodes that have the area more significant than the threshold [32].

2.2.3. Skeletonization

Once the noises are eliminated, a binary image featuring the weld seam as bright pixels is left. However, the thickness of the weld path is more than one pixel. So, to thin the foreground pixels, skeletonization is performed on the image. Hit and Miss transform is used to preserve the geometry, remove noise, isolating the object, as well for the thinning process by applying the erosion morphology operator and pair of structuring filters [33]. A series of structuring elements are defined, and if all the pixel values (0 or 1) of the structure coincide with the image, then the center pixel value is assigned as 0, and this process is repeated. It includes iterative thinning of the boundary pixels until a one-pixel thick structure is obtained without affecting the object’s connectivity and length. The mathematical operations are based on the fundamentals of set theory.

2.2.4. Extrapolation

Lastly, the missing pixels at the end of the weldment are extrapolated as discussed in section 2.8. At the end of these image processing operations, a single weld seam track is extracted with a 0 mm weld gap.

2.3 Camera Calibration

For the conversion of image coordinates into space coordinates, a camera calibration is necessary. As shown in Fig. 11 the projection of a 3D object or world coordinates on image plane and projection plane depends on the distance of focal point from image plane. Also, position of the viewed image depends on the position of image plane with respect to camera. For digital camera, distance of focal points is measured by \({f}_{x}\) and \({f}_{y}\), which are the focal distances in x and y directions. Position of image plane is represented by \({c}_{x}\) and \({c}_{y}\), which represents the shift of image plane origin from camera origin. These parameters form an intrinsic matrix (K), which can be calculated through camera calibration. Without these parameters, calibration provides counter parameters for distortion removal, which takes place due to uneven bending of light through the lenses. Figure 12 shows the distorted and non-distorted images.

Upon calibration of the camera, rotation matrices (R) and translation matrices (T) are calculated. Multiplication of K and (R|T) provides the projection matrix, which converts image coordinates into world coordinate. Later, these coordinates are sent to the robot to perform the controlled motion of welding according to the values received [34]. Assuming an image point is denoted as (u, v) and its corresponding space coordinates are (X, Y, Z). Then the transformation between the image and real-world frame according to the optical principle of the camera Pinhole model is

\(P=K\times \left[R|T\right]\) Eq. (1)

\(K=\left[\begin{array}{ccc}{f}_{x}& 0& {c}_{x}\\ 0& {f}_{y}& {c}_{y}\\ 0& 0& 1\end{array}\right]\) Eq. (2)

\(R|T=\left[\begin{array}{cccc}{r}_{11}& {r}_{12}& {r}_{13}& {t}_{1}\\ {r}_{21}& {r}_{22}& {r}_{23}& {t}_{2}\\ {r}_{31}& {r}_{32}& {r}_{33}& {t}_{3}\end{array}\right]\) Eq. (3)

\(\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_{x}& 0& {c}_{x}\\ 0& {f}_{y}& {c}_{y}\\ 0& 0& 1\end{array}\right]\times \left[\begin{array}{cccc}{r}_{11}& {r}_{12}& {r}_{13}& {t}_{1}\\ {r}_{21}& {r}_{22}& {r}_{23}& {t}_{2}\\ {r}_{31}& {r}_{32}& {r}_{33}& {t}_{3}\end{array}\right]\times \left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]\) Eq. (4)

[K] is the 3×3 matrix of intrinsic parameters. It provides information about the focal length in X and Y directions (\({f}_{x}\), \({f}_{y})\) optical center of the lens (\({c}_{x},{c}_{y})\). [R|T] is the extrinsic camera parameters of the 3×4 matrix, which provides us information about the camera’s orientation in 3D space, where R represents rotational deviations and T is translational deviations. Steps followed to calculate the parameters given in the above equation are discussed below:

(a) Images of the chessboard has been taken from different viewpoints of a known size. The 3D coordinates can easily be localized due to their sharp gradients considering one of the corners as the origin. Figure 13 shows some of the pictures of chess board for calibration purpose.

(b) As welding workpiece and chessboard lies in same plane that is why, rotational and translation matrices have been calculated using chessboard only. In this way projection matrix has been calculated. As it is a 3×4 matrix, that is why direct inversion was not possible. For that pseudo inversing was used to get world coordinate from image pixel coordinates.

Table 1

Calibration Results and camera parameters.
Intrinsic Parameters	Values (mm)	Distortion Coefficients	Values (mm)
\({f}_{x}\)	2386.983	\({k}_{1}\)	-0.0156
\({f}_{y}\)	2387.768	\({k}_{2}\)	6.962
\({c}_{x}\)	956.014	\({p}_{1}\)	-0.0000967
\({c}_{y}\)	539.775	\({p}_{2}\)	-0.0000248
		\({k}_{3}\)	-128.629

Table 2

Verification of accuracy of the algorithm with weld gap greater than 0.5 mm.
Shape	Actual weld length (mm)	Calculated weld length (mm)	Absolute error in weld length measurement (mm)	Points	Actual weld gap (mm)	Calculated weld gap (mm)	Absolute error in weld gap measurement (mm)
Straight	205.28	204.54	0.74	P₁	2.61	2.54	0.07
				P₂	2.78	2.80	0.01
				P₃	2.75	2.77	0.02
Curve	150	150	0.00	P₁	1.78	1.84	0.06
				P₂	1.30	1.34	0.04
				P₃	3.58	3.61	0.03
Zig-zag	116.87	117.83	0.96	P₁	2.84	2.84	0.00
				P₂	1.42	1.31	0.11
				P₃	2.02	1.93	0.09

Table 3

Verification of accuracy of the algorithm with weld gap approx. to 0 mm.
Shape	Actual weld length (mm)	Calculated weld length (mm)	Absolute error in weld length measurement (mm)
Straight	101.98	101.67	0.29
Curve	150	149.15	0.85
Zig-zag	143	143.67	0.67

3.1 Experimental analysis

An experimental platform has been set up to verify the performance of the seam-tracking algorithm. A low-cost webcam was attached to the robot arm. Images were captured at a fixed distance of 300 mm from the workbench and vertical to the worktable. The camera is perpendicular to the workbench in this framework. The components welded in this experiment are of different materials like aluminum, magnesium and stainless steel with variable thickness. The shapes of the weld seams included in experimentation included straight, zig-zag, curve, and multiple straight butt joints. The seam detection results of various shapes are illustrated in Tables 2 and 3. The accuracy of our model has been validated through physical measurements.

3.2 Validation of results

The experimental results are depicted using figures and quantitative values. Weld seam length and gap values obtained from the algorithm are compared with actual situation values to identify any deviation or error in the algorithm. Weld gap is measured at three random locations named P₁, P₂, and P₃, respectively. Tables 2 and 3 show the theoretical and calculated values of weld length and gap. In determining weld length, the absolute error is less than 1 mm, and in determining gap, it is within ± 0.1 mm. As a comparative study with the solution developed using a laser vision sensor and adaptive Fuzzy control algorithm, the present solution gives ~ 0.41 mm lesser maximum deviation between the actual and detected weld lines [35]. Furthermore, the proposed algorithm also performs better than other reported vision-based solution using CCD camera [22]. The performance comparative chart shown in Fig. 15 shows the proposed vision-based solution to be an accurate and low-cost solution for industries. As no significant error is observed between the actual and obtained values of the parameters, it can be inferred through the results that the image processing technique developed is a reliable pipeline of algorithms.

Real-time vision-based seam tracking in robotic arc welding is difficult due to strong arc intensity, spattering, and reflection of laser light. Also, this requires very expensive camera or other vision setup (e.g., laser), and high-performance processing unit, making it inaccessible to small- and medium-sized industries. This paper focuses on developing a low-cost optical vision-based solution for automatic weld seam tracking. This research work presents an efficient pipeline of algorithm for selection of weld region, and extraction of features to determine the weld length and gap at every seam points. In particular, the conclusions made out of this paper are as follows:

(a) A low-cost webcam was attached with a six degree of freedom robot arm and was used to capture the image of the workpiece clamped on the fixture. Given the captured image as input, YOLOv5 model was used to select the weld region (RoI), which reduces the initial complexity for weld path detection. The model is capable of detecting atypical weld regions with an average precision of 99.5%.

(b) Boundary subtraction method followed by connected component algorithm was used to identify the abutting edges by eliminating the boundary lines created by workpiece edges, and other noises in the image.

(c) The developed solution is capable to identify and measure the total length of the weld path with various shapes such as straight, zig-zag and curve with ± 1 mm of accuracy.

(d) A welding path with little to no gap, as well as one with a significant gap, can be precisely measured at every seam point with an accuracy of ± 0.1 mm.

Experimental results show the closeness and accuracy of the predicted parameters to the actual values making the developed solution well suited to industrial application. Nevertheless, there are still some aspects that can be improved. By using a high-performance graphics processing unit (GPU), processing times of the algorithm can be drastically reduced.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Author Contributions

J. Reddy: Investigation, Writing – original draft, Writing – review & editing, Formal analysis. A. Dutta: Investigation, Writing – review & editing, Formal analysis. A. Mukherjee: Writing – review & editing, Data curation, Formal analysis. S.K. Pal: Conceptualization, Methodology, Investigation, Writing – review & editing, Supervision, Data curation, Formal analysis.

Rout A, Deepak BBVL, Biswal BB (2019) Advances in weld seam tracking techniques for robotic welding: A review. Robot Comput Integr Manuf 56:12–37. https://doi.org/10.1016/j.rcim.2018.08.003
Presern S, Gyergyek L (1983) An Intelligent Tactile Sensor–An On-Line Hierarchical Object and Seam Analyzer. IEEE Trans Pattern Anal Mach Intell PAMI-5:217–220. https://doi.org/10.1109/TPAMI.1983.4767375
Suwanratchatamanee K, Matsumoto M, Hashimoto S (2008) A novel tactile sensor torch system for robot manipulator and active object edge tracking. In: IECON Proceedings (Industrial Electronics Conference). pp 2617–2622
Suwanratchatamanee K, Matsumoto M, Hashimoto S (2010) Robotic tactile sensor system and applications. IEEE Trans Ind Electron 57:1074–1087. https://doi.org/10.1109/TIE.2009.2031195
Ushio M, Mao W (1996) Modelling of an arc sensor for DC MIG/MAG welding in open arc mode: Study of improvement of sensitivity and reliability of arc sensors in GMA welding (1st Report). Weld Int 10:622–631. https://doi.org/10.1080/09507119609549059
Moon HS, Ko SH, Kim JC (2009) Automatic seam tracking in pipeline welding with narrow groove. Int J Adv Manuf Technol 41:234–241. https://doi.org/10.1007/s00170-008-1474-9
Jeong S-K, Lee G-Y, Lee W-K, Kim S-B (2001) Development of high speed rotating arc sensor and seam tracking controller for welding robots. In: IEEE International Symposium on Industrial Electronics. pp 845–850
Zhang SB, Zhang YM, Kovacevic R (1998) Noncontact Ultrasonic Sensing for Seam Tracking in Arc Welding Processes. J Manuf Sci Eng 120:600–608. https://doi.org/10.1115/1.2830164
Maqueira B, Jarzynski J (1989) Application of Ultrasonic Sensors to Robotic Seam Tracking. IEEE Trans Robot Autom 5:337–344. https://doi.org/10.1109/70.34769
Bastos TF, Martin JM, Calderon L, Ceres R (1994) Weld seams detection and recognition for robotic arc-welding through ultrasonic sensors. In: IEEE International Symposium on Industrial Electronics. pp 310–315
Stavridis J, Papacharalampopoulos A, Stavropoulos P (2018) Quality assessment in laser welding: a critical review. Int J Adv Manuf Technol 94:1825–1847. https://doi.org/10.1007/s00170-017-0461-4
Gao X, You D, Katayama S (2012) Infrared image recognition for seam tracking monitoring during fiber laser welding. Mechatronics 22:370–380. https://doi.org/10.1016/j.mechatronics.2011.09.005
Pritschow G, Mueller S, Horber H (2002) Fast and robust image processing for laser stripe-sensors in arc welding automation. In: IEEE International Symposium on Industrial Electronics. pp 651–656
Ma H, Wei S, Lin T, et al (2010) Binocular vision system for both weld pool and root gap in robot welding process. Sens Rev 30:116–123. https://doi.org/10.1108/02602281011022706
Xu Y, Fang G, Chen S, et al (2014) Real-time image processing for vision-based weld seam tracking in robotic GMAW. Int J Adv Manuf Technol 73:1413–1425. https://doi.org/10.1007/s00170-014-5925-1
Nele L, Sarno E, Keshari A (2013) An image acquisition system for real-time seam tracking. Int J Adv Manuf Technol 69:2099–2110. https://doi.org/10.1007/s00170-013-5167-7
Xu D, Fang Z, Chen H, et al (2012) Compact visual control system for aligning and tracking narrow butt seams with CO2 gas-shielded arc welding. Int J Adv Manuf Technol 62:1157–1167. https://doi.org/10.1007/s00170-011-3873-6
Gao J, Wu C, Liu X, Xia D (2007) Vision-based weld seam tracking in gas metal arc welding. Front Mater Sci China 1:268–273. https://doi.org/10.1007/s11706-007-0048-3
Xu D, Wang L, Tan M (2004) Image processing and visual control method for arc welding robot. Proc - 2004 IEEE Int Conf Robot Biomimetics, IEEE ROBIO 2004 727–732. https://doi.org/10.1109/ROBIO.2004.1521871
Dinham M, Fang G, Zou JJ (2011) Experiments on automatic seam detection for a MIG welding robot. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 7003 LNAI:390–397. https://doi.org/10.1007/978-3-642-23887-1_49
Dinham M, Fang G (2012) Weld seam detection using computer vision for robotic Arc Welding. IEEE Int Conf Autom Sci Eng 771–776. https://doi.org/10.1109/CoASE.2012.6386339
Mohd Shah HN, Sulaiman M, Shukor AZ (2017) Autonomous detection and identification of weld seam path shape position. Int J Adv Manuf Technol 92:3739–3747. https://doi.org/10.1007/s00170-017-0380-4
Kiddee P, Fang Z, Tan M (2014) Visual recognition of the initial and end points of lap joint for welding robots. 2014 IEEE Int Conf Inf Autom ICIA 2014 513–518. https://doi.org/10.1109/ICInfA.2014.6932709
Du R, Xu Y, Hou Z, et al (2019) Strong noise image processing for vision-based seam tracking in robotic gas metal arc welding. Int J Adv Manuf Technol 101:2135–2149. https://doi.org/10.1007/s00170-018-3115-2
Kuo HC, Wu LJ (2002) An image tracking system for welded seams using fuzzy logic. J Mater Process Technol 120:169–185. https://doi.org/0.1016/S0924-0136(01)01155-4
Rout A, Deepak BBVL, Biswal BB, et al (2018) An optimal image processing method for simultaneous detection of weld seam position and weld gap in robotic arc welding. Int J Manuf Mater Mech Eng 8:37–53. https://doi.org/10.4018/978-1-7998-1754-3.ch064
Shen HY, Ma HB, Lin T, Chen SB (2007) Research on weld pool control of welding robot with computer vision. Ind Rob 34:467–475. https://doi.org/10.1108/01439910710832066
Shen H, Lin T, Chen S, Li L (2010) Real-time seam tracking technology of welding robot with visual sensing. J Intell Robot Syst 59:283–298. https://doi.org/10.1007/s10846-010-9403-1
Shen HY, Lin T, Chen SB (2007) A Study on Vision-Based Real-Time Seam Tracking in Robotic Arc Welding. In: Tarn T-J, Chen S-B, Zhou C (eds) Robotic Welding, Intelligence and Automation. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 311–318
Thuan D (2021) Evolution of Yolo Algorithm and Yolov5: the State-of-the-Art Object Detection Algorithm
Di Stefano L, Bulgarelli A (1999) A simple and efficient connected components labeling algorithm. Proc - Int Conf Image Anal Process ICIAP 1999 322–327. https://doi.org/10.1109/ICIAP.1999.797615
Huang X, Fisher M, Zhu Y, et al (2003) An Efficient Strategy for Implementing Iterative Area Openings Using the Max Tree. In: 8th Australian and New Zealand Intelligent Information Systems Conference. Sydney, pp 115–120
Soille P (2004) Hit-or-miss and Skeletons. In: Morphological Image Analysis: Principles and Applications. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 139–182
Wu A, Xiao H, Zeng F (2019) A Camera Calibration Method Based on OpenCV. ACM Int Conf Proceeding Ser 320–324. https://doi.org/10.1145/3378065.3378127
Zou Y, Wang Y, Zhou W, Chen X (2018) Real-time seam tracking control system based on line laser visions. Opt Laser Technol 103:182–192. https://doi.org/10.1016/j.optlastec.2018.01.010
Zou Y, Chen T (2018) Laser vision seam tracking system based on image processing and continuous convolution operator tracker. Opt Lasers Eng 105:141–149. https://doi.org/10.1016/j.optlaseng.2018.01.008

Download PDF

Version 1

posted

You are reading this latest preprint version

A low-cost vision-based weld-line detection and measurement technique for robotic welding

Status:

Version 1

Abstract

Figures

1. Introduction

2. Methodology

2.1 Weld path detection for visible gap welds

2.1.1 Region of Interest (RoI)

2.1.2 Bilateral filtering and edge detection

2.1.3 Boundary subtraction algorithm

2.1.4 Connected component algorithm

2.1.5 Determining weld gap

2.1.6 Extrapolation

2.2 Weld path detection for zero weld gap

2.2.1 Edge detection using Difference of Gaussian

2.2.2 Boundary subtraction and area Morphology

2.2.3. Skeletonization

2.2.4. Extrapolation

2.3 Camera Calibration

3. Results And Discussion

3.1 Experimental analysis

3.2 Validation of results

4. Conclusion

Declarations

References

Status:

Version 1