F2GAN based few shot image generation for GMAW defects detection using multi-sensor monitoring system

doi:10.21203/rs.3.rs-2221109/v1

Download PDF

Research Article

F2GAN based few shot image generation for GMAW defects detection using multi-sensor monitoring system

https://doi.org/10.21203/rs.3.rs-2221109/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 11 Aug, 2023

Read the published version in The International Journal of Advanced Manufacturing Technology →

You are reading this latest preprint version

Multi-source information intelligent sensing and online evaluation of the weld are two challenging problems in complex welding scenes. The difficulty is particularly pronounced in different welding scenes, where the multi-source sensing data exists large system deviations and small welding defects sample data are at large. In this paper, we propose modified few shot image generation model Fusing-and-Filling GAN (F2GAN) for welding data augmentation. To verify the efficiency of the modified F2GAN, some experiments were performed with various welding conditions. Through our proposed welding data set augmentation approach, two welding quality classification models combining multiple information are designed. We show the final classification accuracy of normal, burn through, incomplete penetration and welding deviation is 96.60%. Our results are beneficial for welding monitoring and quality evaluation in complex welding scenes.

intelligent welding

few shot learning

image generation

defects detection

Gas metal arc welding (GMAW) is a critically important process in advanced manufacturing. To ensure high quality welds with less defects, in-process monitoring and weld quality evaluation are essential. A skilled welders can determine the weld quality according to their sensing abilities of the liquid weld pool and arc sound. However, the welding quality highly relies on the experience and knowledge of the trained welder which may vary from person to person [1]. Hence many researchers have been developed intelligent welding sensors method to monitor the weld quality such as visual sensing, acoustic sensing, infrared sensing, and arc voltage and welding current measurement. Through these efforts, the sensor system can feedback welding defects such as burn through, incomplete penetration, and welding deviation left in the seam. It is still far from the practical application in complex welding scene specifically.

To monitor arc welding processes, many traditional machine learning and deep learning methods are used in the weld quality assessment depending on multiple sensing system. Machine learning methods include Principal Component Analysis (PCA), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Attentional mechanism [2–7], etc. Deep learning methods include convolutional neural network (CNN), back propagation network (BP Network), Recurrent Neural Network (RNN), Long Short Term Memory (LSTM) [8–11] etc. These methods help a lot in welding monitoring to ensure the welding quality.

However, there exists many difficulties for deep learning methods to be used in in-process monitoring of arc welding [12]. Obviously, the multi-source sensing data are collected from the specific welding scene which may show a large deviation in a different welding scene. In other words, machine learning or deep learning model for specific scene can’t be directly used in another scene, it should be re-trained or fine-tuned by target scene welding data. Additionally, it is challenging to deal with unbalanced data in the predictor classes, as there are far more welding sample data for sound weld than defect weld in industrial application. To avoid such situation caused by imbalanced data, researchers often use some data augmentation methods to re-balance the data. Data augmentation can re-balance the dataset by augmenting the defects samples. Zhang et al. [13] used WGAN and traditional image processing methods to augment the data. Two methods augmented porosity, burn through and crack defects’ image. This work demonstrated that the data augmentation model WGAN could solve the data insufficient problem. Yang et al. [14] combined the traditional data augmentation methods and U-Net network to locate defects in high precision. Kaestner et al. [15] used traditional data augmentation methods and compared the effects. Yang et al. [16] used deep semantic segmentation U-net to augment the dataset. And in the subsequent data processing they also used traditional data augmentation method to enlarge the dataset. Ajmi et al. [17, 18] studied the performance of different traditional data augmentation methods (rotate, translate, stretch, twist, mirror) on a small dataset. After the study, they used traditional data augmentation methods to improve the welding defects detection accuracy. Ma et al. [19] built a vision-based welding defects detection system relying on traditional data augmentation methods. Guo et al. [20] relying on condition GAN rebalanced the imbalanced dataset. And by use of the balanced dataset, they trained a defects detection model which worked well on crack, lack of fusion, lack of penetration, porosity, and slag inclusion. Zhang et al. [21] used traditional data augmentation method rotation and noise to help the welding process of aluminum alloy. Li et al. [22] by using traditional data augmentation methods successfully predicted the weld penetration based on this reflected laser pattern. Dung et al. [23] used traditional data augmentation methods to improve the performance of defects detection model. Yang et al. [24] augmented the data by random cropping. And the defects detection model trained on the augmented dataset worked well. Dai et al. [25] combined traditional data augmentation methods with random cropping and arranging to augment the dataset which improved the final accuracy. Zhang et al. [26] relying on traditional data augmentation methods got a robust defects detection model. It is worth mentioned that because the quality is easy to evaluate for images, the data augmentation methods mainly focus on images. However, those methods only deal with the data from target scene while make little utilization of the multi-source data. The ignorance of multi-source data limits the performance of those methods.

To solve unbalanced data as well as utilize multi-source data, semi-supervised learning and transfer learning can help a lot. And semi-supervised learning techniques have been used in welding in literature [27, 28]. For transfer learning in welding, there are also some works using models trained by abundant dataset and modified by target welding scene data [29–31]. And for the augmentation methods based on GAN, they have maximum degree of freedom, but they need a large quantity of data to train. In contrast, few shot image generation methods need less data to train and can augment data from few input. Specifically, it trains on similar scene data and augment target scene data by use of few target scene data. Thus, it not only has more freedom but also need less train data. The commonly used data augmentation methods can solve the unbalanced dataset while do little with the multi-source data. Therefore, the few shot learning method is an improvement to the traditional augmentation methods. However, there is few works for few shot image generation methods used in welding. Inspired by few shot image generation [32] and Fusing-and-Filling GAN (F2GAN) [33], we propose a modified few shot image generation model F2GAN for welding dataset augmentation including burn through, incomplete penetration, and deviation images datasets. To extract and analyze the features from images datasets, we build a defects detection model: modified ResNet50. And for features from current and voltage, we build a voting model based on fully connected layers, SVM (support vector machine [23]), KNN (K Nearest Neighbor [24]), Random Forest [25] and XGBoost (extreme gradient boosting [26]). The result from vision model and the result from current and voltage are fused by voting.

The rest of this paper is organized as follows: Section II describes the details of data augmentation model and classification model. The experiment equipment and dataset prepare are shown in Section III. Section IV contains the data preprocessing and features extraction. The augmentation result, classification result and discussion are shown in section V. Section VI is the conclusion.

2.1 Data augmentation model

Data augmentation model F2GAN comes from few shot image generation field. We use F2GAN for it needs less data for training comparing to common GANs when generate various images from input data. As a variant of GAN, F2GAN has a generator and a discriminator. The architecture of the generator is a U-Net combining skip-connection. And the discriminator contains convolutional layer, down-sampling and fully connected layers. In the training process, the model firstly maps the input image to a low dimension feature space by down-sampling, and then the input mapping back to the sample space from the low dimension feature space. In the mapping back process, the model fuses different features from different input images to generate new image by weight fusion. That is the way to generate new images from limited input without too much target scene data. And to obtain low-level details, F2GAN uses skip connection to pass the details from input to output directly which is lost in down-sampling. The weight fusion used to fuse the features can also be untiled to selectively fill the low-level details. The generated images will be evaluated by the discriminator, which is used for distinguish true and generated. Thus, the whole training process of discriminator begin with the convolution and pooling of input images, and then the features are classified by the fully connected layers. The output of the generator will be distinguished by discriminator which gives the error for backpropagation process. The total training process is similar with the common GAN.

For F2GAN, the core equation is the loss function. The details of the loss function of F2GAN are shown (1). L is the total loss; L_D is the loss of the discriminator; L_GD is the loss of generator; L₁ is the weighted reconstruction loss; L_c is the classifier loss; L_m is the mode seeking loss; L_a is the interpolation regression loss. And the ${\lambda }_{1}$, ${\lambda }_{m}$, ${\lambda }_{a}$ is trade-off parameters. More details is shown in literature [33].

$$L={L_D}+{L_{GD}}+{\lambda _1}{L_1}+{L_c} - {\lambda _m}{L_m}+{\lambda _a}{L_a}$$

F2GAN has two main differences with other GANs: First, F2GAN can utilize low-level details to generate new images from the input images by non-local attention module. This feature enables the F2GAN to generate various and clear images. The utilization process is modified by the relevant weight, and the low-level details comes from the down-sampling process and the filling of the low-level details is accomplished in the up-sampling procedure. The utilization of low-level details is based on the skip-connection which directly linked the down-sampling and up-sampling process. Second, the F2GAN is a model in few shot image generation field, which means the F2GAN augments the data without too much target data. Therefore, F2GAN satisfies the practical needs of welding. That is the reason for us to use F2GAN.

And before using the F2GAN, we make some adaptive modifications. First, to prevent overfitting phenomenon, we use dropout [34] in generator and discriminator. And to have the most various network architecture and a better generalization effect, the dropout ratio is 0.5. Differently, the dropout is used in both training process and testing process. Expressed in formula:

$$\begin{gathered} {x_{i+1}}=f(W\cdot {x_i}+b){\text{ }} \Rightarrow {\text{ }}{x_{i+1}}=f(W\cdot mask({x_i})+b) \hfill \\ {\text{ }}mask(x)=m\cdot x,{\text{ }}m \in {\{ 0,1\} ^D} \hfill \\ \end{gathered}$$

Among them, p is the dropout ratio, x_i+1 is the output of i-layer of the network, the first equation of (2) is the procedure of layer without dropout, the second is the procedure of layer with dropout, mask is generated by Bernoulli distribution with probability 0.5.

Second, to reduce the side effect caused by the wrong label, prevent overconfidence and remain the shape of the optimal judgment function for a various output at the same time, we use one-sided label smoothing to tune the whole training process of F2GAN. Expressed in formula:

$${P_i}=\left\{ \begin{gathered} 1,if(i=y) \hfill \\ 0,if(i \ne y) \hfill \\ \end{gathered} \right.{\text{ }} \Rightarrow {\text{ }}{P_i}\left\{ \begin{gathered} (1-\varepsilon ),if(i=y) \hfill \\ 0,{\text{ }}if(i \ne y) \hfill \\ \end{gathered} \right.$$

$$Loss=-\sum\limits_{{i=1}}^{K} {{p_i}\log {q_i}{\text{ }} \Rightarrow } {\text{ }}Los{s_i}=\left\{ \begin{gathered} (1-\varepsilon )*Loss,{\text{ }}if(i=y) \hfill \\ Loss,{\text{ }}if(i \ne y) \hfill \\ \end{gathered} \right.$$

$$L_{{D\_new}}^{{}}=\left\{ \begin{gathered} (1 - \varepsilon )\cdot {L_D},{\text{ }}if(i=y) \hfill \\ {L_D},{\text{ }}if(i \ne y) \hfill \\ \end{gathered} \right.{\text{ }} \Rightarrow {\text{ }}L_{{GD\_new}}^{\prime }=\left\{ \begin{gathered} (1 - \varepsilon )\cdot {L_{GD}},{\text{ }}if(i=y) \hfill \\ {L_{GD}},{\text{ }}if(i \ne y) \hfill \\ \end{gathered} \right.$$

$$L={L_{D\_new}}+{L_{GD\_new}}+{\lambda _1}{L_1}+{L_c} - {\lambda _m}{L_m}+{\lambda _a}{L_a}$$

Among them, P_i is the true probability distribution, $\epsilon$ is a small hyper-parameter which is 0.1 in this article. ${L}_{D\_new}$ is the new loss function of discriminator and ${L}_{GD\_new}$ is the new loss function of generator. In the left side of (3), (4), (5), it is the original part, and in the right side, it is the tuned part. It is clear that the tuned P_i could reduce the confidence of the label which in turn reduce the risk of overconfidence. And after one-sided label smoothing, the model can restrain the difference between the positive label and negative sample which enhance the generalize ability.

And considering the stability of GAN could be affected by sparse gradient, we replace the activation function relu [35] in discriminator by leaky relu [36] to avoid this phenomenon. Expressed in formula:

$$\operatorname{Re} LU(x)=\hbox{max} (0,x)$$

$$Leaky{\text{ }}\operatorname{Re} LU(x)=\hbox{max} (0,x)+\gamma \hbox{min} (0,x)$$

Equation (7) is the operation process for relu, and (8) is the operation process for leaky relu. Among them, γ is a small hyper-parameter which is 0.01 in this article. And by using leaky relu, the activation function will be activated all the time which improve the stability of the model by avoid the sparse gradient.

At the same time, for the images collected in a same weld bead is similar, we use two methods to increase the variance of the input to modified F2GAN. One is adding Gaussian noise to the input, the other is using gamma transformation. Gaussian noise means the probability density function of the noise follows Gaussian distribution. And the expression of Gaussian distribution is shown below:

$${p_G}(z)=\frac{1}{{\sigma \sqrt {2\pi } }}\exp (-\frac{{{{(z-\mu )}^2}}}{{2{\sigma ^2}}})$$

$$\begin{gathered} G(input){\text{ }} \Rightarrow {\text{ }}G(noised(input)) \hfill \\ D(input){\text{ }} \Rightarrow {\text{ }}\left\{ \begin{gathered} D(noised(input)),{\text{ }}for{\text{ }}real{\text{ }}image \hfill \\ D(G(noised(input))),for{\text{ }}generated \hfill \\ \end{gathered} \right. \hfill \\ \end{gathered}$$

σ and µ² are the expectation and variance of Gaussian distribution, and s is 0, m is 0 ~ 0.03. In the gamma transformation, the gamma varies from 3/4 to 4/3. In this way, the input of the modified F2GAN is different from the original image, but not distorted, which alleviate overfitting. And the input of the generator as well as the discriminator is thus noised in (10).

2.2 Classification model

The data flowing process in this experiment is shown in Fig. 1. After the collection of the data, we first preprocess it and then select the data with features. And as the data is class-imbalanced, we use some methods shown in Table 1 to augment the vision data, current and voltage. Then we use some models to classify the data. The welding states prediction model contains two parts, one is the modified ResNet50 model extracting analyzing the features from vision signals. One is voting result of fully connected layers, SVM, KNN, Random Forest and XGBoost. The parameters of these models were chosen by Bayesian optimization [37]. These inputs are the features from current and voltage. And the final prediction is the voting result of two separate outputs.

Table 1. The augmented dataset and the augmentation methods

Dataset	Augmentation method
Modified F2GAN	Few shot image generation
Repeat	Repeat
Grayscale transformation	Gaussian noise (std: 0-0.03) Gamma_transformation 3/4<gamma<4/3
Affine transformation	Rotation (-25°~25°) Shift (15%) Shear (0.15) Horizonal flip
Wasserstein GAN (WGAN)	Refer to article [13].

3.1 Setup and Experiment

To collect the optical, current, voltage signals in the welding process, we build a signal acquisition system based on the welding equipment. The welding system and monitoring system is shown in Fig. 2. In this system, the welding equipment controls the whole welding process. And the signal acquisition system collect vision, current, voltage signals by sensors. Among them, the optical sensor uses CCD camera with 660nm filter. To collect voltage and current signals, we use Hall sensor. That is the hardware structure. Based on the sensors, we collect vision signal with frequency 5Hz, and for current and voltage signals, the collecting frequency is 10kHz.

The welding material used in this experiment was Q345, and the size of the plate was 300x100x10(mm). We focus on burn through, welding deviation, incomplete penetration and normal welding state. To obtain data from different welding states, we designed the experiment as Table 2 shows:

Table 2. The parameters of experiment

Welding state	Current(A)	Voltage(V)	Gap (mm)	Speed(mm/s)
Normal	220-280	18.2-28	1	5
Burn through	>260	>26	2	5
Incomplete penetration	<240	<18.8	0.5	5
Welding deviation	220-280	18.2-28	1	5

The gap is the distance between the two plates. For deviation, the teaching route is deviated from the center of the weld bead.

3.2 Computational Experiments

The modified F2GAN was trained on SJTU HPC studio using one Nvidia Tesla V100 GPU with 32G memory. We implemented the network using TensorFlow 1.13.1 and Python 2.6.1.

The modified F2GAN was trained with epoch 200 with similar scene data and select the best one from the history. After that we fine-tuned it with target scene data. Then we selected a model from the history and then augmented the data with the input of target scene.

The classification network was trained on NVIDIA GeForce RTX 2070 Max-Q. We implemented the network using torch 1.10.0, Python 3.7.10, sklearn 0.0.

3.3 Dataset preparation

To prepare data, we carried out two experiments. One is to prepare the similar dataset to pre-train the modified F2GAN, which had similar welding parameters and similar camera position. The other is to prepare dataset for defects detection.

In the preparing of normal welding dataset, we removed the data before and after arcing and aligned the vision signals with current and voltage signals on the timeline. Then we extracted the features from the signals by the methods in next part. And in preparing datasets for burn through, welding deviation and incomplete penetration, first, we aligned the vision images with current and voltage, and then selected the images and the corresponding current and voltage signals with defects. After the selection, we extracted the features from the signals. And the dataset from above process is shown in Table 3.

Table 3. The amount of the dataset

Datasets	Amount of weld beads	Welding states	Amount of data
Similar scene dataset	51	Normal	585
		Burn through	156
		Welding deviation	671
		Incomplete penetration	874
Target scene dataset	239	Normal	2477
		Burn through	448
		Welding deviation	949
		Incomplete penetration	1730

The similar scene dataset is firstly augmented by Gaussian noise and Gamma transformation shown in Table 1 to balance the dataset. Then the modified F2GAN will be trained on this dataset.

The dataset for defect detection is randomly split into training dataset, validation dataset and test dataset with ratio 3:1:1. The training dataset is used to fine-tune the modified F2GAN and to train the defects detection model after augmentation, the validation dataset is used to optimize the hyper-parameter and verify the performance of the model in training and the test dataset is used to test the final score.

And for the dataset of defects detection after splitting, it is unbalanced among categories. To balance the dataset, we augmented the welding states with defects until the amount of them is the same with normal state. For it is hard to evaluate the augmentation quality of current and voltage, we just repeated them to align them with augmented images. The augmentation methods for training datasets are shown in Table 1.

4.1 Data preprocessing

We collected signals from the experiment. And we split the current and voltage signals by 200ms to align with image at time. Thus, the dataset includes frames with 2000 point of current and voltage and 1 image.

4.2 Visual features extract and analyze

To classify the welding vision signals, researchers common use networks based on CNN which work well on feature extraction, image classification etc. In this article, we modify a ResNet based on ResNet50. The ResNet performs well in ImageNet for its skip connection reducing the information loss between the output and the input. For the same reason, the ResNet can be built deeper compared with other network to get a better performance. Those contributions enhance the ability of ResNet to extract relevant information from the input image. And in the ResNet series, ResNet50 achieves a balance between the weight and performance of model. Compared with ResNet101, ResNet152, the ResNet50 doesn’t degenerate much in accuracy and compared with ResNet34, the model weight does not increase much. Therefore, in this article we modify ResNet50 to extract information from the vision signals and finally classify them.

The input size of the modified ResNet50 is 100x100x3. And the fully connected layers are 256x4, which finally give the probability of four welding states.

After loading, the input image is processed by convolutional layers and pooling layers with skip connection repeatedly to extract features, and finally the features extracted is utilized by fully connected layers to predict the welding state.

4.3 Current and voltage features extract and analyze

The current and voltage could reflect some information in the welding process. And there is a lot of work on determining the welding states by current and voltage, which is proven to be helpful to the final classification [38].

And for zero drift caused by the equipment, we shifted the overall signals collected. That is, the drift value of the zero point was removed when processing the data, as follows:

$${x_{new}}={x_i} - {x_0}$$

x_new is the value after removing the zero drift, x_i is the value before removing the zero drift and x₀ is the average of drift value. After removing the drift value, we split the data into frames by 2000 points and calculate the features. The features used include avg, vpp, std, rms, wf, cf, sk and kurt. The definitions of them are shown in Table 4.

Table 4. The calculation definitions and formula

Methods	Definitions
avg	average
vpp	Maximum - minimum
std	Std Dev, Standard Deviation
rms	Root-mean-square
wf	Wave form factor
cf	Peak-to-average ratio (PAR)
sk	Skewness
kurt	Kurtosis

4.4 Feature normalization

The features extracted from current and voltage are varying in different range. However, to fuse the features, the weights of the features should be the same, thus we normalized the features according to following expression:

$$\begin{gathered} \hat {x}(t)=\frac{{x(t)-m}}{d} \hfill \\ m=\frac{{\sum {_{{t=1}}^{N}x(t)} }}{N} \hfill \\ d=\frac{{\sum {_{{t=1}}^{N}{{(x(t)-m)}^2}} }}{N} \hfill \\ \end{gathered}$$

After the normalization, the features extracted by different methods have the same weight. The average of the normalized features is 0 and the standard deviation is 1.

After the normalization of features, we use the augmentation model to augmented the dataset and use classification models to classify them. The results are following.

5.1 Augmentation Result

Figure 3 shows the augmented images by fine-tuned modified F2GAN and Wasserstein GAN (WGAN) [13]. It can be seen from the figure that the augmentation still has some shortcomings compared to original data: the diversity of the generation is not enough, the resolution of the augmented images is not satisfactory and the features of augmented images for welding deviation is not obvious. But compared to the data augmented by WGAN, the F2GAN behaves better. Then the augmented dataset is used to train the ResNet50 and fully connected layers to predict the welding state.

5.2 Classification Result

We trained different modified ResNet50 on datasets augmented by different methods. And the training results of models from different image datasets are shown in Table 5. And it is clear that the model trained on dataset augmented by modified F2GAN has the best performance. And based on the vision detection result, the fusion result with features of current and voltage is shown in Table 6.

It can be seen from Table 6 that the modified F2GAN augmentation has the best performance with accuracy 93.82%. And the combination with current and voltage can improve the accuracy to 96.60% while the precision, recall and F1-score are above 95%. In comparison, modified F2GAN model achieves the best augmentation result.

Table 5. The accuracy, recall, precision and F1-score of models on test dataset

Score	F2GAN	Repeat	Gray-scale	Affine	WGAN
Accuracy	0.9382	0.8714	0.8941	0.8941	0.9130
Recall	0.9039	0.8510	0.8652	0.8809	0.8822
Precision	0.9156	0.8137	0.8502	0.8511	0.8866
F1-score	0.9094	0.8211	0.8549	0.8600	0.8814

Table 6. The fusion result of the models

Combined with Voltage, Current	F2GAN	Repeat	Gray-scale	Affine	WGAN
Accuracy	0.9660	0.9521	0.9483	0.9533	0.9433
Recall	0.9533	0.9510	0.9420	0.9529	0.9227
Precision	0.9566	0.9237	0.9223	0.9328	0.9237
F1-score	0.9549	0.9361	0.9313	0.9421	0.9228

In this work, the modified few shot image generation model F2GAN augment the welding visual sensing data with defects. And based on the augmented data, we build classification models modified ResNet50, SVM, KNN, fully connected layers, Random Forest and XGBoost. The performance of the models shows that the augmented data based on the modified few shot image generation model F2GAN is more accurate than data from other augmentation methods. After the fusion of prediction from vision with current and voltage, the model can achieve an accuracy of 96.60%, which satisfies the needs of welding defects detection.

In summary, we implemented the following:

The augmentation based on the few shot image generation model F2GAN is achieved.
The F2GAN is modified for a better performance on the welding data. By using one-sided label smoothing, adding dropout, replacing activating function, adding Gaussian noise and transforming the grayscale, the modified F2GAN is suitable for limited welding datasets.
And based on different augmented datasets, we train the defects detection models. The performance of different models shows that the modified F2GAN has the best augmentation result with the accuracy 96.60%.

Author contribution Kanghong Zhu: proposed research methods and experimental plans, investigated, write and edited the paper. Weiguang Chen: performed the experiments. Huabin Chen: conceived and designed the study. Zhen Hou and Qingzhao Wang: technical support.

Funding This work was financially supported by National Natural Science Foundation of China under Grants 52175351, the Shanghai Sailing Program(21YF1432300).

Data availability Data sharing is not applicable to this article.

Declarations

Consent to participate Consent to participate is not applicable to this article.

Consent for publication The authors declare that they participated in this paper willingly and the authors declare to consent to the publication of this paper.

Competing interests The authors declare no competing interests.

Feng Y, Chen Z, Wang D, Chen J, Feng Z (2020) DeepWelding: A Deep Learning Enhanced Approach to GTAW Using Multisource Sensing Images. IEEE Trans Industr Inf 16(1):465–474. 10.1109/TII.2019.2937563
Wan X, Wang Y, Zhao D (2016) Quality monitoring based on dynamic resistance and principal component analysis in small scale resistance spot welding process. Int J Adv Manuf Technol 86:9:3443–3451. 10.1007/s00170-016-8374-1
Mirapeix J, García-Allende PB, Cobo A, Conde OM, López-Higuera JM (2007) Real-time arc-welding defect detection and classification with principal component analysis and artificial neural networks. NDT & E International 40(4):315–323. https://doi.org/10.1016/j.ndteint.2006.12.001
Lei Z, Shen J, Wang Q, Chen Y (2019) Real-time weld geometry prediction based on multi-information using neural network optimized by PCA and GA during thin-plate laser welding. J Manuf Process 43:207–217. https://doi.org/10.1016/j.jmapro.2019.05.013
Zhang Y, Gao X, Katayama S (2015) Weld appearance prediction with BP neural network improved by genetic algorithm during disk laser welding. J Manuf Syst 34:53–59. https://doi.org/10.1016/j.jmsy.2014.10.005
Wang X, Li R (2014) Intelligent modelling of back-side weld bead geometry using weld pool surface characteristic parameters. J Intell Manuf 25:6 1301–1313. 10.1007/s10845-013-0731-4
Zhu Y, Yang R, He Y, Ma J, Guo H, Yang Y, Zhang L (2021) A Lightweight Multiscale Attention Semantic Segmentation Algorithm for Detecting Laser Welding Defects on Safety Vent of Power Battery. IEEE Access 9:39245–39254. 10.1109/ACCESS.2021.3064180
Jin C, Shin S, Yu J, Rhee S (2020) Prediction Model for Back-Bead Monitoring During Gas Metal Arc Welding Using Supervised Deep Learning. IEEE Access 8:224044–224058. 10.1109/ACCESS.2020.3041274
Wan X, Wang Y, Zhao D, Huang Y (2017) A comparison of two types of neural network for weld quality prediction in small scale resistance spot welding. Mech Syst Signal Process 93:634–644. https://doi.org/10.1016/j.ymssp.2017.01.028
Wan X, Wang Y, Zhao D (2016) Quality evaluation in small-scale resistance spot welding by electrode voltage recognition. Sci Technol Weld Joining 21:5:358–365. 10.1080/13621718.2015.1115161
Liu T, Bao J, Wang J, Zhang Y (2018) A Hybrid CNN–LSTM Algorithm for Online Defect Recognition of CO2 Welding. Sensors 18:124369
Cai W, Wang J, Jiang P, Cao L, Mi G, Zhou Q (2020) Application of sensing techniques and artificial intelligence-based methods to laser welding real-time monitoring: A critical review of recent literature. J Manuf Syst 57:1–18. https://doi.org/10.1016/j.jmsy.2020.07.021
Zhang H, Chen Z, Zhang C, Xi J, Le X (2019) Weld Defect Detection Based on Deep Learning Method 15th IEEE International Conference on Automation Science and Engineering (IEEE CASE), Univ British Columbia, Vancouver, CANADA
Yang L, Wang H, Huo B, Li F, Liu Y (2021) An automatic welding defect location algorithm based on deep learning. Ndt & E International 120. 10.1016/j.ndteint.2021.102435
Kaestner L, Ahmadi S, Jonietz F, Jung P, Caire G, Ziegler M, Lambrecht J (2021) Classification of Spot-Welded Joints in Laser Thermography Data Using Convolutional Neural Networks. Ieee Access 9:48303–48312. 10.1109/access.2021.3063672
Yang L, Fan J, Liu Y, Li E, Peng J, Liang Z (2021) Automatic Detection and Location of Weld Beads With Deep Convolutional Neural Networks. IEEE Trans Instrum Meas 70. 10.1109/tim.2020.3026514
Ajmi C, Zapata J, Martinez-Alvarez JJ, Domenech G, Ruiz R (2020) Using Deep Learning for Defect Classification on a Small Weld X-ray Image Dataset. J Nondestr Eval 39:3. 10.1007/s10921-020-00719-9
Ajmi C, Zapata J, Elferchichi S, Zaafouri A, Laabidi K (2020) Deep Learning Technology for Weld Defects Classification Based on Transfer Learning and Activation Features. Adv Mater Sci Eng 2020. 10.1155/2020/1574350
Ma G, Yu L, Yuan H, Xiao W, He Y (2021) A vision-based method for lap weld defects monitoring of galvanized steel sheets using convolutional neural network. J Manuf Process 64:130–139. 10.1016/j.jmapro.2020.12.067
Guo R, Liu H, Xie G, Zhang Y (2021) Weld Defect Detection From Imbalanced Radiographic Images Based on Contrast Enhancement Conditional Generative Adversarial Network and Transfer Learning. IEEE Sens J 21(9):10844–10853. 10.1109/jsen.2021.3059860
Zhang Z, Wen G, Chen S (2019) Weld image deep learning-based on-line defects detection using convolutional neural networks for Al alloy in robotic arc welding. J Manuf Process 45:208–216. 10.1016/j.jmapro.2019.06.023
Li C, Wang Q, Jiao W, Johnson M, Zhang YM (2020) Deep Learning-Based Detection of Penetration from Weld Pool Reflection Images. Weld J 99(9 239S–245S). 10.29391/2020.99.022
Cao Vu D, Sekiya H, Hirano S, Okatani T, Miki C (2019) A vision-based method for crack detection in gusset plate welded joints of steel bridges using deep convolutional neural networks. Autom Constr 102:217–229. 10.1016/j.autcon.2019.02.013
Yang L, Fan J, Huo B, Liu Y (2021) Inspection of Welding Defect Based on Multi-feature Fusion and a Convolutional Network. J Nondestr Eval 40:4. 10.1007/s10921-021-00823-4
Dai W, Li D, Tang D, Jiang Q, Wang D, Wang H, Peng Y (2021) Deep learning assisted vision inspection of resistance spot welds. J Manuf Process 62:262–274. 10.1016/j.jmapro.2020.12.015
Zhang K, Shen H (2021) Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm. Appl Sciences-Basel 11:2. 10.3390/app11020576
Zhang W, Lang J (2021) Semi-Supervised Training for Positioning of Welding Seams. Sensors 21:21. 10.3390/s21217309
Liu J, Guo F, Zhang Y, Hou B, Zhou H (2022) Defect classification on limited labeled samples with multiscale feature fusion and semi-supervised learning. Appl Intell 52:7 8243–8258. 10.1007/s10489-021-02917-y
Pan H, Pang Z, Wang Y, Wang Y, Chen L (2020) A New Image Recognition and Classification Method Combining Transfer Learning Algorithm and MobileNet Model for Welding Defects. Ieee Access 8:119951–119960. 10.1109/access.2020.3005450
Jiao W, Wang Q, Cheng Y, Zhang Y (2021) End-to-end prediction of weld penetration: A deep learning and transfer learning based method. J Manuf Process 63:191–197. 10.1016/j.jmapro.2020.01.044
Sekhar R, Sharma D, Shah P (2022) Intelligent Classification of Tungsten Inert Gas Welding Defects: A Transfer Learning Approach. Front Mech Engineering-Switzerland 8. 10.3389/fmech.2022.824038
Clouâtre L, Demers M (2019) FIGR: Few-shot Image Generation with Reptile. ArXiv abs/1901.02199
Yan H, Li N, Jianfu Z, Weijie Z, Chen F, Liqing Z (2020) F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation. MM '20: Proceedings of the 28th ACM International Conference on Multimedia 2535–2543 10.1145/3394171.3413561
Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov R (2012) Improving neural networks by preventing co-adaptation of feature detectors. ArXiv abs/1207.0580
Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israel
Maas AL (2013)Rectifier Nonlinearities Improve Neural Network Acoustic Models
Mockus J, Tiesis V, Zilinskas A (1978) The application of Bayesian methods for seeking the extremum
Shin S, Jin C, Yu J, Rhee S (2020) Real-Time Detection of Weld Defects for Automated Welding Process Base on Deep Neural Network. Metals 10:3389

Download PDF

Journal Publication

published 11 Aug, 2023

Read the published version in The International Journal of Advanced Manufacturing Technology →

Editorial decision: Major Revisions Needed
26 Mar, 2023
Reviewers agreed at journal
10 Nov, 2022
Reviewers invited by journal
08 Nov, 2022
Editor assigned by journal
07 Nov, 2022
First submitted to journal
06 Nov, 2022

You are reading this latest preprint version

F2GAN based few shot image generation for GMAW defects detection using multi-sensor monitoring system

Status:

Journal Publication

Version 1

Abstract

Figures

1. Introduction

2. Detail Of Models

2.1 Data augmentation model

2.2 Classification model

3. Experiment And Dataset

3.1 Setup and Experiment

3.2 Computational Experiments

3.3 Dataset preparation

4. Multi-source Feature Extraction And Processing

4.1 Data preprocessing

4.2 Visual features extract and analyze

4.3 Current and voltage features extract and analyze

4.4 Feature normalization

5. Result And Discussion

5.1 Augmentation Result

5.2 Classification Result

6. Conclusion

Declarations

References

Status:

Journal Publication

Version 1