Coal–gangue recognition via Multi–branch convolutional neural network based on MFCC in noisy environment

doi:10.21203/rs.3.rs-1985537/v1

Download PDF

Article

Coal–gangue recognition via Multi–branch convolutional neural network based on MFCC in noisy environment

https://doi.org/10.21203/rs.3.rs-1985537/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 21 Apr, 2023

Read the published version in Scientific Reports →

You are reading this latest preprint version

This paper mainly studies the more accurate recognition of coal–gangue in the noise site environment in the process of top coal caving. Mel Frequency Cepstrum Coefficients (MFCC) smoothing method was introduced in the coal–gangue recognition site. Then, a convolution neural network model with three branches was developed. Experiments show that the proposed coal–gangue recognition method based on multi branch convolution neural network and MFCC smoothing can not only recognize the state of falling coal or gangue, but also recognize the operational state of site device.

In China, fully mechanized top coal caving technology is one of the important mining technologies in coal mine production, and coal–gangue recognition is the key to realize the intellectualization of fully mechanized top coal caving. The realization of intelligent coal caving process can solve the over–caving and less–caving problem which is easily caused by the artificial judgment, improve the quality and efficiency of mining, and reduce the impact of the adverse environment of the fully mechanized top coal caving face on the health of workers. State recognition system is defined from the perspective of pattern recognition, usually including three steps: data acquisition (DAQ), feature extraction and state recognition, among which, the latter two are of great significance and have a great impact on the accuracy of the final recognition. The research on DAQ of coal–gangue recognition can be traced back to the 1960s and 1970s, mainly including natural gamma ray^1,2, image recognition^3,4, sound signal analysis⁵, vibration signal analysis^6–8, sound and vibration signal analysis^9–11. Natural gamma ray method requires that there must be a lot of radioactive elements in coal and gangue, and the cost of sensor is high; Image recognition is greatly affected by on–site dust, so it is difficult to obtain an ideal image of coal–gangue in caving; Sound and vibration signal acquisition is simple, which is a common method at present.

Feature extraction is very important in coal–gangue recognition, the main purpose is to extract representative features, so as to describe the degree of caving and improve the recognition accuracy. To sum up, various signal processing methods, including Hilbert–Huang Transform(HHT)^6,7,10, wavelet packet transform¹¹, Mel Frequency Cepstrum Coefficients (MFCC)^5,9, time domain feature extraction⁸ and other variants, have been applied to extract effective features related to coal–gangue recognition from different perspectives. By using Mel scale based on the sensitivity of the human ear, MFCC can provide better results than the operations in time domain¹², which is commonly used for feature extraction in the frequency domain for different sound–based applications¹³. Su et al. combined log–Mel spectrogram and MFCC to complete the feature extraction of urban noise Urbansound8K dataset, and designed a four–layer CNN architecture based on DS theory, and indicated that the proposed method is suitable for environmental sound classification¹⁴. Rujoie et al. used MFCC and wavelet transform (WT) to obtain the better features of sound signal generated by the heart, and gave the diagnosis and determination of the severity of tricuspid regurgitation with K–nearest neighbors (KNN) classifier, which provided a basis for the early diagnosis of cardiovascular diseases¹⁵. Bharath et al. proposed a speaker recognition method in noisy environment based on MFCC and power normalized cepstrum coefficient (PNCC) technology with fusion strategy¹⁶.Jung et al. used short–time Fourier transform and MFCC to extract the features that can indicate lung sound and reveal the relationship with lung pathology, and employed the depth separable convolution neural network to effectively classify lung sound¹⁷. Nasef et al. reported a recognition technique to distinguish gender using MFCC features and Logistic Regression (LG) classifier, which can be carried out in the presence of background noise and different language, accent, age and emotional states¹⁸. MFCC is a powerful method to exhibit intrinsic characteristics of the sound signal and further to distinguish the sound induced by the impact of top coal and gangue on the hydraulic support. Therefore, we also chose MFCC for representing the features of coal and gangue during top coal caving.

Recently, multi–branch convolutional neural network (MBCNN) has been shown to perform very well in pattern recognition tasks. An end–to–end two multi–Branch CNNs with different convolution layers was proposed to simultaneously detect vehicles and license plates in a traffic system¹⁹. A MBCNN model based on synchro-squeezed wavelet transform was reported to detect rail crack, which usually has complex crack conditions and high operational noise²⁰. An embedded multi–branch structure based on 3D CNN was established to effectively identify the lung nodules and non–nodules of various shapes and sizes²¹. An end–to–end diagnosis framework was established by using MBCNN to realize bearing fault diagnosis under different working conditions²².

Sound can simultaneously contain the status information of conveyor, hydraulic support, shearer and other multiple device in fully mechanized mining face. It is an effective means to make the hydraulic support become an intelligent multiple device in the coal caving process. In addition, the MBCNN has the ability to deal with different features independently. Inspired by these ideas, a new multi–branch CNNs architecture was proposed here for identifying the state of top coal caving.

In the process of top–coal caving, the collected sound signal includes not only the sound of coal or gangue to be recognized, but also the noise induced by the operation of conveyor, shearer and other multiple device. MFCC features have strong robustness and are less affected by background noise, and multi–branch CNN has the ability to process different features independently. Inspired by these ideas, this paper proposes a new multi–branch CNN architecture with MFCC smoothing to recognize the state of top coal running under noise environment. The remainder of this paper is organized as follows. Section 2 briefly reviews MFCC feature extraction and the basic process of CNN for target recognition. Section 3 describes in detail the proposed MBCNN framework for coal–gangue recognition based on MFCC. Section 4 sets up an experimental platform to simulate the top coal caving on site and gives the performance evaluation index. Section 5 evaluates the performance of the proposed method on the noiseless dataset and simulated site dataset with noise. Finally, a brief conclusion is provided in Section 6.

The sound recognition method for control of top coal caving was proposed in this study based on MFCC and MBCNN. It mainly consists of two stages. The first stage is the preprocessing stage, which uses MFCC to obtain more accurate feature representation of sound wave induced by top coal hitting different device. The second stage is the recognition stage of coal–gangue, which uses MBCNN to achieve higher recognition accuracy under various working environments.

Preprocessing stage

MFCC is the coefficient of the short–time windowed signal obtained by fast Fourier transformation (FFT), which has better results than the time domain operation. MFCC feature extraction mainly includes six steps¹⁸: pre–weighting, framing, windowing, FFT, Meyer filter bank and discrete cosine transform (DCT), as shown in Fig. 1.In this work, each sound file took 4 seconds at a sampling rate of 22 kHz, the length of each frame is 2048, and the frame shift is 512.

The auditory characteristics of human ear come from the special structure of the cochlea, which allows people to distinguish various sounds in complex environment. The cochlea is essentially similar to a Mel filter bank, which is mainly composed of many band–pass filters. Here, triangular band–pass filter is adopted, and indexed filter m can be expressed as

$${H_m}(k)=\left\{ {\begin{array}{*{20}{l}} {0\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {}&{}&{}&{} \end{array}}&{}&{}&{\;\;k<f(m - 1} \end{array})} \\ {\frac{{k - f(m - 1)}}{{f(m) - f(m - 1)}}\begin{array}{*{20}{c}} {}&{}&{f(m - 1) \leqslant k \leqslant f(m)}&{} \end{array}} \\ {\frac{{f(m+1) - k}}{{f(m+1) - f(m)}}\begin{array}{*{20}{c}} {}&{}&{f(m)<k \leqslant f(m+1)} \end{array}} \\ {0\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {}&{}&{}&{} \end{array}}&{}&{}&{\;\;k>f(m+1)} \end{array}} \end{array}} \right.$$

where,$0 \leqslant m<M$, M is the number of filters, $\sum\limits_{{m=0}}^{{M - 1}} {{H_m}(k)} =1$.$f(m)$is the center frequency, defined as

$$f(m)=(\frac{N}{{{f_s}}})f_{{mel}}^{{ - 1}}({f_{mel}}({f_l})+m\frac{{{f_{mel}}({f_h}) - {f_{mel}}({f_l})}}{{M+1}})$$

where${f_l}$and${f_h}$are the lowest and highest frequencies of the filter frequency range, respectively, and${f_s}$is the sampling frequency. N is the length of the Discrete Fourier transform.${f_{mel}}$is the approximation of the Mel frequency unit, and$f_{{mel}}^{{ - 1}}$ is its inverse transformation (normal frequency in Hz), and they can be expressed as

$$\left. {\begin{array}{*{20}{c}} {{f_{mel}}=2595\lg 10(1+f/700)} \\ {f_{{mel}}^{{ - 1}}(f)=700({{10}^{{\raise0.7ex\hbox{${{f_{mel}}}$} \!\mathord{\left/ {\vphantom {{{f_{mel}}} {2095}}}\right.\kern-0pt}\!\lower0.7ex\hbox{${2095}$}}}} - 1)} \end{array}} \right\}$$

The Mel spectrum is obtained by superposing the product of power spectrum of each frame and Mel triangular weighted filter in numerical value, which can be given as

$$S(m)=\sum\limits_{{k=0}}^{{M - 1}} {E(k){H_m}(k)} \begin{array}{*{20}{c}} {}&{}&{}&{0 \leqslant m \leqslant M} \end{array} - 1$$

where Hm(k) is calculated with Eq. (1) and expresses the filter bank with M triangular Mel weighing filters.

Furthermore, the logarithmic power spectrum of the corresponding frequency band is computed, and then the discrete cosine transform (DCT) is performed to obtain the Mel cepstrum coefficients as follows:

$${c_{mel}}(n)=\sum\limits_{{m=0}}^{{M - 1}} {logS(m)\cos (\frac{{\pi n(m - 0.5)}}{M})} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} n=1,2,...,L$$

where L is the number of Mel cepstrum frequency coefficients, usually L ≤ M, Each sound signal is divided into F frames, and L MFCCs are calculated for each frame to form F×L MFCCs feature matrix.

Multi–branch convolutional neural network

In this work, the core of coal–gangue recognition is represented by analyzing the difference of sound generated by the top coal caving impacting different device. For their implementation, we design a multi–branch CNN structure for coal–gangue recognition system.

MBCNN Architecture

This paper focuses on the state recognition of falling coal or gangue in the process of top coal caving. In such case, the sound signals measured in different states are greatly affected by noise, which further challenges the separability of simply extracting state classification features using traditional CNN network structure. In order to make full use of the MFCC distribution characteristic of coal–gangue sound pressure signal, a MBCNN model with three branches was proposed to recognize coal and gangue more accurately in top coal caving, as shown in Fig. 2. Before feeding the full–connected layer, the feature vectors produced by the three branches need to be fused, we chose to the more general concatenation layer for feature fusion.

Training algorithm of CNN model

In the process of forward propagation, a convolutional neural network is mainly composed of five components: input layer, convolution layer, pooling layer, full connection layer and output layer.

CNN model is a supervised learning model, which needs to be learned under the supervision of sample tags. Therefore, the input data consists of sample X and sample label Y. For a C classification problem, the input of the model is represented by${\text{\{ }}X{\text{,}}Y{\text{\} =\{ }}{x_i}{\text{,}}{y_i}{{\text{\} }}^N}$, where N represents the number of samples of the input model, x_i represents the i–th sample, and y_i represents the class label corresponding to the i–th sample.

The convolution layer is the core component of CNN model, which realizes the idea of local connection and weight sharing through the convolution kernels (also named learnable filters). The convolution operation extracts features from the input feature map according to the convolution kernel size and the moving step size. The feature extraction process of convolution kernel is defined by

$${\mathbf{x}}_{j}^{L}=f(\sum\limits_{{i \in {M_j}}} {{\mathbf{w}}_{{i,j}}^{L} * {\mathbf{x}}_{i}^{{L - 1}}+{\mathbf{b}}_{j}^{L}} )$$

where$w_{{i,j}}^{L}$and$b_{j}^{L}$represent the weight matrix and bias matrix of the j–th convolution kernel in the L–th convolution layer, respectively. ${\mathbf{x}}_{i}^{{L - 1}}$represents the i–th feature map output by the L–1 convolution layer and ${\mathbf{x}}_{j}^{L}$represents the j–th feature map in the L convolution layer.${M_j}$is the number of convolution kernels. $f( \cdot )$represents the nonlinear activation function. Here, the rectified linear units (ReLU)²³ is used as the nonlinear activation function.

$$\operatorname{Re} LU(x)=\left\{ {\begin{array}{*{20}{c}} {0,}&{x \leqslant 0} \\ {x,}&{x>0} \end{array}} \right.$$

ReLU function is an unsaturated nonlinear function, which ensures that all outputs are non-negative, which can effectively reduce the risk of vanishing gradient in the process of training, and help to train the network faster.

The full–connected layer connects each neuron in the previous layer and the next layer to flatten the feature maps into one vector. In order to improve the performance of CNN model, here the activation function of neurons in the full–connected layer adopts ReLU function, so the output of each neuron is expressed as:

$$z_{j}^{L}=\operatorname{ReLU} (\sum\limits_{{i=1}}^{M} {{\mathbf{w}}_{{i,j}}^{L} * {\mathbf{x}}_{i}^{{L - 1}}+{\mathbf{b}}_{j}^{L}} ),\;\;j=1,2,...,N$$

where M and N represent the number of neurons in layer L–1 and layer L, respectively.

Optimization method of CNN model

BN is an optimization method proposed by Sergey Ioffe²⁴, a researcher at Google. By calculating the mean and variance estimates in the mini–batch of training sets, it adjusts the scale of input features, speeds up the training process of the model and enhances the generalization ability of the network. The BN layer is usually added after the convolution layer or full connection layer and before the activation unit.

The BN layer output is computed with the input moment${{\mathbf{\hat {x}}}_j}$normalized by the second variable${\sigma ^L}$^{24, 25}:

$${\mathbf{y}}_{j}^{L}=c{\mathbf{\hat {x}}}_{j}^{{L - 1}}+d$$

9.1

$${\mathbf{\hat {x}}}_{j}^{{L - 1}}{\text{=}}\frac{{x_{j}^{L} - {{\bar {x}}^L}}}{{\sqrt {{{({\sigma ^L})}^2}+\varepsilon } }}$$

9.2

where c and d are the scale factor and displacement factor respectively, which can be learned and adjusted during the training process.${\mathbf{x}}_{j}^{{L{\text{-}}1}}$is the j-th input in the L-1 layer. ${\bar {x}^L}$and${\sigma ^L}$are the mean and variance of the mini–bath in the L-1 layer respectively. $\varepsilon$is a constant for numerical stability, and$\varepsilon \leqslant 1$.

In this section, a novel method is proposed based on MFCC and MBCNN, which can carry out intelligent coal–gangue recognition in different fully mechanized mining faces, and the working procedures are briefly described as follows:

(1) LabVIEW software is used to develop data acquisition system to collect sound pressure signals of coal and gangue from different fully mechanized working faces and the sound pressure signals of shearer, conveyor and transfer machine and other site device, and signal segmentation is carried out for different working face, and background noise is added to obtain the final data samples for model training and testing.

(2)The discrete cosine transform of log–Mel spectral energy is used to convert the data obtained in step 1 into MFCC, which are the coefficients of short–time windowed signals obtained by FFT. The datasets are sorted according to the status of top coal caving for each hydraulic support.

(3) The recognition system based on MBCNN is constructed with the MFCC feature of the original sound pressure signal as the input and the corresponding state labels as the output. The MBCNN model was trained by using the sound signal obtained in Step 1. In addition, the influences of the number of branches, the number and size of convolution kernel layers and the gradient descent algorithm on the model recognition accuracy are analyzed offline, and the recognition model is determined as shown in Fig. 4. The model is composed of three branches, each of which is composed of two convolution layers, two pooling layers and one fully connected layer. After that, features of different layers are joined together by concatenate feature join method, then the softmax classifier is used to complete the state recognition. Here, the model calculates the prediction results of different operating states of each hydraulic support to give the most possible top coal caving state.

(4) We convert the test samples into one–dimensional MFCC feature data set and input the well–trained state recognition system to directly give the state recognition results of top coal caving.

In the proposed coal–gangue recognition system, each hydraulic support has its own MBCNN and only identifies the state of the top coal caving of that item, while the sound of all other device is regarded as noise. The system can identify and classify the running state of the top coal caving of all hydraulic supports. The advantage of the recognition system is that the noise can be filtered and the state of top coal caving can be correctly recognized even if several devices are running at the same time.

In this section, the construction of the experimental platform and the process of data acquisition are described, and the performance evaluation criteria of the coal–gangue recognition system are given.

Experimental setup

The top coal caving test rig is mainly composed of coal falling device, hydraulic support, acoustic sensor, data acquisition system, and noise source (such as: shearer, conveyor, transfer machine), as shown in Fig. 3. Sound signal is generated when coal or gangue falls and impacts the tail beam of hydraulic support, while the hydraulic cylinder and the stretchable beam and guard plates of the support almost have no influence on the experiment. Above the hydraulic support is the falling coal device, which is composed of coal storage bunker, inserting board and linear push rod. The inserting board can achieve the opening and closing of the coal caving window under the action of the linear push rod, and the working process of top coal caving is simulated.

Evaluation criteria

Our task is to recognize the sound generated by top coal caving and reduce the impact of noise. Therefore, the confusion matrix method is used to evaluate the model. Confusion matrix is an effective tool for multi target recognition system²⁶, which shows the comparison between output class and target class. The model evaluation based on the confusion matrix is given in Table 1. The output class is the recognition result of the model, and the target class is the real class.

Table 1

Model evaluation based on binary confusion matrix
Confusion matrix		Actual value
Confusion matrix		Positive	Negative
Recognition value	Positive	True Positive (TP)	False Positive (FP)
Recognition value	Negative	False Negative (FN)	True Negative (TN)

In order to evaluate the performance of the proposed method, the accuracy, precision and F1–score are used in²⁷:

$$Accuracy=\frac{{TP+TN}}{{TP+FP+FN+TN}}$$

$$\Pr ecision=\frac{{TP}}{{TP+FP}}$$

$$F1 - score=\frac{{2TP}}{{2TP+FP+FN}}$$

Experimental data and model description

Extensive experiments conducted on the coal–gangue recognition in top coal caving prove the superior performance of MBCNN. There are seven different sound classes in the coal–gangue dataset, including the noise induced by the device operation of the front conveyor, the rear conveyor, the right cutting of the shearer, the left cutting of the shearer and the transfer machine, as well as the state sound of falling coal and gangue that needs to be recognized. Table 2 gives a detailed description of state labels for noiseless dataset.

Table 2

Description of noiseless dataset conducted in coal–gangue recognition
States label	Audio event class	States label	Audio event class
0	Falling coal	4	Transfer machine
1	Falling gangue	5	Right cutting of Shearer
2	Rear conveyor	6	Left cutting of Shearer
3	Front conveyor

In this study, feature smoothing method of MFCC is adopted in MBCNN model. In order to illustrate the impact of this method on coal–gangue recognition performance, single–branch and double–branch CNN models are introduced, and their main structural parameters are described as follows:

Model 1: Single–branch CNN model is used with Conv = 256, Kernel size = 6, Dense (10) + BN.

Model 2: There is a double–branch CNN model with Conv = 256, Kernel size = 5 + 6, and Dense (10) + BN.

Model 3: There are three branches of CNN proposed in this paper, Conv = 256, Kernel size = 4 + 5 + 6, and Dense (64) + Dense (32) + BN.

The performance of coal–gangue recognition of the above three models in the process of top coal caving under different conditions will be illustrated in the following section. In addition, the initial learning rate of Adam optimizer was 0.0001, the number of epochs was 200, and the batch size was 64.

Experiment with noiseless dataset

In this section, we discuss the effectiveness of the MBCNN model for noiseless dataset on the coal–gangue recognition. Firstly, Figs. 4 and 5 respectively show the accuracy and loss curves of the training set and the testing set for different CNN models. For Model 1, the training accuracy and the testing accuracy reach the stable values after 69 epochs, and the testing accuracy is 98.50%. For Model 2, the training accuracy and the testing accuracy reach the stable values after 90 epochs, and the testing accuracy is 98.74%. For Model 3, the training accuracy and the testing accuracy reach the stable values after 68 epochs, and the testing accuracy is 99.28%. So, Model 3 converges to the stable value faster than the other two models, which means that training the Model 3 requires less time in practice. It is obvious from Fig. 5 that the loss function of Model 3 is smaller than that of the other two models, and the curve oscillation is the smallest before convergence, indicating that the robustness of Model 3 is the best on the noiseless dataset.

Then, the mean value of F1–score for each class obtained through five–fold cross validation is shown in Fig. 6, where the error bar represents the standard deviation of the stability of recognition performance. By comparing the recognition effects of the three models on 7 kinds of sound signals, it is found that Model 3 can fully recognize the labels 2–6, while the other two models cannot completely identify a certain class state, and the recognition result of Model 3 is superior to the other two models in the recognition state of labels 0–1. In addition, in the case of labels 2–6, the standard deviations of Model 3 are all zero, indicating that the MBCNN model proposed in this paper has a more reliable and stable performance. The results show that the proposed MBCNN can learn the linkage features between different classes from the original sound signals of different branches, so as to obtain higher quality recognition performance.

In a word, the proposed MBCNN model of three branches makes full use of the MFCC feature distribution of falling coal or gangue sound pressure on different time–frequency scales, and has higher sound recognition accuracy than the traditional single branch CNN and double-branch CNN on most class labels.

Site simulation experiment

In order to further verify the generalization ability of MBCNN in coal–gangue recognition during the process of top coal caving, according to Fig. 3, the coal and gangue caving as well as various noises are simultaneously mixed to simulate the site process of top coal caving, forming a simulated site dataset, which is called dataset 1. In order to obtain more abundant features that can represent different sound classes, the datasets with different branches of MBCNN are smoothed, that is, the two adjacent frames in dataset 1 are averaged to form a new dataset 2, and the three adjacent frames in dataset 1 are averaged to form a new dataset 3. For Model 1, the original dataset 1 is input into the convolutional neural network; For Model 2, the original dataset 1 is input into the branch where Kernel size = 6, and the smoothed dataset 2 is input into the branch where kernel size = 5; For Model 3, the first two branches are consistent with Model 2, and the smoothed dataset 3 is input on the kernel size = 4 branch. The simulated site dataset formed in this section has two classes of labels, label 0 represents the sound of falling gangue, and label 1 represents the sound of falling coal. The difference from the previous research^5,9−11 is that these two classes signals are collected when the rear conveyor, the front conveyor, the transfer machine, the shearer right cutting and the shearer left cutting are running at the same time, namely noise induced by the operation of site device is considered. The curves of accuracy and loss function are shown in Figs. 7–8, and the corresponding confusion matrixes are shown in Fig. 9.

As can be seen from Fig. 7, the testing accuracy of Model 1 reaches stable convergence after 96 epochs, and its convergence accuracy is 87.72%; the testing accuracy of Model 2 reaches stable convergence after 110 epochs, and its convergence accuracy is 90.35%; however, the testing accuracy of Model 3 reaches stable convergence only after 63 epochs, and its convergence accuracy is 92.98%. The loss function in Fig. 8 shows that Model 3 has the fastest convergence speed and the least oscillation amplitude change. Figure 9 depicts that the correct recognition rate of Model 1 and Model 2 in recognizing the falling coal is 89.9%, while the correct recognition rate of Model 3 in this class is 90.4%, and the correct recognition rate of Model 3 is the highest in the falling gangue.

On the simulation site dataset, the convergence accuracy of Model 3 is improved by 6.0% compared with Model 1, and 2.9% compared with Model 2, and Model 3 converges to the stable value faster than Model 1 and Model 2. At the same time, under the state of gangue caving, the recognition accuracy rate of Model 3 is 1.4% higher than that of Model 1, and 5.1% higher than that of Model 2. The results show that Model 3 has a better recognition accuracy ratio and requires less time in the practice of coal–gangue recognition. This is mainly because the feature smoothing processing method of three branches can effectively capture useful state feature information, so that Model 3 has better recognition performance in noisy environment.

Comparison with traditional classification algorithm

In order to prove the advantages of the proposed MFCC–MBCNN, we compared several traditional feature extraction and classification algorithms, including Hilbert–Huang Transform (HHT) combined with bimodal deep neural networks (DNN)¹⁰, wavelet packet transform(WPT) combined with fuzzy neural network(FNN)¹¹, MFCC and wavelet transform(WT) combined with K–nearest neighbors (KNN) classifier¹⁵, MFCC combined with self–attention Convolution neural networks(SACNN) and Logistic Regression (LG) classification algorithm¹⁸.

This section focuses on two kinds of experiments, one is target recognition on noiseless dataset, and the other is target recognition on simulated site dataset. Table 3 and Table 4 respectively list the F1– score comparison between other sound recognition methods and our method in these two kinds of experiments, and the comparison results adopt the five–fold cross validation method.

Table 3

F1–score (%) comparison with different methods on noiseless dataset
State Label	Methods
State Label	HHT–DNN10	WPT–FNN11	(MFCC + WT)–KNN15	MFCC– SACNN–LG18	Proposed MFCC–MBCNN
0	96.41 ± 3.29	82.07 ± 2.37	95.72 ± 2.94	93.62 ± 3.14	94.48 ± 2.48
1	95.05 ± 2.13	81.94 ± 3.07	94.73 ± 2.86	93.9 ± 2.93	94.61 ± 2.37
2	93.88 ± 1.23	80.92 ± 2.73	94.07 ± 1.56	97.92 ± 1.62	100 ± 0.00
3	93.27 ± 1.41	79.63 ± 2.38	95.84 ± 2.04	98.17 ± 1.27	100 ± 0.00
4	94.03 ± 2.62	81.53 ± 3.57	96.17 ± 2.99	100.0 ± 0.00	100 ± 0.00
5	93.47 ± 2.08	80.26 ± 1.86	95.94 ± 2.13	98.11 ± 1.07	100 ± 0.00
6	94.83 ± 1.95	81.25 ± 2.14	94.19 ± 1.82	100 ± 0.00	100 ± 0.00
Average	94.42 ± 2.10	81.09 ± 2.59	95.24 ± 2.33	97.39 ± 1.43	98.44 ± 0.69

Table 4

F1–score (%) comparison with different methods on simulated site dataset
States	Methods
States	HHT– DNN10	WPT–FNN11	(MFCC + WT)–KNN15	MFCC– SACNN–LG18	Proposed MFCC–MBCNN
Coal	82.69 ± 3.71	75.19 ± 4.38	81.26 ± 2.84	89.05 ± 2.47	91.37 ± 1.04
Gangue	80.83 ± 4.25	77.24 ± 4.71	81.53 ± 3.17	88.11 ± 2.73	91.49 ± 1.26
Average	81.76 ± 3.98	76.22 ± 4.55	81.40 ± 3.01	88.58 ± 2.60	91.43 ± 1.15

Reference 10 transferred knowledge from relevant data by transfer learning, which improves the weakness of limited labeled samples of deep neural networks. In reference 11, fuzzy logic reasoning was applied to the neural network to form a fuzzy neural network, which improves the self–learning ability of neural network. In reference 15, genetic algorithm (GA) was used to select sound signal features, and traditional K–nearest Neighbors (KNN) classifier was used to realize sound recognition. Reference 18 introduced convolutional neural network as a self–attentional filtering model, which consists of two blocks. The first block is a reduction module, and the second block is the stack of remaining modules. According to Tables 3 and 4, it is not difficult to find that the traditional target recognition methods^{10, 11, 15} has poor F1–score performance on both noiseless dataset and simulated field dataset. The main reason is that the traditional methods can only extract or select a small number of features at different time scales or frequency bands, especially for a single feature extraction method, different classes may be similar, which reduces the classification accuracy.

The method proposed in Reference 18 improves the shortcomings of traditional target recognition methods and can obtain better recognition accuracy on noiseless dataset. The proposed method in this paper can obtain 98.44% average F1–score on the noiseless dataset, and can obtain 100% F1–score for the following five cases, which is obviously better than other methods on simulated field dataset and has more stable performance. This is mainly because MBCNN combined with MFCC smoothing can learn useful state classification features from different frequency scales, which not only retains the information from the original features, but also smooths the noise. It can be seen that the MBCNN based on MFCC smoothing proposed in this paper provides a method for target recognition in noisy environment.

In view of the noise induced by the operation of shearer, conveyor and other device mixed in the sound signal of coal or gangue in the process of top-coal caving, a new multi–branch CNN structure based on MFCC was proposed to improve the recognition ability of coal–gangue in the noise environment in this study. The following are the main conclusions of this study:

(1) On the noiseless coal–gangue recognition dataset, the testing accuracy of the three–branch CNN model proposed in this paper is 0.79% higher than that of the traditional single–branch CNN. Although the accuracy is not much improved, the loss function and F1–score are significantly improved.

(2) On the simulation site dataset with various noises, the testing accuracy of MBCNN is 6.0% higher than that of traditional CNN, the recognition accuracy rate of falling gangue state is 5.1% higher than that of double–branch CNN, and the stable convergence time is 35 epochs less than that of traditional CNN and 47 epochs less than that of double– branch CNN.

(3) In addition, compared with other target recognition methods, the proposed method based on MBCNN and MFCC smoothing has better performance on both noiseless dataset and simulated site dataset in terms of F1–score.

In summary, the model constructed in this paper can provide effective feature information and good recognition accuracy in most cases. Therefore, as an extension of this work, we will try to use feature fusion methods to reduce the impact of noise on the correct recognition ratio of falling coal and gangue.

Data availability

The datasets used and analysed during the current study available from the corresponding author on reasonable request.

Acknowledgements

The authors gratefully thank anonymous reviewers for their valuable comments to improve the paper quality. This work was supported by National Natural Science Foundation of China (52174145) Natural Science Foundation of Shan-dong Provincial China (ZR2020MF101), Shandong Province Key Laboratory of Mine Mechanical Engineering open fund 2019KLMM202, SDUST Research Fund 2018TDJH101.

Author contributions

H.Y.J. conceived the study; Q.J.S., H. Y. J., and D.S.Z. were the principal investigators; K.D.G., H. Y. J., H.Z.S., and Z.J.L. directed the overall study design; K.D.G., H. Y. J., H.Z.S., and Z.J.L. performed the experiments; Q.J.S., H. Y. J., and D.S.Z. analysed the data; Q.J.S. wrote the manuscript. All authors discussed and interpreted the results.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Bessinger S. L., Nelson M. G. Remnant roof coal thickness measurement with passive gamma ray instruments in coal mines. IEEE Transactions on Industry Applications. 29(3), 562–565. https://doi:10.1109/28.222427 (1993)
Zhang N., Liu C. Radiation characteristics of natural gamma-ray from coal and gangue for recognition in top coal caving. Scientific Reports. 8(190), 1–9. https://doi:10.1038/s41598-017-18625-y (2018)
Sun J. P., Su B. Coal–rock interface detection on the basis of image texture features. International Journal of Mining Science and Technology. 23(5), 681–687. https://doi:10.1016/j.ijmst.2013.08.011 (2013)
Liu X. Y., Jing W., Zhou M. X., Li Y. X. Multi-Scale Feature Fusion for Coal-Rock Recognition Based on Completed Local Binary Pattern and Convolution Neural Network. Entropy. 21(6), 622.1–16. https://doi:10.3390/e21060622 (2019)
Xu J., Wang Z., Zhang W., He Y. Coal-rock Interface Recognition Based on MFCC and Neural Network. International Journal of Signal Processing. 6(4), 191–199. https://doi:10.3389/fpsyg.2013.00735 (2013)
Wang B., Wang Z., Xu J. A new coal-rock interface recognition method based on Hilbert marginal spectrum distribution characteristics. Journal of Computational Information Systems. 8 (19), 8137–8142. (2012)
Yang Y., Zeng Q. L., Yin G. J., Wan L. R. Vibration Test of Single Coal Gangue Particle Directly Impacting the Metal Plate and the Study of Coal Gangue Recognition Based on Vibration Signal and Stacking Integration. IEEE Access. 7, 106784–106805. https://doi:10.1109/ACCESS.2019.2932118 (2019)
Yang Y., Zeng Q. L. Impact-slip experiments and systematic study of coal gangue “category” recognition technology Part I: Impact-slip experiments between coal gangue mixture and top coal caving hydraulic support and the study of coal gangue “category” recognition technology. Powder Technology. 392, 224–240. https://doi:10.1016/j.powtec.2021.06.055 (2021)
Song Q. J., Jiang H. Y., Zhao X. G., Li D. M. An automatic decision approach to coal-rock recognition in top coal caving based on MF-Score. Pattern Analysis & Applications. 20(4), 1307–1315. https://doi:10.1007/s10044-017-0618-7 (2017)
Zhang G. X., Wang Z. C., Zhao L., Qi Y., Wang J. S. Coal-rock recognition in top coal caving using bimodal deep learning and Hilbert- Huang transform. Shock and Vibration. https://doi.org/10.1155/2017/3809525 (2017)
Liu Y. B., Dhakal S., Hao B. Y., Zhang W. P. Coal and rock interface identification based on wavelet packet decomposition and fuzzy neural network. Journal of Intelligent & Fuzzy Systems. 38(4), 3949–3959. https://doi:10.3233/JIFS-179620 (2020)
Zhang Y., Gravina R., Lu H., Villari M., Fortino G. PEA: Parallel electrocardiogram-based authentication for smart healthcare systems. J Netw Comput Appl. 117, 10–16. https://doi:10.1016/j.jnca.2018.05.007 (2018)
Uddin M. Z., Nilsson E. G. Emotion recognition using speech and neural structured learning to facilitate edge intelligence. Engineering Applications of Artificial Intelligence. 94, 103775.1–11. https://doi:10.1016/j.engappai.2020.103775 (2020)
Su Y., Zhang K., Wang J. Y., Kurosh M. Environment Sound Classification Using a Two-Stream CNN Based on Decision-Level Fusion. Sensors. 19, 1733.1–15. https://doi:10.3390/s19071733 (2019)
Ahmad R., Ali F., Saeid R., Elaheh R. K., Tirdad S. A. Classification and evaluation of the severity of tricuspid regurgitation using phonocardiogram. Biomedical Signal Processing and Control. 57, 101688.1–9. https://doi.org/10.1016/j.bspc.2019.101688 (2020)
Bharath K. P., Rajesh K. M. ELM speaker identification for limited dataset using multitaper based MFCC and PNCC features with fusion score. Multimedia Tools and Applications. 79, 28859–28883. https://doi.org/10.1007/s11042-020-09353-z (2020)
Jung S. Y., Liao C. H., Wu Y. S., Yuan S. M., Sun C. T. Efficiently Classifying Lung Sounds through Depthwise Separable CNN Models with Fused STFT and MFCC Features. Diagnostics. 11, 732.1–12. https://doi.org/10.3390/diagnostics11040732 (2021)
Nasef M. M., Sauber A. M., Nabil M. M. Voice gender recognition under unconstrained environments using self-attention. Applied Acoustics. 175, 107823.1–11. https://doi.org/10.1016/j.apacoust.2020.107823 (2021)
Chen S. L., Yang C., Ma J. W., Chen F., Yin X. C. Simultaneous End-to-End Vehicle and License Plate Detection With Multi-Branch Attention Neural Network. IEEE Transactions on Intelligent Transportation Systems. 21(9), 3686–3695. https://doi:10.1109/tits.2019.2931791 (2019)
Li D., Wang Y., Yan W. J., Ren W. X. Acoustic emission wave classification for rail crack monitoring based on synchrosqueezed wavelet transform and multi-branch convolutional neural network. Structural Health Monitoring. 20(4), 1563–1582. https://doi:10.1177/1475921720922797 (2021)
Zuo W. X., Zhou F. Q., He Y. Z. An Embedded Multi-branch 3D Convolution Neural Network for False Positive Reduction in Lung Nodule Detection. Journal of Digital Imaging. 33, 846–857. https://doi:10.1007/s10278-020-00326-0 (2020)
Wang G. X., Zhang M., Lei X., Hu Z. H., Li W. D., Cao J. J. A multi-branch convolutional transfer learning diagnostic method for bearings under diverse working conditions and devices. Measurement. 182, 109627. https://doi:10.1016/J.MEASUREMENT.2021.109627 (2021)
Nair V., Hinton G. E. Rectified linear units improve restricted boltzmann machines. In: 27th International conference on machine learning. 807–814. (2010)
Ioffe S., Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In: International conference on machine learning. 1–11. (2015)
Demir F., Ismael A. M., Sengur A. Classification of lung sounds with CNN model using parallel pooling structure. IEEE Access. (8), 105376–105383. https://doi:10.1109/access.2020.3000111 (2020)
Chandel N. S., Chakraborty S. K., Rajwade Y. A., Dubey K., Tiwari M. K., Jat D. Identifying crop water stress using deep learning models. Neural Computing and Applications. 33(10), 1–15. https://doi:10.1007/s00521–020–05325–4 (2020)
Deng X. Y., Liu Q., Deng Y., Mahadevan S. An improved method to construct basic probability assignment based on the confusion matrix for classification problem. Information Sciences. 340/341, 250–261. https://doi:10.1016/j.ins.2016.01.033 (2016)

No competing interests reported.

Download PDF

Journal Publication

published 21 Apr, 2023

Read the published version in Scientific Reports →

Editorial decision: Major revision
08 Mar, 2023
Reviews received at journal
06 Mar, 2023
Reviews received at journal
07 Feb, 2023
Reviews received at journal
26 Jan, 2023
Reviewers agreed at journal
25 Jan, 2023
Reviewers agreed at journal
24 Jan, 2023
Reviewers invited by journal
24 Jan, 2023
Editor assigned by journal
24 Jan, 2023
Editor invited by journal
07 Sep, 2022
Submission checks completed at journal
07 Sep, 2022
First submitted to journal
22 Aug, 2022

You are reading this latest preprint version

Coal–gangue recognition via Multi–branch convolutional neural network based on MFCC in noisy environment

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Proposed Models

Preprocessing stage

Multi–branch convolutional neural network

MBCNN Architecture

Training algorithm of CNN model

Optimization method of CNN model

Proposed Mbcnn For Coal–gangue Recognition Based On Mfcc

Experimental Setup And Performance Evaluation Index

Experimental setup

Evaluation criteria

Experimental Results And Analysis

Experimental data and model description

Experiment with noiseless dataset

Site simulation experiment

Comparison with traditional classification algorithm

Conclusions

Declarations

References

Additional Declarations

Status:

Journal Publication

Version 1