Coal quality prediction based on multi-feature fusion of flotation foam images

: Coal flotation monitoring cannot provide real-time feedback on the yield and ash of coal preparation products because it is influenced by the subjective nature of artificial judgment of coal preparation status and the lag of product quality testing of coal preparation. This paper aims to extract the texture, colour and shape features of floating foam images using various image processing methods, such as colour space, wavelet transform, greyscale co-occurrence matrix and edge operator, and to quantify the characterisation of various characteristic parameters on the basis of the indicative effect of floating foam characteristics on the quality of coal preparation products. The correlation between image features and the yield and ash of flotation products is studied, and a regression prediction model of coal preparation yield and ash was established by combining various image feature parameters using machine learning methods. Experimental results show that the proposed method can realise the real-time monitoring of coal mine flotation and effectively predict coal quality. This paper analyses the correlation between the characteristics of foam image and coal yield and ash and provides a yield and ash prediction method of flotation foam image based on multi-feature fusion. After extracting the characteristics of colour, texture and shape of the foam image, the foam image can be described, and the different characteristics of coal mass with different yields and ash fractions can be captured during flotation process to predict the yield and ash fraction of coal preparation. Sixty-five sets of flotation experiments are designed to verify the validity of the proposed method. Experimental results show that this method can predict the yield and ash content of the image during flotation and verify the effects of foam image characteristics on yield and ash during flotation, providing an effective guiding role in the production and ash prediction during floatation.


Research background
Flotation foam is an indispensable intermediate state variable in coal preparation, and the yield and ash of coal preparation products can be directly predicted by observing their shape, colour and bubble size. With the rapid development of computer and image processing technologies, flotation foam images can be automatically replaced with machine vision to monitor the flotation in real time by extracting the characteristic parameters of foam image and to accurately predict the yield and ash content of coal preparation products [1][2][3] .
In mineral processing, based on the analysis of the shortcomings of the existing flotation production process, Hu et al. [4] proposed an optimization method of the flotation process based on image processing, and described the main theories and solutions adopted by the academia in this field through the research results of three specific processes, including image pre-processing, foam feature extraction and production relationship model. Liang et al. [5] used the method of wavelet change to extract the equivalent size feature, and then took the feature as the basis, used clustering to classify the image, and improved the accuracy of the flotation foam image recognition. Liang X M et al. [6] used the Canny edge detection to extract the parameters of the bubble edge coordinates. According to the coordinate calculate parameters such as bubble area, perimeter, and centre position and convert to physical parameters. Tian et al. [7] obtained a good segmentation effect on coal slime flotation using a watershed algorithm. Chen et al. [8] applied a colour co-occurrence matrix to the texture feature extraction of flotation foam image. Considering that making coal slime flotation froth is frequently disturbed by different noises, Ning et al. [9] proposed an open-close reconstruction area filtering method to increase the area of structural elements for meeting the requirement of preserving image features during image segmentation.
In the field test process, to extract the dynamic characteristics of the froth layer in mineral flotation process, a measuring method based on the change feature detection of the bubbles regions between the adjacent frames was proposed by Chen et al. [10] . Support vector machine (SVM) was introduced to make the measurement problem into a classification problem. The field test results showed that the proposed method can detect the dynamic events of bubbles successfully. The classifier can give judgments of bubble stability based on the feature variation with quantitative measurement values. Xie Peihong et al. [11] proposed a segmentation model based on machine vision to solve the noise interference and bubble adhesion problems frequently encountered in foam images during coal slime flotation. Their proposed model has been applied to laboratory research and has achieved good results. Hosseini et al. [12] combined image analysis with neural network technology, established a characterisation model of the relationship between flotation foam size and flotation performance, and proposed a watershed algorithm based on adaptive markers, which is suitable for the segmentation of foam images and the measurement of bubble size under different operating conditions. The experimental results showed that the average size of bubbles immensely affects the system performance of flotation [12]. Coal flotation increases the coal quality by adding flotation agents to the coal slurry, filling the air and stirring to form a foam containing coal-powder particles. In the current flotation process, the monitoring of floating operation status through artificial observation is susceptible to subjective factors and the lag of coal preparation results cannot determine the flotation performance in real time [13] . The texture, colour and shape of flotation foam image are strongly related to the yield and ash of flotation products. This paper mainly studies the application of image processing and machine learning methods in coal flotation, including automatic extraction and quantification of foam image features, and establishes a correlation model between image features and coal quality to realise real-time and effective monitoring of coal flotation.

Flotation foam image processing and prediction
This method first extracts the three features of foam image, namely, texture, colour and shape characteristics, uses regression methods in machine learning to train the coal prediction model of the foam image, and inputs the flotation test image into the trained prediction model for obtaining the corresponding yield and ash indicators, as shown in Figure 1 Training images  Considering the coal quality and its corresponding flotation foam image bubble size, foam particle content, foam particle thickness degree, foam ash, foam viscosity and other factors, the following correlations can be referenced in accordance with the experience of artificial flotation. Foam ash content is related to the colour information, the thickness of particles in the foam is related to the shape information, and foam viscosity is related to the texture information. This paper uses the characteristics of colour, shape and texture to analyse the foam image. The overall description of image features containing 72 features is shown in Table 1. Colour feature is an important feature of foam images, and the colour change in foam image reflects the content of ash although its colour information is relatively a single natural image. This paper uses the three characteristics of hue, saturation and value (HSV) colour space statistical features, colour richness and colour name to characterise the colour information of the foam. . The information about the colour diversity of an image can be obtained by extracting the circular variance of the H channel R [13] , and the calculation process is expressed as: indicates the pixel value of in the position on the image H channel, and and represent the image width and height, respectively. The average and standard deviation on the S and V channels of the image are calculated. The average saturation represents the colour purity, the average on the V channel is called the colour brightness, and the colour purity and brightness of flotation foam image are highly correlated with the ash content of coal (as shown in Figure 2). The average of hues is not calculated because the hue averages cannot be associated with intensity attributes (low, high), and intensity attributes are measures of angle.
Low ( 0.14 ) High ( 0.17 ) Color richness Figure 3 Comparisons of the image color richness (2) Colour richness: This feature distinguishes a multicolour image from a monochrome, dark grey and low-contrast image. According to literature [14], an image is firstly transformed into the CIELUV colour space, the colour histogram of the image is calculated, and the resultant colour histogram is compared with the colour histogram of an ideal image to obtain the colour richness characteristics of the image, as shown in Figure 3. The colour in the ideal image is evenly distributed, that is, each piece of data in its colour histogram is equal.
(3) Colour name: Each pixel of an image can be classified into one of the following categories through classification, including black, blue, brown, gray, green, orange, pink, purple, red, white and yellow. The sum of pixels in the above categories illustrates the probability of colours appearing in the image and the colour statistics in flotation foam images. This paper uses the proposed method in literature [15] to classify each pixel in the bubble image, simulates the manner in which human beings label colour information in the image, and obtains the ratio of the number of pixels in each class with the total number of images as a colour name feature.

Texture feature
Textures are spatial arrangements of intensity and colour in an image, where texture captures perception (such as more pronounced in sharp images than in blurred images) and provide information about the subject of the image (such as the texture complexity and symmetry of the image reflecting its content). The image texture complexity of flotation foam can reflect the characteristics of foam size, roughness and viscosity, which is correlated with the flotation coal quality index. Therefore, the texture characteristics of wavelet transform, Tamura feature, grey-level co-occurrence (GLCM) feature, GIST feature and greyscale distribution entropy are used to describe the texture characteristics of a foam image. (1) Wavelet transform: Wavelet transform can measure the spatial smoothness and granularity of images [16] . A 2D discrete wavelet transform (2D-DWT) is developed to analyse the frequency components of the image, where the high-frequency portion can be intuitively linked to high edge density. The output of 2D-DWT consists of multilevel wavelet transforms (as shown in Figure 4), where each of them corresponds to a certain frequency of the original image. In the first-level transform decomposition, the image is divided into four parts, each with a quarter of the size of the original image, and the upper-left corner is marked as LL, indicating the low-pass part of the image. The three remaining parts are marked as vertical LH, horizontal HL and diagonal HH, representing the vertical, horizontal, and diagonal edge information of the image, respectively. A deep transform feature can be obtained by performing wavelet transform again on the decomposed LL part. In this paper, the three HSV channels of the foam image are three-level WT, and each level takes three features to represent the edge of the image, which are ， and ( ), where , and represent the horizontal, vertical and diagonal edge information of the image. Thus, the texture feature definition of WT shown in Equation 3 includes nine features (three levels for each of the three HSV channels).
Considering the span of pixel on a single spatial domain and the space area of for a single channel, that is, WT for each colour space channel, the three other characteristics are extracted by calculating the sum of average wavelet coefficients of each HSV channel at all frequencies. A total of 12 features are obtained.
(2) Tamura feature: Literature [15][17] presents the three important texture characteristics, namely, roughness, contrast and directivity related to human visual perception. This paper extracts the characteristics of the three aspects of the foam image.
Suppose that represents the greyscale portion of the foam image, the average in the neighbourhood of each pixel is first calculated, with the field's size being a power of two ( ). The calculation formula is expressed as follows: Secondly, calculate the difference between the areas where the corresponding position of the point does not overlap in the horizontal and vertical directions for each pixel , and the calculation formula is expressed as follows: (5) (6) Then, select the largest difference for each pixel: (7) Finally, the roughness is calculated: (8) where represent the width and height of image , respectively. The contrast of the image represents the texture quality, which is calculated as follows: where is the fourth-order of the average of image pixels , is the variance of image pixels, and is the coefficient . In practice, the contrast of the image is affected by the greyscale range and the polarisation of the black-and-white distribution on the greyscale histogram. The directivity of the image simulates the polarisation distribution in the edge direction, and the high directivity represents the even texture in the edge direction, and vice versa. First, Entropy of all the edge pixel situ distribution stakes in the image is calculated, and then the directionality of the image is obtained by . The edge of the image has a high directionality along a single direction when is zero . Conversely, the image has evenly distributed direction and has a low orientation when is one. Figure 5 shows a comparison of the Tamura features of the image. is the probability of positions and in the same region W. It is a broad texture analysis method used under the premise that the spatial distribution relationship between the pixels in the image contains the image texture information [18] . The four GLCM statistical characteristics of the HSV colour channel for calculating the foam image are contrast, energy, correlation and uniformity.
The darker the texture groove of the image is, the higher the clarity will be, and contrast is expressed as: (10) where indicates the number of pixels. The energy reflects the relationship between the uniformity of the greyscale distribution of the image and texture thickness. The GLCM feature element values are approximately equal, indicating that the energy is small, the texture is detailed, and vice versa. The expression is: (11) Correlation reflects the similarity between the greyscale of an image in horizontal and vertical directions, and the size of its value determines the similarity between greyscale correlations, which can be expressed as follows: (12) where ， .
Uniformity is the frequency where adjacent pixels have the same value, and the feature value is large when the greyscale symbiotic matrix is diagonally large, as shown in Figure 6 (d). The calculation formula is expressed as follows: (13) (4) GIST Features: Represent a low-dimensional scene that captures a set of perceived dimension characteristics, which are naturality, openness, roughness, dilation and severities through a Gabor filter [19] .
Low (3.61) high (4.91) Gray distribution entropy Figure 7 Comparison of image gray distribution entropy (5) Greyscale distribution entropy: The entropy of an image is a characteristic used to measure image uniformity. The foam image is converted to greyscale, the greyscale distribution (that is, the greyscale histogram of the block) is calculated for each pixel in the areas of pixels , and the distribution entropy is calculated. All entropy values are summed and divided by the size of the image to obtain greyscale distribution entropy. The more uniform the intensity of the image is, the lower the entropy will be. Suppose that is the probability that the difference between two adjacent pixels in the image is equal to , and the calculation formula for the greyscale distribution entropy is expressed as follows, as shown in Figure 7.

Shape features
Image shape features refer to the shape composition of visual elements in an image regardless of their actual content. Image shape features are used to describe the size and density of bubbles produced by the foam image, and the foam thickness in the image is measured. This paper describes the characteristics of the foam image using three characteristics, namely, boundary pixel, detail degree and average area.
(1) Boundary pixels: The shape of the image largely depends on the edge, that is, the point at which the brightness of the image shows discontinuity. Therefore, the boundary pixel feature detects the edge of the foam image using the Canny operator [20] and calculates the ratio of the number of pixels at the edge to the total number of pixels in the image.
low (segmented area = 224，area size = 0.446) high (segmented area = 9082，area size = 0.011) Figure 8 Details of the image and the average area zone size (2) Detail level and average area: Images can be split into multiple areas, that is, pixel sets that share common visual features. In this paper, the image is split using an average shift algorithm [21] , and two characteristics are extracted: The number of segmentations indicates the number of image segments. The standardised average expansion of the segmented area is the ratio of the average area to the total area of the divided area. The more the details are, the more fragments are split, as shown in Figure 8.

Coal property prediction
Machine learning has become a branch of artificial intelligence. We can fully exploit the inherent rules of provided data and predict the state of unknown data. Machine learning is mainly divided into supervised and unsupervised learning. This paper aims to reveal the relationship between the colour, texture and shape characteristics of foam images and the corresponding coal ash fractions. Thus, supervised machine learning is adopted. Figure 9 shows the model training through supervised learning, where the training data are marked as training. Supervised learning can predict unknown data using tag training data to build models. In this paper, the coal preparation production and ash prediction models of flotation foam images are trained using support vector machine (SVM) and random forest (RF).

Training data
Meachine learning methods

Model
Unknown data Prediction results   This paper uses three characteristics of colour, texture and shape to train and test the three prediction targets and divides 65 sets of experimental images into 80% training sets and 20% test sets to verify the role and contribution of the three characteristics of the three prediction targets (clean coal yield, clean coal ash and tailing coal ash). Using PLCC and SROCC results on the test set as performance indicators, the average result of 100 repetitions is taken as the final result.  Table 3, the experimental results of the best predictive performance of each feature are shown in bold, and the results show that different types of features immensely differ in the predictive performance of the three prediction targets. The colour characteristics have the best prediction effect on clean coal ash, indicating that the colour of flotation foam image can capture the characteristics of clean coal ash and proving that ash content and colour information have a strong correlation. The texture feature has the best prediction performance on the production of clean coal, showing that the texture feature of flotation foam image can better represent the roughness and uniformity of the image, indirectly reflecting the bubble viscosity during its emergence and representing the different characteristics of flotation foam under different clean coal production conditions to achieve better prediction performance. The shape feature has the best predictive performance on tailing coal ash, showing that the shape feature of the emerging foam image is correlated with tailing coal ash. Although the use of a single feature can provide a better predictive performance to the three prediction targets, the three characteristics have a comprehensive description of the foam image. Therefore, this paper uses the fusion of the three characteristics to predict clean coal production, clean coal ash and tailing coal ash to achieve a better forecast performance.

Conclusions
This paper analyses the correlation between the characteristics of foam image and coal yield and ash and provides a yield and ash prediction method of flotation foam image based on multi-feature fusion. After extracting the characteristics of colour, texture and shape of the foam image, the foam image can be described, and the different characteristics of coal mass with different yields and ash fractions can be captured during flotation process to predict the yield and ash fraction of coal preparation. Sixty-five sets of flotation experiments are designed to verify the validity of the proposed method. Experimental results show that this method can predict the yield and ash content of the image during flotation and verify the effects of foam image characteristics on yield and ash during flotation, providing an effective guiding role in the production and ash prediction during floatation.