Tool wear assessment and life prediction model based on image processing and deep learning

Drilling is one of the most classical machining operations. Real-time monitoring of drill wear can effectively testify whether the product fails to meet the specifications due to drill failure. This paper proposes a tool wear assessment and life prediction model based on image processing and deep learning methods, which works effectively for small sample datasets and for low-quality images. The normal areas and worn areas of the drill bits are extracted using the U-Net network and traditional image processing methods, respectively. Moreover, the original dataset is classified using the migration learning technique. The wear level of a drill bit can be accurately evaluated through experimental tests. Testing results show that the proposed method is more convenient and efficient than previous methods using manual measurements. These results can be applied to real-time drill wear monitoring, thus reducing part damage caused by tool wear.


Introduction
In the current industry 4.0 era, hole machining is currently one of the most crucial machining methods [1]. Particularly in aircraft machining, there are nearly 1.5 million connections in a large modern aircraft, which means that a large number of holes need to be drilled [2]. During drilling, the surface temperature of the tool increases continuously with the rotation and feed movement of the tool, and the drill bit is prone to wear. The conservative approach in the actual machining method is to replace the tool based on experience. This method will misjudge the wear condition of the tool in many cases, replacing the tool when it still has a certain service life or replacing it when it is already severely worn. Replacement is not conducive to production efficiency improvements and product competitiveness. Therefore, the evaluation of the wear condition of the tool can provide a specific reference for the production personnel and effectively improve the quality of the machined parts.
At present, the image processing method is noncontact, mainly used to detect tool wear conditions. It can extract the area of interest without contact, thus helping scholars to establish a series of wear characteristics. Zhou et al. [3] used image processing to obtain the worn area of a tool and improved the accuracy of the subpixel edge detection algorithm based on Zernike moments. Huang et al. [4] used image projection and edge detection algorithms to calculate tool geometry. Sukeri et al. [5] used an image preprocessing operation to obtain edge images of the drill bits and to compare worn to nonworn drill edges. Grayscale transform, filtering, morphological image processing, and image segmentation are commonly used in image processing. Agarwal et al. [6] smoothed the image with the help of a Gaussian filter and binarized the image using Ostu's method [7], thus enabling efficient evaluation of tool wear width and worn area. The segmentation method using adaptive image thresholding does not obtain a region of interest (ROI) for more complex images such as uneven worn areas. Therefore, Bagga et al. [8] introduced two morphological operations, erosion and swelling, into the extraction step of the worn area and experimentally verified that the error between manual and automatic wear measurements is less than 5%.
In the past few years, machine learning technology has become one of the important methods of current research due to the development of hardware technology. Deep learning is a kind of machine learning which plays a crucial role in computer vision, natural language processing, medical diagnosis, and other related fields [9][10][11][12]. However, deep learning is very dependent on the amount of data, and training a newly built deep learning model usually requires thousands or even tens of thousands of images, which is highly difficult for the mechanical field to obtain so many images. Therefore, scholars have studied extracting features of the tool image to classify the degree of wearing through machine learning technology. Gu et al. [13] proposed inputting the wear features extracted from the preprocessed images into the support vector machine (SVM) to predict the degree of wearing of the drill bit. The study [14] used the B-ORCHIZ shape-based feature descriptor and SVM to classify the degree of tool wear. Some scholars also use deep learning algorithms to assist in evaluating the wear condition of the tool. Zhang et al. [15] used YOLOv3 to locate the tool defects, whose width and length were evaluated using the traditional image processing method.
In general, tool wear assessment methods mainly consist of two steps: feature extraction and wear degree classification or regression [16][17][18]. These methods often have a good effect on a tool wear evaluation because they construct a perfect eigenvector to characterize the wear condition of the drill bit. However, the previous studies also have some shortcomings. In feature extraction, these methods often require prior knowledge, and some algorithms cannot evaluate tools with complex defects. While using image processing methods to extract the worn and nonworn areas of the tool, the backgrounds of the grayscale image need to have a strong contrast with the target. For grayscale images with low contrast, neither threshold segmentation nor edge detection algorithms can achieve effective results. Therefore, it is necessary to propose a method that can still extract the tool contour with low contrasts between the tool backgrounds, and establish a highly robust wear quality evaluation system. This paper establishes an automatic tool wear assessment and life prediction model. The main contributions of this article are as follows: (1) An image dataset of drill wear is established by taking images through experiments, and it is divided into six categories according to the number of holes drilled by the drill bit. (2) To solve the low contrast between the foreground and background of the captured drill image, the U-Net semantic segmentation network [19] is used to segment the nonworn area of the drill bit. (3) The transfer learning method is applied to train and test the drill wear image dataset to solve the overfitting caused by the convolutional neural network (CNN) on a small sample dataset.
The remainder of this paper is organized as follows: Section 2 defines the experimental equipment and experimental process. Section 3 introduces the theoretical method of the tool wear assessment model and transfer learning model used in this paper. Section 4 conducts related experiments to verify the accuracy and reliability of the proposed method. Finally, Section 5 is the conclusion.

Experimental equipment
The benchtop drilling machine for drilling is the Z516. The maximum drilling diameter is 16 mm, and the spindle speed ranges from 280 to 3100 rpm. There are 4 series of spindle speeds: 280, 700, 1800, and 3100 rpm. The feed rate is kept as consistent as possible while drilling. The detailed parameters of the processing tool are shown in Table 1. The equipment used to take images of drill wear is shown in Fig. 1, which includes a light source, an industrial microscope, and a laptop. The pixel size of the original image collected by the visual device is 1920 × 1080.

Experimental process
A bench-top drilling machine and ten high-speed steel (HSS) drills with a diameter of 2.9 mm are used as the main tools of the experiment. The object to be drilled is a steel plate with a thickness of 8 mm, and the material is 20-gauge steel. So the feed required to drill a hole is 8 mm. Set the spindle speed of the bench drill to 700 rpm and keep the feed rate as consistent as possible during the drilling process. The processing parameters are shown in Table 2. After continuous drilling, it is observed that the wear of the drill bit is concentrated on the main cutting edge and the cross edge, and the worn area on the main cutting edge and the cross edge increases with the number of holes drilled. Therefore, the number of drill holes is used as the classification standard for drill bit wear. During the experiment, a fracture occurred when the drill bit reached the 57th hole. To be safe, the drill is defined to fail when drilling the 50th hole. Finally, the authors obtain a drill bit wear image dataset with 10 images for each category and a total of 60 images. Each image corresponds to a drill bit. The first category corresponds to new drills. The second to sixth category corresponds to 10, 20, 30, 40, and 50 holes drilled, respectively. Photos of drill bits corresponding to these six categories are shown in Fig. 2.
When evaluating the wear condition of the tool, the nonworn area of the tool cannot be segmented by the traditional image processing method due to the vague outline of the tool. Therefore, the U-Net network is used in this paper, and 35 belt parts are used in the literature. Annotated training images [19], 30 images consisting of 5 drills, are used as the training set, and the remaining 30 images are used as the testing set. The distribution of the datasets is shown in Fig. 3a, while Fig. 3b shows the distribution of the training and test sets for predicting bit life. The reason for the ratio being set as 8:2 will be discussed in Section 4.2.

Theoretical approach
Based on the collected tool wear image dataset, the first step is to apply the deep learning model to extract the tool contour. The worn areas are then extracted using image processing techniques. The intersection between these two binary images constitutes the final worn area. The wear evaluation method flow chart is shown in Fig. 4. Then, transfer learning techniques are applied to classify the extent of tool wear.

U-Net
Due to the influence of light and the shooting angle, the background and foreground areas of the obtained original image of the tool are not obvious. Therefore, it is difficult to segment the nonworn area of the tool using traditional processing methods. The classic image segmentation network   [20].
CNNs are widely used in many studies, such as VGG [21], GoogLeNet [22], and ResNet [23]. The last layer of CNN is fully connected, which compresses the feature map obtained by the original image through the convolution layer, pooling layer, and activation function into a vector. The output result is only the label of the image. Compared with CNN, U-Net is a pixel-level classification. The network is mainly composed of two parts, namely, the feature extraction part and the upsampling part. The feature extraction part mainly uses the convolutional layer, the pooling layer, and the ReLU nonlinear activation function. The sampling part mainly uses the transposed convolution layer and the convolution layer. Only each pixel in the tool image needs to be divided into two categories: foreground and background, namely the background and the nonworn area of the tool. Therefore, the number of channels finally output by the network is 2. The structure diagram of the network is shown in Fig. 5.
A binary image with only black and white colors is obtained after segmenting the nonworn region of the tool using U-Net. The binary image output after network segmentation has many jagged shapes, so the open operation is used as a morphological image processing tool as the open operation is usually used to eliminate image details smaller than the structural elements and smooth the image boundary [24]. The open operation is expressed as follows: where S is a structure that can be set to different shapes. F is the set of foreground pixels. First, structural element F corrodes S, and then the result after the corrosion of structural element F expands.
Denote the binary image as f(x, y), where (x, y) is a plane coordinate. The size of the image is M × N. As the gray value of the binary image obtained through network segmentation is not in the interval [0, 1], it is necessary to normalize the binary image, as shown in the following: The normalized binary image has a gray value of 1 for the white area and 0 for the black area, so it is possible to count the number of pixels in the nonworn area of the tool. The number of pixels in the nonworn area of the tool is defined as as follows:

Segmentation of tool worn area
Through the deep learning network, U-Net can segment the nonworn region from the original tool image. To extract the worn area, it is necessary to grayscale the original tool  In the process of acquiring the image, due to the sensor element or the external environment, the image transmitted to the computer usually has noise [25], so the mean filter is used to filter the grayscale tool, and the expression of the mean filter is shown as follows: where g(s, t) represents the grayscale image of the original image of the tool; h(x, y) represents the filtered image of the grayscale image; S xy represents the mean filter whose center is (x, y), and the size is p × q; and s and t are the row and column coordinates of the pixel contained in the filter when the filter slides to the specified position. After preprocessing, an appropriate threshold should be selected to classify the image at the pixel level. In this study, the worn area of the tool needs to be extracted using a threshold segmentation method, so only the pixels need to be assigned to two categories.
For grayscale images, an appropriate threshold can be selected by observing the grayscale histogram, as shown in Fig. 6. This method can always find a suitable threshold after many attempts for a single image. However, for a large number of images, it will drastically increase our workload. The Otsu method is an adaptive threshold segmentation method [26] that is wholly based on the histogram of the image and separates an optimal threshold according to the gray value to segment the image into two parts, the background and the target. Using (0, 1, 2, …, Q-1) to represent the Q gray levels in the image by selecting the threshold t, the input image is divided into two categories, C1 and C2. The gray pixel value of category C1 is in the range of [0, t], and the gray pixel value of category C2 is within the range [t + 1, Q-1].
By counting the gray value of the gray image of the tool, let n i represent the number of pixels of gray level i, so the histogram can be normalized: The between-class variance can be expressed by the following: where m G is the average grayscale of the image, which can be represented by Eq. (9); m(t) is the cumulative average value when the threshold is t, expressed by Eq. (10); and the  probability that an image pixel is assigned to class C1 can be represented by P 1 (t) , represented by Eq. (11).
Our purpose is to find an optimal threshold k that maximizes the interclass variance: Then, the grayscale image can be segmented by a global threshold k : Due to the very high temperature during tool drilling, the tool may undergo plastic deformation. After the tool undergoes plastic deformation, using the Otsu threshold segmentation method [27] to obtain the wear area will also segment this area, which is unreasonable. Therefore, use the intersection of the tool nonworn area extracted in Section 2.2 and the area obtained by Otsu threshold segmentation. The expression is shown in Eq. (14).
The preliminary tool wear image can be obtained by Eq. (14). However, the extracted wear image has the problem of holes in the center and rough edges. Therefore, the closed operation is used to process the preliminary obtained binary image of tool wear, which can be expressed by the following: After the closing operation, the final binary image of the nonworn area of the tool can be obtained. Similarly, Eqs. (2) and (3) are used to count the number of pixels in the worn area of the tool, and the number of pixels is recorded in the worn area of the tool as WEA.

Transfer learning model
As it is very difficult to obtain image datasets with large sample sizes in the mechanical field [28,29], for datasets with small sample sizes, directly using CNNs without initial weights to train small datasets will result in overfitting. However, transfer learning can first train a network on a large dataset, such as ImageNet [30]. Then, the pretrained model is used to train the small-sample dataset. There are already initial weights in the pretrained model when training the small-sample dataset, which can help us to fit more easily.
This paper uses the ResNet18 deep learning model after pretraining on the large dataset ImageNet [30] to tune a hyperparameter to train our tool wear dataset. The transfer learning method is illustrated in Fig. 7.

Experiments and results
To verify the accuracy of the model, experiments are set up to test its rationality. First, industrial microscopes and computer equipment are used to collect the original tool image. The specific machining parameters, the parameters of the tool, and the method of acquiring the images are described in Section 2. Then, deep learning methods are used to segment the nonworn area of the tool preliminarily. Moreover, Fig. 11 Histogram of two measurement methods. a No drilling, b ten drilled holes, c twenty drilled holes, d thirty drilled holes, e forty drilled holes, and f fifty drilled holes the tool image is processed by a computer using the method proposed in this paper to obtain the worn and nonworn areas. The hardware and software equipment for training U-Net are shown in Table 3. At the same time, to attain a tool life prediction, the influence of different proportions of the training set and testing set and the use of different deep learning models for the accuracy of life prediction are conducted.

Validation of the tool wear evaluation model
The nonworn area of the tool is segmented by a pretrained U-Net. Traditional image processing methods can extract the region of interest (ROI) for the worn area of the tool. The image processing process is shown in Fig. 8. All image processing operations are performed on MATLAB 2021b. To reflect the wear status of the tool, it is essential to calculate the ratio of the tool worn area to the entire tool area. The number of pixels in the worn area of the tool and the number of pixels in the nonworn area can be counted separately, as shown in Fig. 9. Since 5 drills are used for testing, and each drill drilled 10 holes to take a set of photos, 30 images are considered as the testing set.
The wear rate can be calculated based on the number of pixels in the tool wear and nonworn areas according to the following: Compare the wear ratios obtained by the manual annotation method for two of the tools with the wear ratios obtained by the proposed method in this paper. Let the abscissa by the number of drilled holes and the ordinate be the tool wear ratio. Plot the dot-line graph of the change in the tool wear ratio with the increase in the number of holes, as shown in Fig. 10. The comparison of the wear ratio obtained by manual measurement with the wear ratio obtained by the proposed method in this paper is shown in Fig. 11, where the abscissa is the number of tools, a total of 5 tools are used for experimental testing, and the ordinate is the wear ratio.
To evaluate the error between the manual measurement and the experimental measurement of the wear ratio, the error is defined as Eq. (17).
where test is the wear ratio using the tool wear assessment method, and label is the wear ratio using the manual measurement method.
A maximum error of 16% between the experimental and manual measurements can be observed through the tool wear evaluation model experiment. There are 76% of the sample errors less than 10% and 40% of the sample errors less than 5%.

Validation of the transfer learning model
In the life prediction step, the transfer learning model is established to classify the wear degree of the tool to attain a life prediction of the tool. Transfer learning training is more accessible than directly training a new deep learning model as it uses a pretrained weight model [31,32]. A deep learning model from 10 years ago usually increased the number of network layers, while the more modern ResNet uses a residual structure to fuse the shallow features of the network with the in-deep features, dramatically improving the accuracy of image classification. Given the small sample size of the tool wear dataset constructed in this paper, the ResNet18 network is used, as a ResNet with more layers will be more difficult to train and prone to overfitting. The main settings for training the transferred residual network are shown in Table 4. Moreover, data augmentation methods such as horizontal zoom, vertical zoom, reflection, and translation are used.
The R ratio of the training set and testing set is set to 9:1, 8:2, 7:3, 6:4, and 5:5 in the experiment. The experimental results obtained are shown in Fig. 12. It can be seen from the results that when the ratio is selected as 9:1, the highest accuracy is obtained in the experiment the most times. However, the fluctuation is significant, and the accuracy of one experiment decreases below 70%. When the ratio is selected as 8:2, although the number of times that the highest accuracy is obtained in the experiment is few, the result is relatively stable. The average accuracy of 10 experiments can reach 91.67%, which is over 90%. In image classification problems, using classification accuracy alone does not fully reflect the effect of the deep learning models. Therefore, the confusion matrix is used to evaluate the model's performance further. The confusion matrix is an n × n matrix, where n represents the number of categories of the classification, the elements on the diagonal of the matrix represent the number of correct predictions, and the elements on row i and column j represent the correct category but classify it as j. For the case where the proportion of the training set is 0.8, the confusion matrix is plotted, as shown in Fig. 12, through several experiments. The errors are mainly concentrated in the middle categories. The average classification accuracy exceeds 90%, and the minimum classification accuracy is also higher than 80%. Through experimental comparative research, the migration model used in this paper predicts tool life accurately and effectively (Fig. 13).

Conclusions
This article presents a novel tool wear assessment and life prediction method. First, a tool wear image library is created by taking images of drill bits on a drilling machine experimentally to evaluate the rationality of the method in this paper. Then, the entire contour of the tool is segmented using the semantic segmentation network U-Net. The Otsu threshold segmentation and morphological image processing methods are used for the worn area in the tool. The migration pretrained residual network helps predict the number of holes in the tool. It compares the impact of the proportion of different training sets and test sets on the classification accuracy, thereby indirectly attaining a life prediction of the tool. The wear ratio calculated by the ratio of the extracted two parts of the tool is at most 17% different from the manual measurement, and the error of more than half of the samples is no more than 10%. The tool life prediction model also has high accuracy, with an average accuracy of over 90%.
Based on the above characteristics of the tool wear assessment and life prediction model, the proposed method can be used in the real-time monitoring of machining tools. It does not require much prior knowledge from researchers and avoids the step of extracting tedious features of tool wear, which is one of the advantages of deep learning networks compared to traditional machine learning models. However, there are still some limitations in this study. On the one hand, the shooting angle, brightness of the surrounding environment, and outside temperature still need to be considered in the model. On the other hand, the plastic deformation of the tool affects the accuracy of extracting the worn area. Therefore, it is necessary to develop a more general and robust model. We will report these studies in future publications.
Funding The work is supported by the National Natural Science Foundation of China (grant no. 12172226).

Availability of data and material Not applicable.
Code availability The authors confirm that the code supporting the findings of this work is available from the corresponding author upon reasonable request.

Declarations
Ethics approval The authors declare the compliance with the ethical standards.