Early apple bruise recognition based on near-infrared imaging and grayscale gradient images

Early apple bruises, especially those occurring within half an hour, usually have no external symptoms and are difficult to recognize. This study developed a fast and nondestructive detection method for early bruises based on near-infrared camera imaging and image recognition. A total of 31 apple samples were photographed on both sides of each apple. Grayscale images of the sound apples were captured using a near-infrared camera with a wavelength region between 900 and 2350 nm. Images (n = 62) of apples without bruises were collected. The identical apples were artificially damaged and photographed at two stages (0 h, 0.5 h) by the near-infrared camera, and a total of 186 grayscale images were collected. As the light spot on the surface of apples limits the accuracy in detecting defects, a compound image pre-processing method was proposed consisting of nonlinear grayscale transformation and frequency-domain image filtering techniques. Finally, the gradient image is obtained by taking the first-order derivative of the preprocessed image. Since bruise had distinct edges, the gradient grayscale images of apples are more favorable for bruise identification. The compound method obtained a 97.62% classification accuracy for nonbruised apples and apples with fresh bruises. The experimental results show that it is feasible to identify early bruises in apples based on near-infrared camera imaging and gradient grayscale images. In the subsequent study, the method will be further improved, especially the parameters involved in the algorithm could be adjusted to adaptive variables. In addition, the system will be explored to be suitable for online apple bruise detection.


Introduction
Apple defects can be divided into four main categories: mechanical damage, pathological disorders, contamination, and physiological disorder [1]. These defects reduce the quality of apples, affect the sales price, and may harm human health. Apple bruises are typical mechanical damage that can easily occur during picking, sorting, packaging, and transportation [2]. Identifying and classifying bruised apples as early as possible is conducive to the storage and circulation of apples. It can also prevent decaying and deteriorating apples from affecting other healthy apples [3]. Severe bruises can be identified with the naked eye or machine vision technology. However, early mild apple bruises, especially those occurring within half an hour, usually have no obvious external symptoms and are difficult to find [4]. Therefore, finding a fast, accurate, convenient and nondestructive discrimination method for apples with early bruises is necessary.
The physiological metabolism of the damaged part will be disordered over time, the release of ethylene will increase, the respiratory strength will increase significantly, and the soluble solid content will be relatively reduced [5]. In the initial stage of the bruises, under visible light, there is no apparent difference between the appearance of the bruised area and the healthy area; thus, a detection method based on visible light is subject to certain restrictions in this case. Due to the complex background and significant discrepancies in the surface color, slight and early bruises are challenging to be distinguished by image processing technology under visible light [6]. The penetrability of near-infrared light in a tissue sample is more excellent than that of visible light [7, 1 3 8], and it is pretty sensitive to changes in water content and soluble solids in the sample [9]. Therefore, a near-infrared image of an apple can more easily reveal bruising than a visible light image, which makes it possible for near-infrared imaging to identify early bruises in apples. Methods based on near-infrared imaging, including hyperspectral imaging and near-infrared camera imaging, have developed rapidly in recent years and are appropriate for the nondestructive detection of early bruises in fruit.
Hyperspectral imaging technology integrates imaging, spectroscopy, chemometrics, and other optical sensing technologies [10]. It is a nondestructive, noncontact detection technology that allows the simultaneous acquisition of sample spatial and spectral information [11]. A typical example is the use of hyperspectral imaging to detect aflatoxin in pistachios. Researchers successfully detected aflatoxin contamination using LDA and QDA methods based on hyperspectral imaging [12]. In recent years, hyperspectral imaging technology has been widely used to identify fruit defects, including apple bruises. For example, Tan et al. [13] used hyperspectral imaging combined with principal component analysis to identify early bruises in apples with a recognition rate of 99.9%. Luo et al. [14] used hyperspectral and multispectral reflectance imaging technology to identify apple bruises within one hour, and the recognition rate reached 99.2%. Compared with other technologies, hyperspectral imaging technology reflects not only external characteristics such as the texture and size of the measured object but also internal quality information such as the chemical composition content of the object [15]. Since the spectral and image information is sufficient, hyperspectral imaging technology usually has a higher recognition rate. However, the three-dimensional data cube in this method has considerable redundant information, which increases the complexity and time of data processing [16]. Additionally, the calibration models are usually not independent of the particular calibration samples, measurement technique, or the setup of specific experiments.
Near-infrared cameras have the advantages of small size, light weight, high luminous flux, and convenient data collection [8]. Aleixos et al. [17] developed an imaging system based on a multispectral camera that can obtain visible light and near-infrared images from the same scene. The system can detect the size, color, and presence of defects in citrus and correctly classify lemons and citrus. Kleynen et al. [18] proposed a method based on quadratic discriminant analysis that selects the best filter wavelength and constructs a multispectral imaging system to detect multiple defects in "Jonagold" apples. The combination of filters with three different wavelengths is sufficient to detect most bruised apples. The combination of filters with four different center wavelengths has a classification accuracy of 100% for apples with severe and moderate defects. However, filters are usually expensive, and it is difficult to find matching filters during experiments. A near-infrared camera developed with T2SL material has a smaller dark current, lower noise, and higher sensitivity. It offers great target recognition and detailed expression under environmental conditions such as night vision and low visibility. This study used a T2SL near-infrared camera with a wavelength range of 900-2350 nm to collect near-infrared images of bruised apple samples, and the experimental setup does not contain any type of filter.
Based on the above discussion, the objective of this work was to establish an early apple bruise detection method based on near-infrared camera imaging. In addition, in order to avoid modeling, the recognition method is based on image processing, including threshold segmentation, gray-scale transformation, image frequency-domain filtering, gradient image extraction, etc.

Samples and bruise manufacturing device
As one of the major varieties of apples, Red Fuji apples have a wider peel variation than other apple varieties. Thus, it is harder to detect early bruises on Red Fuji apples, and apple bruise recognition experiments using Red Fuji apples are more representative. This experiment used 31 Red Fuji apples from Yantai Qixia as the research object. Since the difference in position 180° apart along the horizontal line was even greater than that of different apples, the relative position of every apple was photographed separately, and the sample set can be regarded as 62. Before the experiment, the apples were placed at room temperature for 12 h, and the surfaces of the apples were disinfected with 75% alcohol.
The purpose of this study is to detect bruised apples that are damaged within half an hour. Early bruises (within 0.5 h) can only be artificially created because the naked eye can not distinguish them. An inverted pendulum was used to create apple bruises, and the schematic diagram is shown in Fig. 1. The device is mainly composed of a 50 cm long pendulum rod and a steel ball at the end of the rod. The diameter of the steel ball is 18 mm, and the weight is 0.085 kg. The pendulum was raised to an angle of 45° with the vertical direction and released to make the steel ball impact fixed apples. According to the impulse theorem ( Ft = mΔv ), the impact impulse that the apple surface suffered is about 0.14 N • s . The bruised area is small and approximately 0.8 cm 2 . Commonly, the apple surface is intact and has a slight depression that is not easily recognized by the naked eye.

Near-infrared camera imaging system
The apple images were collected with a near-infrared camera (Xeva-2.35-320, Xenics, Leuven, Belgium), which uses a thermoelectrically cooled T2SL detector with a 320 × 256 pixel resolution and a 900-2350 nm wavelength response. The camera lens is from the KOWA Group of Japan Kowa Co., Ltd., which has manual focus and aperture functions. The maximum aperture is F1.4.
The system consists of a near-infrared camera, a tripod, and three halogen tungsten lamps, as shown in Fig. 2. The near-infrared camera is fixed by a tripod, and the lens is vertically downward to the apple. As the apple has an irregular spherical shape and the camera has no light source, three tungsten halogen lamps are added to minimize the impact of uneven illumination while collecting images. The three lamps are placed at an angle of 120° with the apple sample as the center. The angle between the irradiation direction and the vertical direction is approximately 45°, and the lampshade is wrapped with a soft cloth to make the light more uniform.
Images were collected by the built-up system, and near-infrared images (n = 62) of nonbruised apples were collected first. Then, the device shown in Fig. 1 was used to create samples with bruises, and near-infrared images (n = 62) of the apples were acquired immediately after the bruise appeared. Near-infrared images (n = 62) of the same bruised apples were collected again 30 min later.
Compared with visible light, near-infrared light is more sensitive to changes of organic components in apple samples. Therefore, near-infrared images of apples can more easily reveal bruises in the early stage of bruises than visible light images. In most cases, it is challenging to distinguish bruises with the naked eye in the visible light images of apples bruised for less than 30 min. We used a mobile phone to take an image of the apple immediately after the bruise occurred, as shown in Fig. 3a. At the same time, a near-infrared image of the same apple was taken using a near-infrared camera, as shown in Fig. 3b. The region of the bruise in the apple is visible in the near-infrared image; however, the region of the bruise in the visible light image cannot be found.

Image processing method for early apple bruise recognition
This paper presents an image processing method for the detection of bruised apples. The compound method contains two-level image preprocessing, including nonlinear grayscale transformation and frequency-domain image filtering. In order to eliminate the interference of background information on the target apple image and extract the region of interest, i.e., subentity, a mask was pre-made using a threshold segmentation method. After background processing, the image is subjected to nonlinear gray transform, high-pass filtering and image derivation. In the binary images of bruise edges obtained, the number of pixels with a gray value of 1 determines whether the apple is damaged or not. When the number of pixels is greater than a given threshold, the sample is determined to be damaged. This image processing-based approach for bruise identification is a non-modeling approach.

Nonlinear grayscale transformation
The gray value of the bruised part of the apple is lower than that of the nonbruised part, so the low gray areas in the apple image can be used to identify the bruises. In the process of extracting the edge of the bruise, the change of gray value in the target area caused by uneven illumination will affect the identification and extraction of the bruises. For example, the light spot caused by the supplementary light source in the target image is the region with a high gray value. The light spot would be easily recognized as a bruise when the edge recognition method was used to extract damage [19]. Therefore, it is essential to eliminate light spots in the original apple image. A two-level image processing method was designed to solve this problem.
First, a piecewise-linear grayscale transformation was used to weaken the light spot with a high gray value.
The grayscale level transformation uses a transformation function to map the input pixel gray value f (x, y) to a new gray value g(x, y) , which belongs to an image enhancement technology, as in Formula (1).
The " T " indicates the mapping relationship between g(x, y) and f (x, y) , as shown in Formula (2) and (3).
In order to reduce the gray value of the light spot without affecting the contrast of the low gray area, the average gray value aver1 of the apple image is calculated before the grayscale conversion. The average gray value was set as a threshold, the pixels with a gray value that was smaller than the average gray value remained constant, and the gray value greater than the average gray value was compressed. The grayscale transformation function is shown in Formula (2), and the value of aver1 is calculated as Formula (3).
The " n " is the number of image pixels and the " a " is a proportional coefficient.
Nonlinear transformations, such as logarithmic transformations, can increase the smoothness of image processing. The logarithmic transformation is expressed as follows.
The " b " is a proportional coefficient. The gray value of the apple image is adjusted to ensure that the average brightness of the transformed images of different apple samples remains consistent. The average gray value of h(x, y) is set to be aver3 , and the gray value of each pixel is h i before adjustment. The average gray value of the apple image is aver4 , and the gray value of each pixel is h ′ i after adjustment. Formula (5) can be obtained. The above is the first level of image processing, and the contrast of the image is improved after the processing.

Frequency-domain image filtering
The image is further processed by a second-level processing method. A high-pass filter in the frequency domain is designed to further weaken the influence of uneven illumination and improve the contrast of the bruised area. First, the Fourier transform is performed on the image signal. In the frequency domain, the image signal can be regarded as composed of high-frequency components and low-frequency components. According to Retinex Theory, an apple's nearinfrared image signal can also be regarded as composed of illumination components and diffuse reflection components LAND [20]. The illumination component of the sample caused by uneven illumination corresponds to the low-frequency signal, and the diffuse reflection component corresponds to the high-frequency signal, which carries the primary characteristic information of the sample. Therefore, a high-pass filtering method is selected as the second level of image preprocessing.
The transfer function of the selected Butterworth filter is shown in Formula (6).
where D 0 > 0 is the cut-off frequency of the filter, and D(u, v) = √ u 2 + v 2 is distance from the origin to point (u, v) in the frequency domain coordinate system, u and v are the coordinates. " N " is the order of the filter.
After frequency-domain filtering, the bruise edge is more obvious, and the contrast of the image is improved, which is conducive to bruise identification and extraction.

Image derivation to obtain the gray gradient image
Due to the low grayscale region caused by bruises having a greater grayscale change ratio at the edge of the low grayscale region, a grayscale gradient image by the first derivative is acquired to manifest this character. Two images, including a horizontal grayscale gradient image and a vertical grayscale gradient image, are acquired by the first derivative. The absolute values of the two images are calculated and then added. Where the gray value of the original grayscale image has a small change ratio, the gray value of the corresponding position in the grayscale gradient image will be close to zero. In contrast, the gray value of the corresponding position in the grayscale gradient image deviates from zero. The greater the rate of change in the original image gray value, the greater the gray value of the corresponding position in the gradient grayscale image. Then, threshold segmentation is used to binarize the gradient image, which can well express the edge characteristics of the gradient grayscale image. In the process of identifying the edge of the bruised area, it is necessary to exclude the influence of the profile of the apple. In the background processing process, the binarized image of the original grayscale image is obtained, which is a mask. The corrosion-treated mask is multiplied by the new binarized image to eliminate the entire edge of the apple, and then the binary image only shows the edges of the bruise. The bruise identification program calculates the number of pixels with a gray value of 1 in the binary image and determines whether the image is an image of a bruised apple.

Results of the image processing flow
A low gray area can appear in the apple image for two reasons: the apple tissue in the region is damaged, or the region cannot receive enough light while the apple image is collected. As the surface of the apple is not a regular spherical shape, some images of nonbruised apples have low gray areas after image preprocessing. Such sound apples would be classified as bruised apples easily using the threshold segmentation method directly, which is an essential factor that lowers the recognition rate of the threshold segmentation method. The low gray areas of nonbruised apples have no distinct edges, unlike apple bruises. The recognition of apple bruises using gradient images can make good use of this distinction. Therefore, the image gradient is calculated according to subsection "Image derivation to obtain the gray gradient image". The bruise edge calculation for an apple sample is shown in Fig. 4.
Every apple image was processed by nonlinear gray transform, high-pass filtering, and derivation. We used an image of a bruised apple as an example, and the whole process is shown in Fig. 5. Figure 5a shows the original images, Fig. 5b shows the images after nonlinear grayscale transformation, Fig. 5c shows the images after frequency-domain image filtering, and Fig. 5d shows the images after derivation. The region of bruises could be visualized by binarization of the image, as shown in Fig. 5e. Then, Fig. 5f can be obtained by further corrosion processing. The number of pixels with a pixel value of 1 is counted to determine whether the image is from a bruised apple.

Identification results
The filtered images were derived and binarized in turn. If it was an image of a sound apple, only the outline of the apple could be seen. Figure 6 randomly lists the identification results of 20 sound apples in the test set, and all samples were correctly identified.
If there was an image of a bruised apple, the outline of the apple and the outline of the bruise was all visible. Image processing was performed on some bruised apple images in the test set, and the results are shown in Fig. 7. The samples of bruised apples were all correctly identified. In image processing, there were two key parameters: one was the cut-off frequency of the frequency domain filter, and the other was the threshold for binarization after image derivation. The parameters were set by 20 images of sound apples and 40 images of bruised apples. The remaining 126 apple images, including 42 images of nonbruised apples and 84 images of bruised apples, were used to verify this method. The test results are shown in Table 1.
The image recognition method commonly used is threshold segmentation without derivation, and the threshold is determined according to the grayscale histogram. To compare with the method in this paper, all the images were preprocessed in the same way, and a threshold segmentation method was used to recognize the bruised region instead of extracting the bruised edge. The same 126 pictures were used for verification. The results are shown in Table 2.
The nonbruised apple image is marked as 0, and the damaged apple image is marked as 1. The binary classification confusion matrix of the method with derivation is shown in Fig. 8a. The accuracy, precision and recall of the method are 97.6%, 98.8% and 97.6%, respectively. The binary classification confusion matrix of the method without derivation is shown in Fig. 8b. The accuracy, precision and recall of the method are 92.9%, 94.1% and 95.2%, respectively. Therefore, using an image gradient to identify apple bruises can effectively improve the recognition rate.  5 Two-level image preprocessing and image derivation for a bruised apple: a Image after denoising; b Image after nonlinear grayscale transformation; c Image after frequencydomain filtering; d Gradient grayscale image; e Binary image; f Binary image without apple edge, this image would be used for computer recognition Fig. 6 Identification results of 20 sound samples in the test set: The odd columns were the original images, and the even columns were the corresponding binary images. These images allowed us to deter-mine whether the apple was damaged or not. Since no bruised edges were present, these samples were identified as undamaged apples

Discussion
Hyperspectral imaging technology has been used for the detection of apple bruises. Wenkai Che et al. [21] used Vis-NIR hyperspectral imaging for bruise region extraction of apple. The mean accuracy of pixel-based Random Forest models reached 99.90%, slightly higher than the recognition rate of the method of our work. Hyperspectral images of 60 bruised apples at three stages (0 h, 12 h, and 18 h after bruises) were obtained by the HSI system with a spectral range of 400-1000 nm. In our experiment, nearinfrared images of bruised apples at two stages (0 h, 0.5 h) were obtained by the near-infrared camera with a spectral range of 900-2350 nm. The experiment mainly focuses on detecting early apple bruises within 0.5 h, which is more complicated. A similar study about apple bruise identification using hyperspectral imaging was completed by Wei Luo et al. [14], which has been mentioned in the introduction. Identification of apple bruises (1 h) was achieved using hyperspectral imaging combined with PCA methods. Hyperspectral imaging technology has an advantage in terms of the amount of information. However, it also increases the complexity of information processing. In contrast, the method used in this study has less redundant band information and can significantly improve the data processing speed.
The near-infrared camera is a better option for improving the efficiency of sample data collecting. Mahmut HEKIM et al. [22] have completed a study on apple bruise identification using image processing algorithms based on nearinfrared camera imaging. 1200 images were taken from 200 apples by using a near-infrared camera. Different algorithms were used to establish models, and the CNN model had a maximum recognition rate of 98.33%. The study did not consider the generation time of bruises and the degree of bruises. On the other hand, although the calibration method-CNN deep learning can achieve high accuracy and recognition speed, there are still many aspects to be considered, such as the establishment of models usually needing a long time and thousands of samples. The apple bruise recognition method proposed in our study required only a small number of samples to set the relevant parameters for image processing. The odd columns were the original images, and the even columns were the corresponding binary images. Due to the appearance of bruised edges, these samples were identified as damaged apples Spectroscopic techniques combined with chemometrics for detecting and classifying fruit products are common and successful. In a recent study [23], the apple bruise severity was tested by using NIR spectroscopy, and the classification accuracy of the method was up to 96%. Nearinfrared analysis generally requires building a model in advance using chemometric methods. The long-term stability of the model has always been a concern in spectral analysis, as we know. The model may become inapplicable due to changes in the environment and the samples as time goes by. The method used in our study does not require modeling, and only a small number of samples are needed to adjust the parameters involved. The sample itself does not influence the recognition accuracy of the image process method.
Threshold segmentation is commonly used for extracting the bruised region. Preprocessing is necessary before recognizing an apple image. It is impossible to eliminate the influence of uneven illumination during the pretreatment; thus, the misjudgment of sound apples is a common outcome. A low grayscale region in apple images will appear due to uneven illumination or bruises; however, the low gray areas caused by bruises usually have obvious edges. Therefore, it is more reasonable to identify the bruise edge pixels instead of the entire bruise pixels.
Image processing methods for identifying apple bruises do not involve establishing statistical models. These methods require a small number of samples to determine the critical parameters of the algorithm, such as the cut-off frequency used in filtering. If the cut-off frequency is larger, there will be a more obvious filtering effect. The recognition rate of sound apples increases while the recognition rate of bruised apples decreases. In contrast, if the filtering effect is weakened, the recognition rate of the sound apples decreases, and the recognition rate of bruised apples increases. If the cut-off frequency is an adaptive variable, the bruise recognition result will be improved further.

Conclusion
In this study, a near-infrared camera was used to collect near-infrared images of sound apples and apples bruised within half an hour. A two-level image preprocessing method was used. Then, the edge of the bruised area was extracted by the gradient grayscale image, which reflected the rate of change in the grayscale. The recognition accuracy of all apples was 97.62%. From the obtained gradient grayscale image, the region of the bruises in the apples could be seen clearly. Therefore, it is feasible to use nearinfrared camera imaging technology and image processing methods to identify bruises on apples or other fruits. In summary, the implementation of early detection of apple bruises, the development of a non-modeling recognition method, and the need for small data volume are the focus of this paper. These requirements have been met to a certain extent and are also the characteristics of this work different from other works. In the subsequent study, the method will be further improved, especially the parameters involved in the algorithm can be adjusted to adaptive variables, such as the filter cut-off frequency. In addition, the system will be explored to be suitable for online apple bruise detection, including the combination of hardware and software.
Author contributions ZY developed the theory, performed the computations and wrote the manuscript with support from JZ and JL. HW and YY experimented. LZ investigated and supervised the findings of this work. All authors discussed the results and contributed to the final manuscript.
Funding National Natural Science Foundation of China, Grant No 31172064, Longlian Zhao Data availability Some or all data, models, or code generated or used during the study are available from the corresponding author by request.