Identifying Defects and Varieties of Malting Barley Kernels

doi:10.21203/rs.3.rs-4647502/v1

Download PDF

Article

Identifying Defects and Varieties of Malting Barley Kernels

https://doi.org/10.21203/rs.3.rs-4647502/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

This study introduces a comprehensive approach for classifying individual malting barley kernels, involving dual-sided kernel imaging, a specifically designed image processing algorithm, an optimized deep neural network architecture, and a mechanical sorting system. The proposed method achieves precise classification into multiple classes, aligning with quality standards for malting material assessment.

Throughout the study, various image analysis techniques were assessed, including traditional feature engineering, established transfer learning deep neural network architectures, and our custom-designed convolutional neural network tailored for barley kernel image analysis. Comparative analysis underscores the superior performance of our network model.

The study reveals that our proposed deep learning network achieves a 94% accuracy in classifying barley kernel defects and varieties, outperforming well-established transfer learning models with complex architectures that attain 93% accuracy. Additionally, it surpasses the traditional machine learning approach involving feature extraction and support vector machine classifiers, which achieve accuracy below 90% in detecting defective kernels and below 70% in varietal classification. However, we also noted the traditional approach's advantage in morphological feature recognition. This observation guides new research toward integrating morphological feature extraction techniques with modern convolutional networks.

This paper presents a deep neural network designed specifically for the analysis of cereal kernel images in two applications: defect and variety classification. It emphasizes the importance of standardizing kernel orientation and merging images from both sides of the kernel, and introduces a device for image acquisition that fulfills this need.

Biological sciences/Biological techniques/Bioinformatics

Physical sciences/Engineering/Mechanical engineering

Physical sciences/Optics and photonics/Optical techniques/Imaging and sensing

Physical sciences/Optics and photonics/Other photonics/Optical metrology

Physical sciences/Engineering/Electrical and electronic engineering

Physical sciences/Mathematics and computing

barley kernels

dual-sided imaging

quality assessment

varietal classification

defects classification

convolutional neural networks

Barley grain is the primary raw material (Rani and Bhardwaj, 2021) used in the brewing industry. It undergoes malting (Briggs, 1998; MacLeod, 1967), a process that produces barley malt – a fundamental fermentation ingredient for beer and whiskey production, and also used to sweeten some confectionery products. Malting involves the initial sprouting of grains, which is carried out industrially in batches of several tens of tons of grain at a time, under specific warm and humid conditions. Enzymes produced during sprouting catalyze the conversion of starch into sugars. It is crucial to stop this process at the point of maximum sugar content in the grains. This process is stopped by drying the malt.

Barley (Latin: Hordeum vulgare L.) comes in many varieties, but only a certain subset, known as brewing barley varieties, possess the necessary physicochemical parameters for malting (Celestina et al., 2023; Gupta et al., 2010). In addition to selecting the appropriate variety, it is important that the individual grains in a malted batch are of uniform size. This ensures that they sprout with similar intensity and reach optimal sugar saturation at the same time.

The malt house evaluates the quality of a batch of barley kernels by randomly selecting a 100g sample. Visual evaluation of the kernels in the sample is one element of this assessment (Fig. 1), which involves determining if they belong to a malting variety and estimating the percentage of kernels unfit for germination. Defects that decrease the suitability of the kernels for germination and thus for malting include mechanically damaged, broken, or immature kernels, those with missing embryos, those that have begun to sprout, or those infected with mold. The expert evaluator sorts the kernels according to the type of defect, places them in separate containers, weighs the amount of the containers, and calculates their percentage share in the analyzed sample. If kernels infected with mold, most likely containing toxins (Felšöciová et al., 2021; Pascari et al., 2018), are found, the entire batch is rejected. If other defects are present, the suitability of the batch for malting is reduced, which results in offering a lower price.

In Poland, the procedure for assessing the quality of barley for brewing and the permissible quantities of defective grain fractions are defined by the industry standard "Grain of cereals – Barley for brewing" (BN-87/9131-13). This standard applies to the purchase, trade, and processing of barley for brewing, as well as for export deliveries. In addition, malt houses use two other standards, "Grain of cereals - Barley" (PN-R-74109) and "Barley - Test methods" (PN-R-74110), to determine the protein and starch content and the types and content of impurities. The barley kernels must be healthy, mature, free from pests and mold, clean, without a foreign odor, and the level of pesticide residues must not exceed the tolerance established for cereal grains intended for processing in food industry.

The evaluation of barley quality and the identification of its variety is labor-intensive and subjective, and it depends on the experience and training of the evaluator. Moreover, during the harvest season, the procurement of grains intensifies, requiring the employment of additional, seasonal, and inexperienced evaluators. This can lead to a lack of reproducibility in evaluation results. Furthermore, incorrect identification of mold-infected kernels can negatively impact consumer health. Hence, resolving this issue is crucial both economically and for public health.

The increasing demand for high-quality cereal grains and the necessity of identifying and removing defective kernels have made automated sorting an essential component of the production process (Hamid et al., 2013). Currently, there are several technologies available for kernel sorting that leverage mechanical or optical properties (Henry and Jie, 2017; Inamdar and Suresh, 2014; Pearson, 2010). Mechanical sorters utilize physical attributes like size, shape, and weight to separate the kernels, while optical sorters employ camera and image processing technologies to detect and sort kernels based on their color, shape, and texture. Optical sorters are particularly effective in removing foreign material such as stones or sticks, which can cause equipment damage during processing or harm to consumers if left in the final product.

The latest generation of industrial sorters incorporates artificial intelligence (AI) and machine learning (ML) technologies, enabling them to adapt and learn in response to changing conditions and requirements. These advanced systems can efficiently handle large volumes of grain within a short period, meeting the crucial demand for industrial applications. Consequently, the computer vision methods employed in these sorters often prioritize speed over precision, as their primary objective is to significantly increase the proportion of correctly sorted grains and reject as many damaged grains as possible in a limited time. While the accuracy of such sorting is satisfactory for industrial purposes, industrial sorters are not suitable for precise quality assessment, identification of varieties and the precise recognition of multiple types of damages.

A competitive solution (He et al., 2021; Velesaca et al., 2021) to industrial scanners is the use of flatbed scanners for imaging small quantities of grains. Employing a scanner is a relatively inexpensive approach. Grains are dispersed or arranged on the glass surface of the scanner, often using templates to separate individual kernels from each other. Next, image segmentation methods are employed to identify individual kernel areas. Subsequently, feature extraction methods are applied to these areas, aiming to characterize the morphology, color, and texture of the depicted kernels. By conducting discriminant analysis on these features, it becomes possible to identify distinctive attributes that differentiate between species, varieties, or types of damage in kernels.

However, a flatbed scanner can capture only one side of each kernel. This limitation leads to incomplete input data for analysis. If a specific damage or characteristic is present on the side of a grain which is not visible in the acquired image, the classification outcome for that grain will be erroneous. This creates the need for the development of techniques that enable the omnidirectional or at least two-sided visualization of cereal kernels. An additional challenge arises from the positioning of the light source on one side of the image sensor in the scanner. This leads to uneven illumination of the kernels, resulting in strong lighting on one side and shadowing on the other.

Therefore, it is evident that utilizing a flatbed scanner or other imaging methods that capture only one side of the kernel falls short in providing comprehensive information about its appearance. Moreover, computer vision methods, which involve algorithms for image segmentation, feature extraction, and traditional predictors, do not yield classification accuracy at a satisfactory level.

In recent years, there has been a growing interest in utilizing convolutional neural networks (CNNs) (Bengio et al., 2017; Krizhevsky et al., 2012; LeCun et al., 1998) for the classification of biological materials, including barley grains (Boniecki et al., 2022; Dolata and Reiner, 2018; Kozłowski et al., 2019; Kozłowski and Szczypiński, 2019; Shah et al., 2022) and other cereals (Ceyhan et al., 2023). CNNs are a type of deep neural network specifically designed for image analysis tasks. They consist of two main components: the feature extraction part and the classification part.

The feature extraction part comprises multiple layers of convolutional filters, which enhance image fragments with specific frequency characteristics. The resulting filtered images are then transformed using nonlinear activation functions and downsized through pooling layers to reduce the number of attributes. This filtering, transforming, and downsizing process can be repeated in successive layers of the network, allowing the images to convey only essential attributes from the original images. These resulting images serve as inputs for the classification part of the network, which consists of fully connected layers of neurons resembling a multilayer perceptron (MLP) architecture.

While CNNs have proven to be effective tools for image analysis, there is currently a lack of CNNs specifically designed to analyze both sides of a kernel simultaneously, encompassing the ventral and dorsal aspects. As a result, these networks require adaptation to effectively classify barley kernels.

In this article, we propose the use of a grain image scanner with an original design that enables the capture of evenly illuminated images of both sides of the grain, as well as its side profile. We employ image analysis techniques based on image segmentation and brightness gradient analysis to extract kernel areas from the image, correctly orient the kernels, and differentiate between the ventral and dorsal sides of the kernel. In the subsequent analysis stage, we utilize a CNN with a custom architecture designed specifically for recognizing barley kernel varieties and detecting and classifying defective kernels.

The objective of this research was to develop a relatively simple neural network structure that would facilitate accurate and efficient image classification. Through a comparison with the reference method, we evaluated whether the neural network achieves higher prediction accuracy than the previously employed method. Additionally, we investigated the impact of the image preprocessing procedure on the classification accuracy. By utilizing the proposed image scanner and employing image analysis methods in conjunction with the dedicated CNN architecture, we aim to enhance the accuracy and efficiency of barley kernel classification, offering a promising alternative to existing approaches.

2. 1. Image Acquisition

Previously used methods of image acquisition (Femenias et al., 2022; Velesaca et al., 2021), particularly those employing flatbed scanners, have exhibited several drawbacks, including imaging only one side of the grain, uneven or one-sided lighting, and the need for meticulous arrangement of grains on the scanner’s surface to prevent contact or overlap. Industrial devices utilizing linear cameras also fail to capture grains from multiple angles and they produce low-resolution images that are inadequate for qualitative analysis according to applicable standards. Therefore, we propose a device design (Fig. 2) that eliminates these limitations (Dolata and Reiner, 2018; Lampa et al., 2016).

In this device, barley kernels are placed on the surface of a rotating glass disc. A vibratory feeder dispenses the grains individually, ensuring specific distances between them and from the disc's rotational axis. Two Allied Vision Mako G-125C color cameras, equipped with 16mm lenses and a 3mm spacer ring, are positioned above and below the disc surface. When a kernel arrives in location between the cameras, a ring LED lamp flushes, providing uniform and intense white cold light illumination on the kernel. This configuration allows for capturing clear, high-resolution images of both sides of the kernel (Shrestha et al., 2016). Additionally, a third camera with a 25mm lens and 8mm spacer ring is integrated into the system to capture the kernel's side profile. By analyzing the kernel's geometry in these three images, we can estimate its length, width, and height, enabling the evaluation of kernel size uniformity in the sample under examination.

Once the kernels pass through the cameras, they are conveyed to a pneumatic sorting system. Through the use of nozzles, the grains are directed into various containers based on their assigned class. It is crucial to accomplish the kernel image analysis and classification within a short timeframe to ensure the result is available when the grain reaches the sorting system. The assigned class determines the container in which the grain is placed. During the device's development, we addressed several mechanical challenges related to grain transportation, separation, and classification into quality classes. The analysis of images captured from three perspectives facilitated measurements of grain geometry, replacing the conventional approach of using slit sieves, and allowed for an assessment of their technological class. The attained efficiency enabled the evaluation of a 100-gram sample (approximately 2000 grains) in approximately 10 minutes.

One of the technical challenges encountered while using the device was maintaining the quality of the glass surface. Over time, the surface would accumulate dust and develop scratches. To address this issue, a solution was implemented whereby a series of several dozen images of the glass surface are captured immediately after the device is turned on, during the initialization period. These images solely present the surface of the glass disc and do not include any barley kernels. By analyzing the pixel brightness variation across the image sequence, the level of background variability could be assessed, providing an indication of surface contamination and wear. Additionally, the images in the sequence are averaged, resulting in a representative background reference for subsequent image segmentation procedures, then involving barley kernels.

2. 2. Image Preprocessing

Images of kernels captured by the top and bottom cameras undergo processing (Kociołek et al., 2017; Szczypiński and Zapotoczny, 2012a) (Fig. 3) before being passed to the feature extraction procedure or the CNN. Firstly, both images are converted into binary images, which have only two levels of brightness: one indicating the barley kernel area and the other representing the background. Since the kernels appear as relatively bright with comparison to the dark background, a brightness thresholding algorithm is applied to achieve binary image. The threshold value is adjusted locally based on the background reference image. Pixels that are significantly brighter than the background reference image are assigned one level of brightness, while the other pixels are set to the other level.

In addition to the barley kernel regions, the binary images may contain relatively small and separate areas representing dust particles, pieces of awns, or husk fragments. To eliminate these unnecessary areas and slightly smooth the contour of the barley kernel region, mathematical morphology operations such as opening and closing are performed using a small circular-shaped structuring element. If multiple areas still remain in the binary image after these operations, only the largest one is retained. This area defines the region of interest for subsequent processing procedures.

Following the binarization process, both the original and binary images undergo cropping to encompass the region of interest, resulting in reduced-size images that are more suitable for further processing. These cropped images are designated by the letter C. The regions of interest exhibit an elongated shape that narrows on the side corresponding to the kernel's embryo. Analyzing the shape of these regions allows for estimating the elongation direction, approximate axis of symmetry of the kernel, and its orientation relative to the embryo side. This information is then utilized to rotate the image content, aligning the kernel in a vertical orientation with the narrowing pointing upward. The resulting rotated images are indicated by the letter R.

Due to the flattened shape of the kernel, it can rest on the glass surface in two ways: with the dorsal or ventral side facing down. In the first scenario, the crease of the kernel is visible in the image from the top camera, while in the second scenario, it is visible in the image from the bottom camera. In the rotated versions (R) of the images captured by the top and bottom cameras, the brightness gradient is computed horizontally, perpendicular to the approximate axis of symmetry. The absolute value of the gradient calculated for the image displaying the ventral side, which exhibits the crease, is significantly larger compared to the other side. This distinction enables the identification of which image represents the ventral and which represents the dorsal side of the kernel. Finally, the rotated images from the top and bottom cameras are combined into a single image, presenting the ventral view of the kernel on the left-hand side and the dorsal view on the right-hand side. These combined-view images are denoted by the letter B.

The image pre-processing procedures yield three sets of images (C, R, and B) with different levels of pre-processing. This allows us to investigate the hypothesis that by initially processing the images, standardizing the kernel orientation, and identifying the ventral and dorsal sides, we can utilize a less complex CNN architecture while achieving higher classification accuracy.

2. 3. Reference Method

In the previously employed methods for analyzing barley kernel images, a traditional approach (Ramirez-Paredes and Hernandez-Belmonte, 2020; Szczypiński et al., 2015; Szczypiński and Zapotoczny, 2012b; Xu et al., 2021) was utilized, which involved extracting image features, selecting significant features, and applying either an SVM classifier (Chandra and Bedi, 2021) for variety recognition or a cascade of such classifiers for defect recognition. The extracted features included surface texture-related characteristics, color features, and morphological attributes. When using a pair of images capturing both sides of the kernel, feature extraction was performed separately for each side, and subsequently, the features were combined into a single data vector. However, only a small subset of features in this vector exhibited relevance to the recognized classes. As a result, a feature selection procedure was implemented to identify a limited set of features that are highly discriminative based on Fisher's linear discriminant. With this reduced feature set, machine learning processes were conducted using SVM classifiers with polynomial basis functions.

In the case of identifying defected kernels, the implementation of a cascade of binary SVM classifiers yielded good classification results. The first classifier in the cascade was trained to recognize broken kernels, followed by the next one to detect grains with damaged germs. Subsequent classifiers were designed to identify whole and healthy kernels, and the final classifier was trained to recognize moldy ones. Kernels that remained unrecognized by any stage of the classification process were considered to be immature or sprouted.

When classifying new kernels with unknown class membership, only the features with high discrimination ability are extracted. By computing a limited number of features, the computational process is accelerated and becomes more efficient. Finally, the trained SVM classifiers utilizing these features are applied to predict the variety or type of defect.

2. 4. Deep Learning Networks

CNNs have become the most advanced tools for many tasks related to classification and recognition of digital images. As already mentioned, they consist of two main parts: the feature extraction part and the classification part. The feature extraction part contains convolutional (CONV) layers, which use filters to extract features from the input image, and activation layers, which apply non-linear transformations to the features. The most commonly used activation functions (Apicella et al., 2021; Dubey et al., 2022) in CNNs are ReLU, sigmoidal, Elu, and Gelu. The maxpool layer is used to downsample the feature maps to reduce computational complexity and to extract the most salient features. In the classification part, the extracted features are fed into fully connected (FC) layers, which perform the final categorization of the image.

In learning process of the CNN, images of known class affiliation are presented to the network. The network predicts the class of images, and in case of incorrect prediction the weights of the filters and the FC layers are adjusted through error backpropagation. This process evolutionarily decrease the number of differences between the predicted and real class affiliations.

The learning process aims at minimization of a loss function. There are various types of loss functions that can be used depending on the task and type of data. For classification problems, commonly used loss functions include cross-entropy loss and categorical hinge loss. The choice of loss function depends on the specific problem and often requires comparative experiments. A well-chosen loss function enables the network to achieve high classification accuracy. Moreover, there are various optimization algorithms to minimize the loss function such as, stochastic gradient descent (SGD) (Bottou, 2010), Adam (Kingma and Ba, 2014), or Nadam (Ruder, 2016). The correct selection of the algorithm affects both the final optimization result, and thus the classification accuracy.

The initial design of the CNN dedicated for barley kernel analysis (CNN-Barley) was developed based on deep networks that achieved high accuracy in image classification of the ImageNet competition (Mishkin et al., 2017). A constant size of convolutional filters was adopted in subsequent layers, and MaxPool layers were used to reduce size of feature maps. Finally, SoftMax neurons were used in the output of FC layers, which provide probabilities that the analyzed image belongs to certain classes.

During a series of experiments, the performance of different network variants was compared, in which a different number and size of convolutional filter kernels, different types of activation functions, the number of network layers and various optimization algorithms were used and tested. All classifiers were compared in terms of balanced accuracy of classification, which was computed for the validation set images. The use of a separate, validation set of images, not used in the learning process of the network, allows to assess the level of knowledge generalization of the neural network obtained during the learning process and prevents the network from remembering only teaching examples without the ability to properly classify other data – to generalize knowledge.

The results produced by the CNN-Barley were compared to the results obtained with the reference method described earlier. In addition, the performance of the network was also compared with two other CNN networks, ResNet (He et al., 2016) and AlexNet (Krizhevsky et al., 2012). They are two commonly used pre-trained networks for image analysis tasks. AlexNet is a shallow network with 5 CONV layers and 3 FC layers, while ResNet is a deeper network with skip connections to improve performance. In such pre-trained networks the feature extracting layers are already trained to extract features suitable for recognizing various classes of natural images. Thus, instead of training such the networks from scratch the transfer learning is applied. This means, the CONV layers and their weights are transferred to the new network, in which only the FC layers are specifically designed for a specific classification problem and subjected to the learning process.

Summarizing, in this work we present a neural network particularly designed for the analysis of barley kernel images. In this network, the structure was optimized, and it was trained from scratch using exclusively images of barley kernels belonging to different classes. The performance of the developed network was compared with the performance of two other pre-trained CNNs with transfer learning applied, and with the reference method, designed with traditional feature engineering approach, which was previously used to analyze images of barley kernels.

There were two collections of images captured using different devices: one comprising various varietal classes and the other consisting of examples of kernel defects. The first collection consisted of healthy and non-defective kernels obtained from selected farms in Warmia-Mazury region of Poland between 2011 and 2013. These kernels originated from controlled crops, ensuring varietal uniformity and purity. The collection included six varieties of two-rowed barley: Blask, Bordo, Conchita, Kormoran, Mercanda, and Signora (Kozłowski et al., 2019; Szczypiński et al., 2015). The images were acquired using an Epson 4990 flatbed scanner at a resolution of 400 dpi. Each image depicted 450–480 individual kernels, evenly distributed and spaced on the scanning surface. The orientation of the kernels with respect to the dorso-ventral axis was random. However, the images underwent segmentation, individual kernel identification, and preprocessing as explained earlier, resulting in the generation of C and R types of images (Fig. 3). Overall, the collection encompassed six varietal classes, with over 10,000 example images per class. The collected dataset is publicly available and named MaBaKI (Kozłowski et al., 2024).

The second collection of kernel images comprised both healthy and defective kernels (Kociołek et al., 2017; Kozłowski and Szczypiński, 2019; Szturo and Szczypiński, 2017) obtained from various farms in the western part of Poland. Barley kernel samples were sorted into quality classes by an expert from Soufflet Polska Sp. z o.o. malting house in Poznań. The five classes consisted of 1) broken kernels, 2) kernels with missing germs, 3) green or undeveloped kernels, 4) infected or sprouted kernels, and 5) healthy kernels (Fig. 4). Each class was represented by over 2,000 example kernels. The images were captured using a device with a transparent disc shortly after being received and assessed at the malting house. This ensured that the characteristics of the grains had not changed, such as due to drying or mold development. All the images underwent preprocessing to generate C, R, and B types of image sets (Fig. 3).

For training and validation purposes, the images in both collections were divided into 10 equally sized subsets – folds. To ensure the reproducibility of the machine learning experiments, the training and testing process was repeated 20 times – cross-validated. Each time, eight randomly selected folds were used for training, one fold was used for validation during the training process, and the remaining fold was used for final testing after the training was completed. The separation of the validation and test sets ensured that the test results were independent from the data used in the learning process. This allowed us to assess whether the classifiers were able to generalize the knowledge or whether they had overfitted to the training data.

The initial design of the CNN-Barley architecture requires adjustment to suit the specific application, image classes, problem complexity, and size of the training dataset. Factors such as the training method, activation function, feature selection, normalization techniques, filter size, number of filters, and layer count can significantly impact both the classification accuracy and efficiency of the designed network. Consequently, a series of experiments were conducted to compare accuracy and speed across various network configurations and different hyperparameter values that define the network's structure.

Given the belief that recognizing subtle varietal characteristics is more challenging than detecting damaged kernels, the optimization process for CNN-Barley was primarily undertaken using an image collection containing diverse varieties of kernels. Subsequently, the resulting network structure, after appropriate adaptation of the final classification layers, was also employed for the classification of defective kernels.

In the initial experiment, we conducted a comparison of optimization algorithms, which included stochastic gradient descent (SGD), Adam, and Nadam. It is important to highlight that both the Adam and Nadam algorithms are enhancements of the SGD algorithm, designed to refine the learning process. All three algorithms were subjected to testing with varying optimization step lengths. In each case, the training phase was halted when the classification accuracy for the validation set showed a decline over the subsequent two training epochs. The final state of the network was determined by selecting the configuration that achieved the highest classification accuracy for the validation set during training.

The outcomes of these comparisons are detailed in Table 1. Slightly elevated classification accuracies were achieved using the SGD method. As a result, there was no confirmation of the Adam and Nadam algorithms having an advantage over SGD. It should be noted that all three algorithms underwent testing with different optimization step lengths. Typically, reducing the step size to a certain value leads to improved classification accuracy at the expense of longer training time. However, further decreasing the step size does not yield significant improvements. Therefore, the presented results were obtained using relatively small optimization step values, where reducing them further did not substantially increase the achieved accuracies.

Table 1

Comparison of classification accuracy for different optimization algorithms.
Optimizer	Step	Validation accuracy	Test accuracy
SGD	0.0100	0.906	0.904
Adam	0.0001	0.902	0.902
Nadam	0.0001	0.903	0.903

In the designed CNN-Barley architecture (Fig. 5), it was assumed that the resolution of input images and feature maps in successive network layers would be reduced by a factor of two in both the vertical and horizontal directions. This practical approach enables the construction of a network comprising a maximum of six layers. Beyond this number, the number of features in the final layers would be insufficient. The primary objective of the subsequent experiment was to ascertain the optimal number of CONV layers. Different network configurations, ranging from 2 to 6 CONV layers, were examined. Notably, the highest classification accuracy of barley varieties, exceeding 0.9, was achieved with a network employing four CONV layers. Consequently, a network featuring this number of layers was adopted for subsequent experiments.

Subsequently, the network's performance was assessed by comparing different sizes (3×3, 5×5, and 9×9) and quantities (4, 8, 16, 32, 64, 128, and 256) of filters in the initial CONV layer. The outcomes of these experiments are presented in Table 2, illustrating the results for specific combinations of filter counts and sizes. Notably, the highest classification accuracy was attained using relatively compact 3×3 filters, with a count of 128 filters in the first layer. Adding more filters began to curtail the network's ability to generalize knowledge and led to diminished classification accuracy for both the validation and test datasets. It was also observed that employing 64 filters resulted in a 25% faster classification process and reduced memory usage, with only a minor decline in accuracy. Consequently, a configuration featuring 64 filters of size 3×3 was selected for further experimentation.

Table 2

The effect of the size and number of filters in the first CONV layer on the classification accuracy of the CNN-Barley network.
Size of filters	Count of filters	Resulting accuracy
5 × 5	64	0.894
9 × 9	64	0.872
3 × 3	4	0.740
3 × 3	8	0.750
3 × 3	16	0.894
3 × 3	32	0.898
3 × 3	64	0.904
3 × 3	128	0.907
3 × 3	256	0.899

In the subsequent experiment, a comparison was conducted involving various activation functions, encompassing the sigmoidal function, rectified linear unit (ReLU), error linear unit (ELU), and Gauss error linear unit (GELU). The classification accuracies attained for each activation function are presented in Table 3, with ReLU outperforming others to achieve the highest accuracy.

Table 3

The impact of applied activation function on the accuracy of classification obtained.
Activation function	Resulting accuracy
sigmoidal	0.170
ReLU	0.904
ELU	0.903
GELU	0.903

The finalized structure of the CNN-Barley is depicted in Fig. 4. The architecture consists of four successive blocks, each incorporating a CONV layer, a ReLU activation function layer, and a MaxPool layer responsible for selecting features with the highest values. As the blocks progress, the size of the image maps reduces by a factor of four, while their quantity (referred to as channels) increases. After the final MaxPool layer, 256 channels are retained. These 256 feature values are then fed into a multilayer perceptron (MLP) network, with layers designated as fully connected (FC). The FC configuration employs one hidden layer and one output layer. The output layer employs neurons with a linear output function – Softmax. A dropout technique of 30% is applied to eliminate insignificant connections between these two layers. Incorporating the dropout technique during training facilitates an increase in the classification accuracy for barley varieties, reaching up to 0.92.

The dimensions of the input layer in the designed network vary based on the size of the input images. For C and B-type images, the dimensions are set at 170×170 pixels. In contrast, for R-type images, the dimensions are 170×80. Both cases involve the analysis of color images featuring three RGB color channels. Conversely, the number of neurons in the output layer must correspond with the number of classes. Activation of a specific neuron within this layer signifies the recognition of the image's association with its respective class. In cases involving variety recognition, the network has six outputs, whereas for defective kernel detection, a network with five outputs is employed.

In the subsequent experiment, a comparison was conducted between the developed CNN-Barley and three other state-of-the-art methods. The first one involved a traditional approach encompassing feature extraction, feature space reduction, and classification. For this purpose, the QMaZda software package was employed (Szczypinski et al., 2009; Szczypiński et al., 2017; Szczypiński and Klepaczko, 2017). The remaining two methods employed CNNs with weight transfer from the AlexNet and ResNet networks, respectively. Therefore, throughout the rest of this paper, the reference methods will be denoted as QMaZda, AlexNet, and ResNet.

In the traditional approach (QMaZda), image feature extraction was carried out, encompassing the calculation of morphological, color, and texture features. Subsequently, a feature selection process (using a wrapper approach) was employed to identify feature groups with the highest discriminative capabilities. The discriminative power of these features was evaluated based on Fisher's discriminant, derived from linear discriminant analysis.

The selection procedure was executed separately for the images representing different varieties and for the images of defective kernels. Additionally, as feature selection constitutes part of the machine learning process, the selection procedure was repeated for each of the 20 cross-validation instances. As a result, during each instance, feature groups consisting of 30 to 50 of the most discriminative features were chosen. Vectors of the selected feature values, extracted from images of the training set, were used for training SVM classifiers.

In the case of AlexNet and ResNet, models with 18 CONV layers were employed. This choice was regarded as a reasonable trade-off, taking into consideration the model's complexity and the available hardware resources. Network models with input layers of 224×224 pixels were utilized. To align the existing images with the input layer size, black pixel margins were added around the original images. Furthermore, the number of neurons in the output layers of the network was adjusted to match the recognized classes, either 5 or 6.

Two approaches were employed for training the network. In the first approach, the training process was confined to the FC layers responsible for data classification, while the weights of the CONV layers remained unaltered. In the second approach, training was extended to the CONV layers, encompassing fine-tuning of filters. This latter approach resulted in a 2 to 5 percentage point increase in classification accuracy. Due to the significant number of weights present in the architectures of AlexNet and ResNet, there arose a necessity to enlarge the training datasets. Augmentation of these datasets was applied, involving random rotations within a 5° limit, shifts, and mirror-image flipping of the original images. This augmentation contributed to enhancing the classification outcomes for both the reference methods and the developed CNN-Barley.

Table 4

Comparison of classification accuracies between CNN-Barley and reference methods.
Data set	CNN-Barley	AlexNet	ResNet	QMaZda
Varieties C	0.873	N/A	N/A	0.644
Varieties R	0.942	0.918	0.932	0.681
Defects C	0.871	N/A	N/A	0.825
Defects R	0.901	0.895	0.901	0.877
Defects D	0.938	0.912	0.932	0.888

Table 4 presents the classification accuracy values obtained for the developed neural network and three reference solutions. Since, fine-tuning of filters in AlexNet and ResNet emerged as the superior solution, the outcomes achieved through fine-tuning are presented. All the accuracy values provided are averages derived from the 20 cross-validation repetitions.

Table 5. Confusion matrices of defective and healthy kernels classification

Table 5 displays the confusion matrices for the classification of healthy and defective kernels using CNN-Barley and QMaZda. The set of B-type images (Fig. 3) were used in this experiment. The rows of the matrices correspond to the actual classes, while the columns represent the predictions. The values are presented as percentages of predictions relative to the number of kernels in the actual classes, and values in parentheses indicate the standard deviations calculated from 20 repetitions of the learning process. These matrices enable the evaluation of the correct prediction ratio for individual classes and provide a quantitative representation of classification errors between the actual and predicted classes. In most instances, the confusion matrices reaffirm the superiority of the proposed CNN-Barley method. The only exception lies in the recognition of broken kernels, where QMaZda slightly outperforms the CNN.

It is important to highlight that in the classification of broken kernels using the QMaZda method, the most discriminative features were the kernel's area and elongation ratio. These features fall into the category of morphological (shape descriptive) attributes. In contrast, such shape characteristics cannot be computed by the CONV layers of the CNNs.

All experiments were conducted using a computer equipped with an Intel® Core i7-4930K 3.40GHz processor (CPU) and an NVIDIA GeForce GTX 780 Ti graphics card (GPU), operating under Linux-Kubuntu 20.04. Image preprocessing, feature extraction, and SVM classification utilized the CPU, while the majority of computations associated with deep neural networks were performed on the GPU. Given the diverse computational resources used by the compared image classification methods, it becomes challenging to gauge their efficiency purely based on time measurements. Nonetheless, information on the analysis time can be beneficial when deciding which method to deploy on modern personal computers, typically offering both CPU and GPU technologies.

For the conventional image classification approach (QMaZda), the analysis time for a single grain remains within 380 ms per computational thread. This encompasses the time needed for preprocessing a pair of images, taking around 100 ms, the computation of a selected feature group ranging from 90-250 ms (dependent on the number and type of features selected), and the classification process itself, requiring approximately 30 ms. Leveraging the multiple cores of the processor, computations can be run concurrently for several kernel images, resulting in average analysis times of 25-50 ms per image.

In the case of the developed CNN-Barley, the average time for kernel analysis is approximately 0.7 ms. For both AlexNet and ResNet, the classification process demands 2.5 ms and 4.8 ms respectively. It is worth noting that these times do not include the image preprocessing conducted by the CPU nor the time needed for image transfer to GPU memory.

This paper introduced a dedicated deep neural network model for analyzing barley images, specifically designed for varietal classification and the detection of defective barley kernels. The classification results were compared with those obtained from reference methods, including ResNet and AlexNet networks, as well as the traditional method that involves feature extraction and SVM classifiers. The proposed CNN-Barley demonstrated superior performance compared to the other methods.

The effectiveness of image pre-processing, which encompassed standardizing the orientation of kernels and combining images from both sides of each kernel, was also validated. This approach improved classification accuracy across all the compared methods. Furthermore, this pre-processing technique enabled the achievement of high classification accuracies through a computationally efficient network structure, characterized by fewer convolutional layers and small-sized filters.

Neural networks have demonstrated their superiority and potential to replace traditional methods in evaluating images of grains and likely other biological materials. However, the experiments also suggest that supplementing these networks with classical image pre-processing procedures could be beneficial. Standardizing the orientation of kernels in images is a procedure that contrasts with typical data augmentation practices applied in CNNs, often involving random rotations introduced into the images. In our case, standardizing the orientation proved advantageous for analyzing barley images, allowing for a reduction in the number of layers while retaining classification accuracy.

When comparing the developed CNN-Barley with the previously utilized traditional method, which achieved an accuracy of no more than 0.7 in varietal classification, the CNN-Barley exhibited significantly higher accuracy, surpassing 0.9. The proposed method also outperformed the others in detecting defective kernels. The only class where the traditional method displayed better accuracy was the category of broken kernels. This deficiency might be attributed to the limited ability of CNNs to extract shape-related features. This hypothesis prompts further research to explore whether enhancing neural networks with the extraction of morphological attributes could improve their classification capabilities.

The presented results of barley kernel defect and varietal classification are reproducible and satisfactory. Consequently, the proposed methods may have practical applications in assessing the quality of raw materials acquired by malting plants.

FUNDING

This research was supported by the National Centre of Research and Development (Grant number: PBS3/A8/38/2015).

Author Contribution

M.K, P.M.S, wrote the main manuscript text,P. L, K.S wrote parts of the manuscript text,M.K, P.M.S prepared figures 1,3, 4, 5P.M.S, P. L prepared figure 2MK prepared tables 1-3, 5P.M.S prepared table 4All authors reviewed the manuscript.

Acknowledgement

The authors would like to express their gratitude to the Polish Ministry of Science and Higher Education, and the National Centre of Research and Development for providing financial support for the project PBS3/A8/38/2015 – Development of industrial methods of automatic evaluation of technological parameters and classification of grain using image analysis. Special thanks are extended to Barbara Sobkowiak for her invaluable contribution in preparing the research materials.

Data Availability

The datasets generated and/or analysed during the current study are available in the Malting Barley Kernel Images (MaBaKI) database repository. The MaBaKI dataset is available at https://doi.org/10.34658/RDB.MMLNNX.

Apicella, A., Donnarumma, F., Isgrò, F., Prevete, R., 2021. A survey on modern trainable activation functions. Neural Netw. 138, 14–32.
Bengio, Y., Goodfellow, I., Courville, A., 2017. Deep learning. MIT press Cambridge, MA, USA.
Boniecki, P., Sujak, A., Pilarska, A.A., Piekarska-Boniecka, H., Wawrzyniak, A., Raba, B., 2022. Dimension Reduction of Digital Image Descriptors in Neural Identification of Damaged Malting Barley Grains. Sensors 22, 6578.
Bottou, L., 2010. Large-scale machine learning with stochastic gradient descent, in: Proceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers. Springer, pp. 177–186.
Briggs, D.E., 1998. Malts and malting. Springer Science & Business Media.
Celestina, C., Hunt, J., Kuchel, H., Harris, F., Porker, K., Biddulph, B., Bloomfield, M., McCallum, M., Graham, R., Matthews, P., 2023. A cultivar phenology classification scheme for wheat and barley. Eur. J. Agron. 143, 126732.
Ceyhan, M., Kartal, Y., Özkan, K., Seke, E., 2023. Classification of wheat varieties with image-based deep learning. Multimed. Tools Appl. 1–23.
Chandra, M.A., Bedi, S.S., 2021. Survey on SVM and their application in image classification. Int. J. Inf. Technol. 13, 1–11.
Dolata, P., Reiner, J., 2018. Barley variety recognition with viewpoint-aware double-stream convolutional neural networks, in: 2018 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE, pp. 101–105.
Dubey, S.R., Singh, S.K., Chaudhuri, B.B., 2022. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing.
Felšöciová, S., Kowalczewski, P.Ł., Krajčovič, T., Dráb, Š., Kačániová, M., 2021. Effect of long-term storage on mycobiota of barley grain and malt. Plants 10, 1655.
Femenias, A., Gatius, F., Ramos, A.J., Teixido-Orries, I., Marín, S., 2022. Hyperspectral imaging for the classification of individual cereal kernels according to fungal and mycotoxins contamination: A review. Food Res. Int. 155, 111102.
Gupta, M., Abu‐Ghannam, N., Gallaghar, E., 2010. Barley for brewing: Characteristic changes during malting, brewing and applications of its by‐products. Compr. Rev. Food Sci. Food Saf. 9, 318–328.
Hamid, G., Deefholts, B., Reynolds, N., McCambridge, D., Mason-Palmer, K., Briggs, C., 2013. Automation and robotics for bulk sorting in the food industry, in: Robotics and Automation in the Food Industry. Elsevier, pp. 267–287.
He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778.
He, X., Zhao, T., Shen, F., Liu, Q., Fang, Y., Hu, Q., 2021. Online detection of naturally DON contaminated wheat grains from China using Vis-NIR spectroscopy and computer vision. Biosyst. Eng. 201, 1–10.
Henry, T., Jie, F., 2017. Design and construction of color sensor based optical sorting machine, in: 2017 5th International Conference on Instrumentation, Control, and Automation (ICA). IEEE, pp. 36–40.
Inamdar, A.A., Suresh, D.S., 2014. Application of color sorter in wheat milling. Int. Food Res. J. 21, 2083.
Kingma, D.P., Ba, J., 2014. Adam: A method for stochastic optimization. ArXiv Prepr. ArXiv14126980.
Kociołek, M., Szczypiński, P.M., Klepaczko, A., 2017. Preprocessing of barley grain images for defect identification, in: 2017 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA). IEEE, pp. 365–370.
Kozłowski, M., Górecki, P., Szczypiński, P.M., 2019. Varietal classification of barley by convolutional neural networks. Biosyst. Eng. 184, 155–165. https://doi.org/10.1016/j.biosystemseng.2019.06.012
Kozłowski, M., Szczypiński, P.M., 2019. Barley Defects Identification by Convolutional Neural Networks, in: Pietka, E., Badura, P., Kawa, J., Wieclawek, W. (Eds.), Information Technology in Biomedicine, Advances in Intelligent Systems and Computing. Springer International Publishing, Cham, pp. 187–198. https://doi.org/10.1007/978-3-030-23762-2_17
Kozłowski, M., Szczypiński, P.M., Reiner, J., Lampa, P., Mrzygłód, M., Szturo, K., Zapotoczny, P., 2024, "Malting Barley Kernel Images (MaBaKI) database", https://doi.org/10.34658/RDB.MMLNNX, | Repozytorium Otwartych Danych Badawczych , V1
Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25.
Lampa, P., Mrzygłód, M., Reiner, J., 2016. Methods of manipulation and image acquisition of natural products on the example of cereal grains. Control Cybern. 45.
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324.
MacLeod, A.M., 1967. The physiology of malting—A review. J. Inst. Brew. 73, 146–162.
Mishkin, D., Sergievskiy, N., Matas, J., 2017. Systematic evaluation of convolution neural network advances on the imagenet. Comput. Vis. Image Underst. 161, 11–19.
Pascari, X., Ramos, A.J., Marín, S., Sanchís, V., 2018. Mycotoxins and beer. Impact of beer production process on mycotoxin contamination. A review. Food Res. Int. 103, 121–129.
Pearson, T., 2010. High-speed sorting of grains by color and surface texture. Appl. Eng. Agric. 26, 499–505.
Ramirez-Paredes, J.-P., Hernandez-Belmonte, U.-H., 2020. Visual quality assessment of malting barley using color, shape and texture descriptors. Comput. Electron. Agric. 168, 105110.
Rani, H., Bhardwaj, R.D., 2021. Quality attributes for barley malt:“The backbone of beer.” J. Food Sci. 86, 3322–3340.
Ruder, S., 2016. An overview of gradient descent optimization algorithms. ArXiv Prepr. ArXiv160904747.
Shah, S.A.A., Luo, H., Pickupana, P.D., Ekeze, A., Sohel, F., Laga, H., Li, C., Paynter, B., Wang, P., 2022. Automatic and fast classification of barley grains from images: A deep learning approach. Smart Agric. Technol. 2, 100036.
Shrestha, B.L., Kang, Y.-M., Yu, D., Baik, O.-D., 2016. A two-camera machine vision approach to separating and identifying laboratory sprouted wheat kernels. Biosyst. Eng. 147, 265–273.
Szczypiński, P.M., Klepaczko, A., 2017. MaZda – A Framework for Biomedical Image Texture Analysis and Data Exploration, in: Biomedical Texture Analysis. Elsevier, pp. 315–347. https://doi.org/10.1016/B978-0-12-812133-7.00011-9
Szczypiński, P.M., Klepaczko, A., Kociołek, M., 2017. Qmazda—Software tools for image analysis and pattern recognition, in: 2017 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA). IEEE, pp. 217–221.
Szczypiński, P.M., Klepaczko, A., Zapotoczny, P., 2015. Identifying barley varieties by computer vision. Comput. Electron. Agric. 110, 1–8. https://doi.org/10.1016/j.compag.2014.09.016
Szczypinski, P.M., Strzelecki, M., Materka, A., Klepaczko, A., 2009. MaZda - A software package for image texture analysis. Comput. Methods Programs Biomed. 94, 66–76. https://doi.org/doi: 10.1016/j.cmpb.2008.08.005
Szczypiński, P.M., Zapotoczny, P., 2012a. Computer vision algorithm for barley kernel identification, orientation estimation and surface structure assessment. Comput. Electron. Agric. 87, 32–38. https://doi.org/10.1016/j.compag.2012.05.014
Szczypiński, P.M., Zapotoczny, P., 2012b. Computer vision algorithm for barley kernel identification, orientation estimation and surface structure assessment. Comput. Electron. Agric. 87, 32–38.
Szturo, K., Szczypiński, P.M., 2017. Ontology based expert system for barley grain classification, in: 2017 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA). IEEE, pp. 360–364.
Velesaca, H.O., Suárez, P.L., Mira, R., Sappa, A.D., 2021. Computer vision based food grain classification: A comprehensive survey. Comput. Electron. Agric. 187, 106287.
Xu, P., Yang, R., Zeng, T., Zhang, J., Zhang, Y., Tan, Q., 2021. Varietal classification of maize seeds using computer vision and machine learning techniques. J. Food Process Eng. 44, e13846.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
05 Aug, 2024
Reviews received at journal
03 Aug, 2024
Reviews received at journal
02 Aug, 2024
Reviews received at journal
22 Jul, 2024
Reviewers agreed at journal
18 Jul, 2024
Reviewers agreed at journal
16 Jul, 2024
Reviewers agreed at journal
16 Jul, 2024
Reviewers invited by journal
15 Jul, 2024
Editor assigned by journal
10 Jul, 2024
Editor invited by journal
07 Jul, 2024
Submission checks completed at journal
05 Jul, 2024
First submitted to journal
27 Jun, 2024

You are reading this latest preprint version

Identifying Defects and Varieties of Malting Barley Kernels

Status:

Version 1

Abstract

Figures

1. INTRODUCTION

2. METHODS

2. 1. Image Acquisition

2. 2. Image Preprocessing

2. 3. Reference Method

2. 4. Deep Learning Networks

3. MATERIALS

4. RESULTS

5. CONCLUSIONS

Declarations

FUNDING

Author Contribution

Acknowledgement

Data Availability

References

Additional Declarations

Status:

Version 1