A state-of-the-art approach for plant disease detection systems is discussed in this paper. Most proposed disease detection models in literature utilize single infeed cameras to capture the images of sample plant organs for classification. Single-input cameras might compromise the classification accuracy of these models depending on which plant organ is being used. Single input camera classification models have operated with high classification accuracy and efficiency with leaf samples because of their flat surface area nature, however, this is not always the case for fruit samples because of their general spherical or cylindrical nature such as oranges or bananas. The symptoms of a disease on the surface area of a sample fruit might not be distributed evenly, hence a single input camera sensor might miss the vital diseased part if the sample is orientated such that the diseased area is directly opposing to the line of sight of the camera sensor, which can consequently lead to an incorrect classification of that sample under evaluation. Hence, this study has proposed a multi-camera input fruit disease classification model aiming to solve this problem. Citrus orange fruits were used to demonstrate the capability of the proposed model to classify healthy and black rot-affected orange samples. A healthy sample and two black-rot-affected oranges, one with even and the other with uneven distribution of black rot symptoms, were put under evaluation of the proposed multi-camera input model and the classification accuracy was 100% when utilizing a deep learning Convolutional Neural Network classification algorithm.
Article
Design and Modeling of a Multi-camera-based Disease Detection Model
https://doi.org/10.21203/rs.3.rs-2874541/v1
This work is licensed under a CC BY 4.0 License
You are reading this latest preprint version
Artificial Neural Network (ANN)
Deep Learning
Fuzzy Classification
Fourth Industrial Revolution (4IR)
k-Nearest Neighbour (k-NN)
Machine Learning (ML)
Precision Agriculture
Space Vector Machine (SVM).
The purpose of this paper is to design and model a Multi-Camera Input-Based fruit disease detection system for farming applications. The agricultural sector is a major contributor to the economy of the African continent, hence it is an integral part of the global economy while it also serves an important purpose of food production for a continuously rising global human population [1, 2]. The United Nations projects that the global human population will cross the 10 billion people mark approximately three decades from now (by 2050). This population increase is expected t effectively double the global food demand from what we are experiencing currently [3]. Therefore, the agricultural sector is under pressure to keep up with this annually increasing food demand. Studies project that the global agricultural yield needs to rise by 1.75% annually to keep up with the rising food demand [3, 4].
Precision agriculture has been developing at a relatively faster rate to help improve farm yields on both quality and quantity [4]. This paper, however, discusses a particular area in precision agriculture called decision support systems. These systems are capable of identifying and classifying different diseases or nutrient deficiencies in different farm crops by processing images of fruits, stems, leaves, or roots of a particular plant [3, 4]. Some farmers rely on visual inspection of the specimen to identify different diseases and nutrient deficiencies that affect their crops and make decisions accordingly [1, 3, 4]. This traditional disease/nutrient deficiency detection method is inaccurate since farmers rely on their experience and personal knowledge which might be inaccurate itself [4]. Sourcing expert services may prove to be costly and have long turnaround times. This explains the significance of more accurate and efficient decision support systems when it comes to crop diseases and nutrient deficiencies [1, 4].
This study aims to improve the efficiency and accuracy of these decision support systems by incorporating multiple camera sensors as inputs to get different image angles of a single sample under evaluation before detection and classification can be made. This improves the classification accuracy of the disease detection system by letting the system process the majority of the sample surface other than a single side view. Hence the classification is now processing a 3-dimensional image of a sample instead of a two-dimensional image that is utilized in almost all the citations of this paper.
The classification accuracy and efficiency of disease detection models can be compromised by several factors. These factors can range from the size to variability of the training dataset [5]. If a classification model is trained with fewer data samples, the model will learn fewer features during the training process[5, 6]. Similarly, if a classification model is trained utilizing enough, but similar data samples, the model will also learn a few features during the training process [6]. When the training is complete and the model is presented with an object (fruit in the case of this paper) that has slightly different features relative to the training dataset, the classification accuracy and efficiency are greatly impacted.
Defects on the fruits are not always distributed evenly across the whole surface area of fruits. This implies that signs and symptoms of fruit diseases and nutrient deficiencies might appear on certain parts of fruits while the rest appears healthy. During the operation of a classification model, if the sample fruit is orientated such that the defective parts of a fruit are not in the line of sight or the view of a camera sensor, then that fruit can be wrongly classified as a healthy fruit hence compromising the accuracy of a classification model. Very few publications that the authors have reviewed so far (including the citations of this paper) utilize multiple infeed cameras, the majority of related work publications utilize single infeed which might not be able to capture the vital details of sample fruits due to their general spherical nature.
Figure 1 shows a sample orange fruit that has the symptoms of a black rot disease on the small section of its surface area (less than 30% of the total surface area) while the rest of the fruit appears healthy. If this fruit was orientated such that this rotten part of the fruit is 180˚ opposing the camera sensor, the model would have classified this orange as healthy. This may be a challenge in high throughput automated production lines where a model classification is processing multiple samples at a unit of time because samples can come at any orientation into the classification model.
In the literature, various studies discuss plant disease detection systems as support decision systems in farming operations. This section is separated into two main parts, the general architecture of the plant disease detection systems, and the second part evaluates the previous related studies that utilize single and multi-camera inputs.
3.1 A General Architecture of Plant Disease/Nutrient Deficiency Detection Models
Machine Learning-based plant disease/nutrient deficiency detection systems comprise two main sub-systems, image processing, and classification systems [6]. The image processing further comprises four steps while there are also four most cited different classification protocols. Table 1 summarizes the image processing steps and the four popular Machine Learning classification models utilized in plant disease/nutrient deficiency detection systems.
| 1. Image processing steps | 2. Different classification techniques |
|---|---|
| 1. Image processing: • Image acquisition • Image pre-processing • Image segmentation • Feature extraction • Machine learning classification | 3. SVM Classifier |
| 5. ANN Classifier | |
| 7. KNN Classifier | |
| 9. FUZZY Classifier |
Image processing steps.
Image acquisition.
Acquiring an image from a sample to be classified is the starting point of Machine Learning-based plant disease/nutrient deficiency detection systems [15, 6]. The image acquiring devices such as image sensors or canners, and unmanned aerial vehicles (UAVs) are the common devices to be utilized for image acquisition purposes [7]. These image sensors utilize two main image sensing technologies, the charge-coupled device (CCD) and complementary metal oxide semiconductor (CMOS) technologies [7]. Both these camera technologies convert light signals and protons to digital data, from digital data to image data [7, 8]. However, their method of forming image data vary [8]. The light signals for a CCD camera are transmitted through a string of neighboring pixels, where they are then amplified and transformed into image data [9]. This makes it possible for the CCD cameras to acquire images with little degradation [9]. The CCD cameras produce images that are sharp and have minimal distortion [10]. In contrast, each pixel of the image sensor of CMOS cameras collects, amplifies, and converts light signals [10–12]. Because each pixel can locally convert light impulses into a picture, CMOS devices can produce images more quickly than CCD devices [11–15].
Since they are less expensive than CCD devices, use less power, and can capture high-quality images faster than CCD devices, CMOS devices are typically used in low-budget projects [15, 16]. Figure 2 depicts the conversion of CCD and CMOS image sensors' serial and localized pixels, respectively.
Image Pre-processing.
Pre-processing an image involves adjusting the image contrast appropriately and filtering interference signals that cause noise and subsequently fuzzy images [17, 18]. This process can improve feature extraction precision and illness identification accuracy more generally [18]. Preprocessing often entails simple operations like deblurring, image clipping, cropping, filtering, and trimming, to mention a few [17, 18]. According to Khalili [19], the typical image preparation steps used in image-based detection systems include picture acquisition, grayscaling, filtering, binarization, and edge filtering.
A colored image is converted into a gray image as the first stage in the process shown in Fig. 3 [19]. Otherwise, this step is essential since processing an image in a grey-scale format is much easier and quicker [17–19]. This conversion stage into a grey image may be skipped in applications where color elements are important. The second stage entails denoising a specimen picture because, in the majority of cases, noise signal interference impacts the visibility of features in the specimen images [19]. The next phase is then picture segmentation, which is covered in more detail in the coming section. The final stage entails creating an outline image, which can be done by preserving the outer linked region while masking the leafstalk and holes [19].
Patil [20]. suggested a similar process to that shown in Fig. 7 for applications involving the recognition of plant-leaf features in a variety of illumination environments. This process involves converting a sample image to grayscale, noise suppression, smoothing, and edge filtering to create the image outline [20]. A histogram equalization technique was shown to be the most efficient way to improve the quality of the greyscale versions of color photos in comparative research by Orchi [21]. In contrast, Zhang [22] discovered that when used to identify illnesses on plant leaves, RGB camera photographs provide more useful image improvement than those that are transformed to greyscale.
Image Segmentation.
An essential component of image-based plant feature recognition and phenotyping systems is image segmentation [23]. To segment an image, the foreground, and background must be separated [23], which entails isolating the feature of interest and masking the irrelevant area of the image [24]. By comparing nearby pixels for similarity based on the three major criteria of texture, color, and shape, the features of interest are often found [24]. Thresholding is one of the most well-known examples of an image segmentation technique [25]. To make feature classification simpler, a procedure known as Thresh-old segmentation transforms a color or grayscale image into a binary image, as seen in Fig. 4 [24, 25]. Black and white colored pixels that correspond to the background and foreground, respectively, make up the output binary pictures [25].
Threshold segmentation is mathematically defined as follows, where T refers to a certain threshold intensity, g is the black or white pixel of a binary image and f is the grey level of the input image [26].
$$g\left(x,y\right)=\left\{\begin{array}{c}0, if f(x,y) ˂ T\\ 1 if f(x,y) ˃ T\end{array}\right.$$
1
Threshold segmentation is classified into three; global, local, and adaptive thresholding [26]. When there is a sufficient difference between the intensity distribution of the foreground and the background, global thresholding is used [26]. To discriminate between the features of relevance and the background, a single threshold value is used [26]. When there is no obvious difference in the intensity distribution of the background and the foreground, local thresholding is used since it is difficult to choose only one threshold value [26]. In this scenario, an image is divided into smaller images and various threshold values are chosen for each divided image [15]. When a threshold-old value is calculated for each pixel in an image with an uneven intensity distribution, this is called adaptive thresholding [26, 27].
Another thresholding technique used for image segmentation is the Otsu thresholding approach [27]. By repeatedly looping over all possible threshold values, this method calculates the spread between the pixel intensity levels on either side of the threshold [27]. The goal is to determine the value below which the sum of the foreground and background diminishes to the smallest amount [26, 27]. The Otsu thresholding method's key characteristic is that the threshold values are applied automatically rather than being preselected by the user [27]. The Otsu method's thresholding has a mathematical definition (2).
| \(g\left(x,y\right)=\left\{\begin{array}{c}1, f\left(x,y\right) ˃ T\\ 0, 0 otherwise\end{array}\right.\) | (2) |
Watershed transformation is another segmentation technique used in image processing [28]. A watershed is a change that occurs in a grayscale image [28]. The name alludes, metaphorically speaking, to a geologic drainage split that separates adjacent catchments [28]. The image that the watershed conversion operates on is treated as a topographic map, with the luminosity of each pixel designating its elevation, and this method locates the lines that follow the tops of ridges [28]. Figure 5 shows an illustration of a watershed-segmented image, where the background is shown by the black pixels, the features to be extracted are indicated by the grey pixels, and the watershed lines are indicated by the white pixels [28].
On the other hand, Grabcut is a popular and ground-breaking segmentation method that takes into account an image's boundary and textural circumstances [29]. This segmentation technique is based on the iterative graph-cut method, where the background and foreground are implemented using a mathematical function [29]. Then, it is determined whether each pixel in a picture belongs in the background or the foreground [29]. The grab-cut segmentation method is popular in most applications since it requires little human involvement in its use, but it has limitations as well [30]. The intricacy of the thresholding equation renders it hard to construct the Grab-cut sequence cycles [30]. When the background is complicated and there is little contrast between the characteristics of interest and the background, the segmentation is also inferior [30]. There are numerous unique segmentation methods and algorithms in the literature. This study is unable to rule out specific segmentation methods or recognize those that perform better than the others because the applicability of one method depends on a specific application.
Image Feature Extraction.
The extraction of features is one of the key components of computer vision-based picture recognition [31, 32]. An element of a raw image that is used to solve a specific computer vision problem is known as a feature [31, 32]. The features that have been extracted from a picture are included in the feature vectors [32]. For creating feature vectors, a wide variety of approaches are employed to identify the objects in an image [31, 32]. The main characteristics are edges, picture pixel intensity, geometry, texture, image alterations like Fourier or Wavelet, or permutations of pixels from different color images [31, 32]. The main application of feature extraction is a set of classifiers and machine learning algorithms [31, 32]. The feature extraction in plant leaf disease monitoring systems is classified into three spheres which include texture, color, and shape [32].
The shape is a fundamental aspect of a leaf/fruit that is used in the feature extraction method for leaf/fruit images [33]. The length (L) and width (W), represent the displacement between the two points in the longest axis, the diameter (D), represents the greatest distance between the points, the area (A), represents the surface area of all the pixels found within the margin of a leaf picture, and the perimeter (P), which represents the cumulative length of the pixels around the margin of a leaf picture is the primary shape parameters [33–35]. From these 5 shape primary characteristics, several secondary characteristics can be formed such as circularity, rectangularity, aspect ratio, etc [34, 35].
Some academics and researchers decide to use color features as the key factors in the extraction process [35]. Some examples of the color features that are frequently discussed in the literature for leaf/fruit feature extraction include color mean, color skewness, color kurtosis, etc. Also, several textual characteristics are mentioned by scholars including Sharif [36], Hayit [37], and Jasim [38]. These textural features include correlation, entropy, contrast, etc [36, 38].
After the image processing part is complete from image acquisition to feature extraction, the chosen features are trained into a machine learning classification model of choice to complete the design phase of a plant disease/nutrient deficiency detection model. The coming section discusses the most cited machine learning classification algorithms for this purpose.
Machine Learning Classification Algorithms.
Machine learning algorithms are used to categorize input sample data into several classes or groups of membership using classification techniques [39]. During their training, these classifiers may use supervised learning, unsupervised learning, and reinforcement learning techniques [39, 40]. When a person is a model trainer and uses pre-formed data sets to carry out the training, supervised learning takes place [39, 40]. Since there is no training data available, unsupervised learning takes place; as a result, the algorithm must train itself and increase its classification accuracy by iterative adjudication [39, 40]. When the algorithm makes classification decisions depending on the feedback the environment provides to it, reinforcement learning takes place [39, 40]. The most frequently cited classification algorithms for vision-based plant disease monitoring systems include Space Vector Machines (SVM), Artificial Neural Networks, k-Nearest Neighbor machines, and Fuzzy machines. These classification strategies are covered in the ensuing subsections.
Support Vector Machines.
A Support Vector Machine also referred to as SVM is a predictive model that may be used to handle both classification and regression problems [41]. It is a supervised learning model that can tackle both linear and non-linear tasks and performs well for many real-world issues [41]. The SVM technique generates a vector or a hyperplane that divides the data into groups, which is a very straightforward notion [42].
Figure 6 shows the two classes of data separated by the best hyperplane (the blue squares and green circles). The positive and negative imaginary planes, which pass through the nearest data points on either side of an ideal hyperplane, are the two planes (dashed lines) parallel to the optimal hyperplane [42]. The support vectors are utilized to pinpoint the precise location of an optimum hyperplane and are the locations that are closest to the optimal hyperplane [42]. There may be several potential hyperplanes, but the one with the highest marginal distance i.e., the distance between the two marginal planes, is the best option [41, 42] When compared to lesser margins, the maximal margin yields a more generic solution, and the algorithm with the smaller margin will face accuracy problems if the training data change [42]. Data classes are occasionally difficult to distinguish with a clear light or location, as in Fig. 6. The Kernel method is used to change the low-dimension (typically 2-dimensional) space where these data classes occur into a high-dimension (commonly 3-dimensional) space when data classes exhibit the property of nonlinearity.
In the new high-dimension space, the Kernel approach computes the dot product of the dimensions [41, 42]. In the case where \(\overrightarrow{x}\) is any data point or support vector, \(\overrightarrow{\omega }\) is the weight vector that applies the support vectors' bias \({\omega }_{0}\) which is also a constant [42], Eq. (3) provides the generic solution of a hyperplane.
Neural Network.
A neural network (NN) is a supervised learning model that consists of a network of connected input and output nodes, where each link has a weighted bias value [43]. A general NN has a single input layer, one or maybe more intermediate layers, sometimes referred to as hidden layers, and one or possibly more output layers [43]. When the network runs, the weight of each connection is altered to aid neural network learning [44]. The network's performance is improved by continuously altering the weight [44]. According to connection types, NN can be split into two groups: feed-forward networks and recurrent networks [43, 44]. Feed-forward neural networks do not have cycle-forming connections between units, in contrast to recurrent neural networks [43, 44]. A neural network's behavior is influenced by its architecture, transfer function, and learning rule [44]. The weighted sum of input causes the neurons in neural networks to activate [43, 44]. A generalized NN model comprising the input layer, the hidden intermediate layer (blue layer), and the output layer is shown in Fig. 7.
K-Nearest Neighbors.
The simplest basic machine learning method is the k Nearest Neighbors algorithm, also referred to as kNN [45]. It is a non-parametric method applied to regression and classification issues [45, 46]. Non-parametric implies that no initial training dataset is required [45, 46]. As a result, kNN excludes the use of any presumptions [46]. For the classification and regression tasks, respectively, the k-closest training instances in the feature space serve as the input [45, 46]. The outcomes depend on whether kNN is used for classification or regression [45, 46]. A class of belonging is the result of the kNN classifier [46] The provided data point is categorized according to the prevailing kind in its neighborhood [46]. The category with the highest frequency receives the input point [46].
Figure 8 shows a space with numerous data points or vectors that can be classified into two classes: class A and class B. If the red vector was to be classified, then the k-NN classifier would compute the spacial distance between the red vector and its neighbors for a given constant k value and assign the class to the unknown red vector according to the class in which the majority of neighboring vectors fall. In Fig. 8, for k equal to 3, the red vector will be classified as class B. The most cited method of computing the spatial distance between the data point p to be classified and its neighbors qn is the Euclidean formula (3) [45, 46].
| \(\text{d}\left(\text{p},\text{q}\right)=\text{d}\left(\text{q},\text{p}\right)=\sqrt{{\left({q}_{1}-{p}_{1}\right)}^{2}+{\left({q}_{2}-{p}_{2}\right)}^{2}+\dots +{\left({q}_{n}-{p}_{n}\right)}^{2}}=\sqrt{\sum _{i=1}^{n}{\left({q}_{i}-{p}_{i}\right)}^{2}}\) | (3) |
Fuzzy Classifier.
A supervised learning model called the fuzzy classifier system allows computational variables, outputs, and inputs to take on a spectrum of values over predetermined bands [46]. The fuzzy classifier system is trained by creating fuzzy rules that link the values of the input variables to internal or output variables [46]. Typical fuzzy classifier systems are combined with methods for credit assignment and conflict resolution [47]. The fuzzy classifier system creates suitable fuzzy rules using a genetic algorithm [47].
As seen in Fig. 9, continuous membership is displayed by fuzzy sets, and the degree (µ) to which a data point belongs to a given fuzzy set can be used to classify its membership. For instance, the degree of membership µ(960) on the close fuzzy set for the 690 mm in Fig. 9 is 0.7. Figure 9 further demonstrates that a data point may belong to more than one fuzzy set, with the degrees of membership to each set varying at the intersection locations since some fuzzy sets overlap.
Over and above these most cited machine learning classifiers, this study has also noticed that several other authors have used deep learning for classifying plants in plant disease/nutrient deficiency classification models. Deep learning has an advantage over other machine learning algorithms in the sense that feature learning and classification both occur within a deep learning model while they occur separately in the case of the above-listed classification algorithms [46, 47].
3.2 Previous Related Studies
Numerous studies have proposed plant disease/nutrient deficiency detection systems that have similar architecture as described in 3.1. Table 2 summarizes the recent publications that have been consulted. A few aspects have been noted in each related work publication, they are the plant/crop, the number of diseases/nutrient deficiencies a model can classify for a given crop, the type of those diseases/nutrient deficiencies, the classification accuracy results achieved, and the number of input camera sensors utilized for the classification model.
| Classification Model | Plant | Citation | No. of diseases | Disease type | Classification results | No. Of input cameras |
|---|---|---|---|---|---|---|
| SVM Classification | Maize | [12] | 1 | Not Specified | 79% accuracy | 1 |
| Grapefruit, Lemon, lime | [13] | 2 | Canker And Anthracnose Diseases | 95% accuracy for both | 1 | |
| Grape | [14] | 2 | Downy Mildew And Powdery Mildew | 88.89% accuracy for both | 1 | |
| Oil palm | [3] | 2 | Chimaera And Anthracnose | 97% and 95% accuracy respectively | 1 | |
| Potato | [15] | 4 | Late Blight And Early Blight | 95% for both | 1 | |
| Grape | [2] | 3 | Black Rot, Esca, And Leaf Blight | Not specified | 1 | |
| Tea | [16] | 3 | Not Specified | 90% accuracy | 1 | |
| Soybean | [13] | 3 | Downy Mildew, Frog Eye, And Septories Leaf | 90% accuracy average | 1 | |
| Tomato | [17] | 6 | Not Specified | 96% accuracy | 1 | |
| Rice | [18] | Not specified | Pests Diseases | 92% accuracy | 1 | |
| Soybean | [19] | 1 | Charcoal Rot | 90% accuracy | 1 | |
| Cucumber | [20] | 1 | Downy Mildew | Not specified | 1 | |
| Rice | [21] | 1 | Rice Blast | 93% accuracy | 1 | |
| Rice | [22] | 1 | Rice Blight | 80% accuracy | 1 | |
| Tea | [23] | 1 | Not Specified | 90% accuracy | 1 | |
| ANN Classification | zucchini | [24] | 1 | Soft-Rot | Not specified | 1 |
| Not specified | [25] | 4 | Alternaria Alternata, Anthracnose, Bacterial Blight, Cercospora Leaf Spot | 96% accuracy average | 1 | |
| Grapefruit | [26] | 3 | Grape-Black Rot, Powdery Mildew, Downy Mildew | 94% accuracy average | 1 | |
| Apple | [27] | 3 | Apple Scab, Apple Rot, Apple Blotch | 81% accuracy average | 1 | |
| Pomegranate | [28] | 3 | Bacterial Blight, Aspergillus Fruit Rot, Gray Mold | 99% accuracy average | 1 | |
| Not specified | [29] | 4 | Early Scorch, Cottony Mould, Late Scorch, Tiny Whiteness | 93% accuracy average | 1 | |
| Cucumber | [30] | 2 | Downy Mildew, Powdery Mildew | 99% accuracy average | 1 | |
| Pomegranate | [31] | 4 | Leaf Spot, Bacterial Blight, Fruit Spot, Fruit Rot | 90% accuracy average | 1 | |
| Groundnut | [32] | 1 | Cercospora | 97% accuracy | 1 | |
| Pomegranate | [33] | 1 | Not Specified | 90% accuracy | 1 | |
| Cucumber | [34] | 1 | Downy Mildew | 80% accuracy | 1 | |
| Rice | [35] | 3 | Bacterial Leaf Blight, Brown Spot, Leaf Smut | 96% accuracy average | 1 | |
| Citrus | [36] | 5 | Anthracnose, Black Spot, Canker, Citrus Scab, Melanose | 90% accuracy average | 1 | |
| Wheat | [37] | 4 | Powdery Mildew, Rust Puccinia Triticina, Leaf Blight, Puccinia Striifomus | Not specified | 1 | |
| k-NN Classification | Not specified | [38] | 5 | (YS) The Yellow Spotted, (WS) White Spotted, (RS) Red Spotted, (N) Normal, And (D) Discolored Spotted | 86% Accuracy | 1 |
| Groundnut | [39] | 5 | Early Leaf Spot, Late Leaf Spot, Rust, Early And Late Spot Bud Necrosis | 96% Accuracy | 1 | |
| Tomato, Corn, Potato | [40] | Not Specified | No Disease: Leaf Recognition | 94% Accuracy (Corn) 86% Accuracy (Potato) 80% Accuracy | 1 | |
| Tomato | [41] | 3 | Rust, Early, And Late Spot Bud Necrosis | 95% Accuracy | 1 | |
| Banana | [42] | 2 | Bunchy Top, Sigatoka | 99% Accuracy | 1 | |
| Tomato | [43] | 3 | Rust, Early And Late Spot Bud Necrosis | 97% Accuracy | 1 | |
| Rice | 4 | Bacterial Blight Of Rice, Rice Blast Disease, Rice Tungro, False Smut | 88% Accuracy Average | 1 | ||
| Fuzzy Classification | Mango | [44] | 3 | Powdery Mildew, Phoma Blight, Bacterial Canker | 90% Accuracy Average | 1 |
| strawberry | [45] | 1 | Iron Deficiency | 97% Accuracy | 1 | |
| Cotton, Wheat | [46] | 18 | Bacterial Blight, Leaf Curl, Root Rot, Verticillium Wilt, Anthracnose, Seed Rot, Tobacco Streak Virus, Tropical Rust, Fusarium Wilt, Black Stem Rust, Eaf Rust, Stripe Rust, Loose Smut, Flag Smut, Complete Bunt, Partial Bunt, Ear_Cockle, Tundo | 99% Accuracy Average | 1 | |
| Soybean | [47] | 1 | Foliar | 96% Accuracy | 1 | |
| Cotton | 3 | Bacteria Blight, Foliar, Alternaria | 95% Accuracy Average | 1 |
Table 2 shows that the majority of plant disease/nutrient deficiencies use single-camera sensor feeds. Several studies have used multi-camera infeed for different purposes. Yu [48] proposed a model for estimating leaf and stem shapes for nursery paprika plants using a multi-camera image acquisition system. This model is significant for monitoring the growth of a paprika plant and detecting deviations from the expected growth trend which is crucial for informing timely corrective actions. Ricardo [49] combined multiple camera inputs and deep learning to propose a human fall detection model which can help facilitate the first aid delivery to fall victims. Bojan [50] proposed a multicamera object detection and distance estimation. This system can have multiple in precision agriculture (in terrestrial navigation robotics) and the automotive industry for self-driving and self-parking features.
The coming conceptualize a proposed system, a multi-camera disease detection system, and model it on MATLAB.
This section conceptualizes and models a multi-camera Infeed Plant disease detection system. The distinction of this model from the previously existing models is the utilization of multiple infeed cameras which helps capture the round surface area of samples when fruits/stem images are used for disease classification.
4.1 Conceptualizing the proposed system
The proposed system operates as per the block diagram presented in Fig. 10. The first level has four image camera sensors to capture different angles of a sample to be classified. The next level has a sequencer whose purpose is to select each image sensor to feed the input image into the deep learning classification algorithm sequentially from cameras 1 to 4. The classification model classifies each image from cameras 1 to 4 and stores the resulting classes into variables of output 1 to 4 respectively. The last operation compares the resulting outputs and makes a final classification using the following simple rules:
-
If at least 1 output predicts that a sample is infected by a disease, set the final classification to a ‘diseased sample’.
-
If all the output variables predict a healthy sample, set the final classification to a ‘healthy orange’
-
If all the output variables predict a diseased sample, set the final classification to a ‘diseased sample’ and display an averaged classification efficiency.
The camera sensors should be arranged in a circular arrangement displaced 90˚ away from each other and equidistant from the position point of a sample under evaluation as shown in Fig. 11. This arrangement is crucial to capture the images of a sample from all directions effectively sending a 3-dimensional representation of a sample in a classification algorithm.
4.2 Modeling the proposed system
This section designs a deep learning multi-camera infeed plant disease detection model that classifies healthy and black-rot affect oranges. This is achieved through the utilization of the MATLAB deep learning toolbox. Here, only the training and validation of the classifier will be covered while the next section will be covered in the coming section. The first step towards training a neural network is to create a dataset. The dataset should be dissected into training and validating data. Orange fruits have been chosen to demonstrate the system proposed in this paper. Orange fruits are scientifically known as Citrus × Sinensis, from the Rutaceae family. Figure 12(a) shows a healthy orange which serves as a control in this pre-processing procedure. Figure 12(b) shows an orange affected by a disease called Citrus Black Rot.
This disease is caused by a fungal infection and can be distinguished by dark spots of decaying tissue on the surface of an orange. This disease induces early fruit drops and incurs serious yield losses at the end of a farming cycle. The disease is air-borne and water-borne, it spreads when spores are released in rainwater and wind during storms. It also spreads through contact with contaminated leaves, stems, fruits, or even debris carried by stormwater.
The coming sections address the design of the proposed classification model.
Creating a Training and Validation Dataset
. The training and validating datasets were sources from an open library website called Kaggle, the citrus and citrus diseases datasets respectively. The dataset folder has two sub-folder in it called ‘healthy oranges’ and ‘black-rot affected oranges’. Each folder contains 160 corresponding images. This dataset is loaded into MATLAB’s current folder before proceeding to the next step as shown in Fig. 13. During the operation of this model when training and validation are complete, the model will be getting input images through snapshots from live video feeds from Macally USB video class (UVC) webcams in Fig. 14. This webcam has a rated resolution and image transfer speed of 1080p and 30 fps respectively.
Training a Classification Model
The second step is training a classification model. For a purpose of this research paper, a deep learning convolutional neural network called GoogleNet was selected as the classification model. It is important to note that if other machine learning models were utilized for this classification problem other than deep neural networks, the image pre-processing, segmentation, and feature extraction would have had to be performed manually before training a classification model. However, since this project is utilizing deep neural network transfer learning and these tasks are performed inside a neural network. Only minor image pre-processing tasks are imperative in this case such as image dimension resizing to match the size of an input layer of GoogleNet. The MATLAB code used for transfer training a GoogleNet network is as follows:
OrangeDataSet = imageDatastore("dataset","IncludeSubfolders",true,"LabelSource","foldernames");
[TrainingData, ValidationData] = splitEachLabel(OrangeDataSet, 7.0);
ClassificationNetwork = googlenet;
DataInputSize = ClassificationNetwork.Layers(1).InputSize(1:2);
ResizedTrainingData = augmentedImageDatastore(DataInputSize,TrainingData);
ResizedValidationData = augmentedImageDatastore(DataInputSize,ValidationData);
FeatureLearner = ClassificationNetwork.Layers(142);
OutputClassifier = ClassificationNetwork.Layers(144);
NumberClasses = numel(categories(TrainingData.Labels));
OrangeFeatureLearner = fullyConnectedLayer(NumberClasses,"Name","Orange Feature Learner",...
"WeightLearnRateFactor",10, "BiasLearnRateFactor",10);
OrangeClassifierLayer = classificationLayer ("Name", "Orange Classifier");
Layer_Graph = layerGraph(ClassificationNetwork);
New_Layer_Graph = replaceLayer(Layer_Graph,FeatureLearner.Name,OrangeFeatureLearner);
New_Layer_Graph = replaceLayer(New_Layer_Graph,OutputClassifier.Name,OrangeClassifierLayer);
Minibatch = 10;
ValidationFrequency = floor(numel(ResizedValidationData.Files)/Minibatch);
Training_Option = trainingOptions("sgdm","MiniBatchSize",Minibatch,"MaxEpochs",10 ...
, "InitialLearnRate",3e-4, "Shuffle", 'every-epoch', "ValidationData",ResizedValidationData,...
"ValidationFrequency",ValidationFrequency, "Verbose", false, "Plots", "training-progress");
ClassificationNetwork = train network(ResizedTrainingData,New_Layer_Graph,Training_Option);
This Matlab program first exports the dataset, with its sub-folders, from MATLAB’s current folder to its workspace while setting folder names to classification labels. A pre-trained GoogleNet neural network is also loaded into the workspace. The image dataset is then resized into 224\(\times\)224 pixels (the input size of GoogleNet as seen in layer 1 in Fig. 15) and dissected into 70% and 30% training and validation data respectively. As seen in Fig. 15, the original feature learner (layer 142) and classifier (layer 144) of Googlenet are originally designed to classify 1000 different classes of images. The feature learner and classifier are renamed to orange feature learner and orange classifier as shown in Fig. 16 and GoogleNet is trained according to a specified training rule and options.
During training, the training progress was observed during the specified 10 different epochs (1 iteration per each epoch), refer to Fig. 17. As training progresses, the training efficiency increase to maximum by the 4th epoch, and the validation efficiency increase steadily as epochs increase which means that the model becomes more accurate as it processes more and more images. Similarly, the training and validation losses decrease as epochs increase which tells the model is learning correctly. The validation efficiency and learning duration were 98.58% and 163 seconds respectively, at the end of the last epoch (refer to Fig. 18).
At this point, a deep neural network has been successfully transfer-trained to classify between healthy and black rot-infected oranges. The classification model has also been cross-validated utilizing 30% of the initially created dataset to see if it does classify the sample images correctly. The coming section discusses the practical testing of the model utilizing the practical orange samples.
Testing a Multi-camera Input Healthy-Black Rot-Infected Oranges Classification Model
At this point, the design and validation of a neural network orange fruit classifier are complete. Now it is time to test this model on a practical example to see if it can classify correctly between healthy and black rot-affected oranges. Unfortunately, at the time of presenting this work, the authors were not in a position of a stand-alone microprocessor, hence a Lenovo ThinkPad was used as a host microprocessor to execute the training and testing MATLAB program. This computer has two USB ports and hence can support two UVC webcams. This was a major limitation encountered during this research study. Hence, for demonstration purposes, two UVC webcams positioned 180˚ opposite to each other on either side of a sample orange position were utilized for input images (refer to Fig. 19).
Therefore, input images were acquired by acquiring sample images and passing them to an already-trained model in the previous section for classification. The testing orange samples were collected from a local vendor where they have already been haversted. The oranges with the desired black-rot symptoms and disease distribution were specially requested from a vendor. The following MATLAB code was used to archive the purpose of this section:
the_camera1 = webcam(1);
the_camera2 = webcam(2);
RealTimeImage = figure;
while ishandle (RealTimeImage)
single_image1 = snapshot(the_camera1);
single_image2 = snapshot(the_camera2);
Resized_single_image1 = imresize(single_image1,DataInputSize);
Resized_single_image2 = imresize(single_image2,DataInputSize);
[predicted_item1, probability1] = classify(ClassificationNetwork, Resized_single_image1);
[predicted_item2, probability2] = classify(ClassificationNetwork, Resized_single_image2);
subplot(1,2,1),imshow(Resized_single_image1);
title({char(predicted_item1),num2str(max(probability1),2)})
subplot(1,2,2),imshow(Resized_single_image2);
title({char(predicted_item2),num2str(max(probability2),2)});
drawnow;
if isequal( predicted_item1, 'black-rot affected oranges')
fprintf('Diseased Orange Detected!\n');
fprintf('Final Classification = Black rot affected orange!\n');
elseif isequal( predicted_item2, 'black-rot affected oranges')
fprintf('Diseased Orange Detected!\n');
fprintf('Final Classification = Black rot affected orange!\n');
else
fprintf('Healthy Orange Detected!\n');
fprintf('Final Classification = Healthy orange!\n');
end
end
This program accepts snapshot images from the webcams (refer to Fig. 20) connected to a microprocessor of a host computer Lenovo Laptop. These two infeed cameras capture the images of a sample orange on either side as shown in Fig. 19. Each of these two images passes through a previously trained classifier, classified individually and the final classification results are formed by consolidating the individual results from webcam 1 and 2. If at least 1 camera captured an image classified as black rot-infected, the final result is set to a diseased orange. Otherwise, the orange is classified as healthy orange. Three different orange samples were collected to test the model operation, a healthy orange and two diseased oranges where black rot symptoms have spread evenly throughout the whole orange surface area on only a small surface area on the other. The classification results of each sample are shown in Figs. 21–26.
Figure 21 show two images of a healthy orange captured from two opposite angles by camera 1 and 2 and each of them was classified correctly as a healthy orange. Figure 22 shows the final classification results of this sample which are derived from consolidating the results of each camera angle. Similarly, Fig. 23 show two images of a diseased orange (with uneven black rot symptoms distribution) captured from two opposite angles by camera 1 and 2, and each of them was classified correctly as a diseased orange. Figure 24 shows the final classification results of this sample which are also derived from consolidating the results of each camera angle. Lastly, Fig. 25 show two images of a diseased orange (with even black rot symptoms distribution) captured from two opposite angles by camera 1 and 2. Figure 26 shows the final classification results which were formed similarly to those in Figs. 22 and 24. The classification accuracy of this multi-camera input fruit-disease classification model was 100%, and it has the added advantage of being capable of classifying correctly even the samples with an uneven disease distribution on their surface areas.
In this paper, a novel multi-camera input plant disease detection system was proposed and tested on orange fruits infected with black rot disease. This proposed system was conceptualized to have four camera inputs however, due to a limitation of a host computer used as a microprocessor, the input cameras were scaled down to two. The proposed system achieved its objective by correctly classifying all the orange fruit samples presented to it regardless of the black rot disease distribution on the surface area of the orange. All the methods (dataset collection, model training, model testing, etc.) were performed in arccodance with the relevant guidelines and regulations.
Future work recommendations of this study include investigating different arrangements of input cameras including the top and bottom views other than the side views. Post-harvest benefits of this system should also be investigated. What if the single input camera was maintained and induced a rotation of a sample under evaluation (orange in the case of this study) such snapshots of all different sample surfaces are captured? What would be the classification cycle time for multi-camera input models versus single camera infeed where samples are to be rotated?
The dataset that was used to train the classification model proposed in this paper has been sourced from Kaggle which is one of many open-source libraries which are available freely for any scholars to access. It is available from https://www.kaggle.com/. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
- Swaminathan, B., Identification of Plant Disease and Wetness/Dryness Detection.
- Anju, S., et al., Various Approaches for Plant Disease Detection. 2021.
- Shruthi, U., V. Nagaveni, and B. Raghavendra. A review on machine learning classification techniques for plant disease detection. in 2019 5th International conference on advanced computing & communication systems (ICACCS). 2019. IEEE.
- Swain, S., S.K. Nayak, and S.S. Barik, A review on plant leaf diseases detection and classification based on machine learning models. Mukt shabd, 2020. 9(6): p. 5195-5205.
- Prashar, N. A Review on Plant Disease Detection Techniques. in 2021 2nd International Conference on Secure Cyber Computing and Communications (ICSCCC). 2021. IEEE.
- du Preez, M.-L., 4IR and Water Smart Agriculture in Southern Africa: A Watch List of Key Technological Advances. 2020.
- Perez-Sanz, F., P.J. Navarro, and M. Egea-Cortines, Plant phenomics: An overview of image acquisition technologies and image data analysis algorithms. GigaScience, 2017. 6(11): p. gix092.
- Padmavathi, K. and K. Thangadurai, Implementation of RGB and grayscale images in plant leaves disease detection–comparative study. Indian Journal of Science and Technology, 2016. 9(6): p. 1-6.
- Kolhalkar, N.R. and V. Krishnan, Mechatronics system for diagnosis and treatment of major diseases in grape vineyards based on image processing. Materials Today: Proceedings, 2020. 23: p. 549-556.
- Wang, A., W. Zhang, and X. Wei, A review on weed detection using ground-based machine vision and image processing techniques. Computers and electronics in agriculture, 2019. 158: p. 226-240.
- Anger, P.M., et al., Implementation of an open source algorithm for particle recognition and morphological characterisation for microplastic analysis by means of Raman microspectroscopy. Analytical Methods, 2019. 11(27): p. 3483-3489.
- Panigrahi, K.P., et al., Maize leaf disease detection and classification using machine learning algorithms, in Progress in Computing, Analytics and Networking. 2020, Springer. p. 659-669.
- Kashyap, Y. and S. Shrivastava, ROLE OF VARIOUS SEGMENTATION AND CLASSIFICATION ALGORITHMS IN PLANT DISEASE DETECTION.
- Dwari, A., et al., Smart Solution for Leaf Disease and Crop Health Detection, in Advances in Intelligent Computing and Communication. 2021, Springer. p. 231-241.
- KUMAR, N.K. and A. PRATHYUSHA, SOLANUM TUBEROSUM LEAF DISEASES DETECTION USING DEEP LEARNING.
- Hossain, S., et al. Recognition and detection of tea leaf's diseases using support vector machine. in 2018 IEEE 14th International Colloquium on Signal Processing & Its Applications (CSPA). 2018. IEEE.
- Coulibaly, S., et al., Deep neural networks with transfer learning in millet crop images. Computers in Industry, 2019. 108: p. 115-120.
- Cherukuri, N., et al. Automated Classification of rice leaf disease using Deep Learning Approach. in 2021 5th International Conference on Electronics, Communication and Aerospace Technology (ICECA). 2021. IEEE.
- Khalili, E., et al., Machine learning techniques for soybean charcoal rot disease prediction. Frontiers in plant science, 2020. 11: p. 590529.
- Patil, M.P., Nearest neighborhood approach for identification of cucumber mosaic virus infection and yellow sigatoka disease on banana plants. 2022.
- Orchi, H., M. Sadik, and M. Khaldoun, On using artificial intelligence and the internet of things for crop disease detection: A contemporary survey. Agriculture, 2021. 12(1): p. 9.
- Zhang, D., et al., Detection of rice sheath blight using an unmanned aerial system with high-resolution color and multispectral imaging. PloS one, 2018. 13(5): p. e0187470.
- Yashodha, G. and D. Shalini, An integrated approach for predicting and broadcasting tea leaf disease at early stage using IoT with machine learning–a review. Materials Today: Proceedings, 2021. 37: p. 484-488.
- Zubler, A.V. and J.-Y. Yoon, Proximal methods for plant stress detection using optical sensors and machine learning. Biosensors, 2020. 10(12): p. 193.
- Thushara, B.L. and T.M. Rasool, Analysis of plant diseases using expectation maximization detection with BP-ANN classification. XIII (VIII), 2020.
- Thakur, P.S., et al., Trends in vision-based machine learning techniques for plant disease identification: A systematic review. Expert Systems with Applications, 2022: p. 118117.
- Khan, A.I., S. Quadri, and S. Banday, Deep learning for apple diseases: classification and identification. International Journal of Computational Intelligence Studies, 2021. 10(1): p. 1-12.
- Das, A.J., et al., Microbiome Analysis of the Rhizosphere from Wilt Infected Pomegranate Reveals Complex Adaptations in Fusarium—A Preliminary Study. Agriculture, 2021. 11(9): p. 831.
- Gaikwad, S.S., Fungi classification using convolution neural network. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 2021. 12(10): p. 4563-4569.
- Priya, D., Cotton leaf disease detection using Faster R-CNN with Region Proposal Network. International Journal of Biology and Biomedicine, 2021. 6.
- Joshi, B.M. and H. Bhavsar, Plant leaf disease detection and control: A survey. Journal of Information and Optimization Sciences, 2020. 41(2): p. 475-487.
- Gangadevi, G. and C. Jayakumar. Review of Classifiers Used for Identification and Classification of Plant Leaf Diseases. in Applications of Artificial Intelligence in Engineering. 2021. Springer.
- Vučić, V., et al., Composition and potential health benefits of pomegranate: a review. Current pharmaceutical design, 2019. 25(16): p. 1817-1827.
- Sahni, V., S. Srivastava, and R. Khan, Modelling techniques to improve the quality of food using artificial intelligence. Journal of Food Quality, 2021. 2021.
- Patidar, S., et al. Rice plant disease detection and classification using deep residual learning. in International Conference on Machine Learning, Image Processing, Network Security and Data Sciences. 2020. Springer.
- Sharif, M., et al., Detection and classification of citrus diseases in agriculture based on optimized weighted segmentation and feature selection. Computers and electronics in agriculture, 2018. 150: p. 220-234.
- Hayit, T., et al., Determination of the severity level of yellow rust disease in wheat by using convolutional neural networks. Journal of Plant Pathology, 2021. 103(3): p. 923-934.
- Jasim, S.S. and A.A.M. Al-Taei, A Comparison Between SVM and K-NN for classification of Plant Diseases. Diyala Journal for Pure Science, 2018. 14(2): p. 94-105.
- Vaishnnave, M., et al. Detection and classification of groundnut leaf diseases using KNN classifier. in 2019 IEEE International Conference on System, Computation, Automation and Networking (ICSCAN). 2019. IEEE.
- DAYANG, P. and A.S.K. MELI, AS: Evaluation of image segmentation algorithms for plant disease detection. Int. J. Image Graph. Signal Process.(IJIGSP), 2021. 13(5): p. 14-26.
- Agarwal, M., S.K. Gupta, and K. Biswas, Development of Efficient CNN model for Tomato crop disease identification. Sustainable Computing: Informatics and Systems, 2020. 28: p. 100407.
- Devi, R.D., et al. IoT enabled efficient detection and classification of plant diseases for agricultural applications. in 2019 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET). 2019. IEEE.
- Harakannanavar, S.S., et al., Plant Leaf Disease Detection using Computer Vision and Machine Learning Algorithms. Global Transitions Proceedings, 2022.
- Sutha, P., et al., Plant Disease Detection Using Fuzzy Classification. Annals of the Romanian Society for Cell Biology, 2021: p. 9430-9441.
- Altıparmak, H., et al. Fuzzy classification for strawberry diseases-infection using machine vision and soft-computing techniques. in Tenth International Conference on Machine Vision (ICMV 2017). 2018. SPIE.
- Toseef, M. and M.J. Khan, An intelligent mobile application for diagnosis of crop diseases in Pakistan using fuzzy inference system. Computers and Electronics in Agriculture, 2018. 153: p. 1-11.
- Defrianto, D., et al., Fluorescence spectrum analysis on leaf and fruit using the ImageJ software application. Science, Technology & Communication Journal, 2022. 3(1): p. 1-6.
- Zhang, Yu, et al. "Estimating 3D leaf and stem shape of nursery paprika plants by a novel multi-camera photography system." Sensors 16.6 (2016): 874.
- Espinosa, Ricardo, et al. "A vision-based approach for fall detection using multiple cameras and convolutional neural networks: A case study using the UP-Fall detection dataset." Computers in biology and medicine 115 (2019): 103520.
- Strbac, Bojan, et al. "YOLO multi-camera object detection and distance estimation." 2020 Zooming Innovation in Consumer Technologies Conference (ZINC). IEEE, 2020.
No competing interests reported.