In the current study, three different predictive models (SS, MS, and FS model) for DON contamination levels at the regional scale in oats in Sweden were developed. Model classification accuracy showed to be high, ranging from 0.7 to 0.9 depending on different years and models. The developed models can provide valuable information to three different stakeholder groups in the oat supply chain; farmers, crop collectors, and food safety authorities, as a tool that can help in the management of mycotoxins in the oats supply chain and risk-based testing. Results showed that 1) weather variables are the most important for predicting DON contamination in oats, 2) adding relevant agronomical and site specific factors, such as crop variety, crop rotation, soil type and DON contamination condition in the previous year could improve the performance of the models, 3) good predictions could be made already in June by using the SS-model, as based on internal validation, and 4) rainfall, relative humidity, and wind speed in different growing stages as well as crop variety and elevation were the most important features for predicting DON contamination levels in oats. However, predicting individual years not included in the training of the models proved to be difficult.
To date, few studies have incorporated weather, agronomical, and site-specific data to predict the regional DON contamination in oats using machine learning. But many studies have paved the way for using these data for DON contamination prediction. One study modeling the effects of weather features on DON contamination in oats indicated that the model accuracy could be improved if more factors (such as field tillage and the soil type) were included, in addition to the weather data (Marzec-Schmidt et al., 2021). One study investigated the association of several agronomic factors (including harvest date, crop season, county, farming system, moisture, test weight, oats variety, and previous crop) to the occurrence of Fusarium mycotoxins in Irish oats (Kolawole et al., 2021). This study concluded that the level of DON was modelled best by the variables of the previous crop and oat variety, and indicated the importance of exploring crop rotation in future studies. Another study investigated the prevention and control of mycotoxins in grains, and emphasized the importance of matching crop varieties to a specific agro-ecological zone with specific weather conditions (Matumba et al., 2021), indicating the necessity of linking weather data to the crop variety for model development. In our study, we used weather factors and relevant agronomical and site-specific factors as model inputs. Similarly to the previous studies, oat variety was the non-weather feature that had the highest impact on the model output, whereas information on previous crop could not explain much of the DON variation in our study. One of the reasons for the lack of influence of pre-crop information could perhaps be the aggregation of the data to the weather grids. The comparison of model performance using weather data, with and without agronomical and site specific factors confirmed that, in improving the performance of the DON predictive models, weather variables are the most important factors, and adding agronomical and site specific factors could further improve the overall classification accuracy (from 0.72 to 0.73 using dataset 1, from 0.81 to 0.95 using dataset 2). This was in line with the expectation of a previous study which suggested that DON prediction in oats could potentially be improved by combining weather-based risk index outputs with agronomic factors (Persson et al., 2017).
The feature impact analysis indicated that rainfall, relative humidity, and wind speed in different oat growing stages as well as oat variety and elevation were the most important features for predicting DON contamination levels in oats. In general, weather variables (e.g., temperature, rainfall) in December of the previous year, weather variables (e.g., humidity, wind speed) around end of June (close to flowering season), and weather variables (e.g., humidity, temperature) around August (week 31,32,and 33 close to harvest season) were the most important features (Fig. 7). These results are in line with Hjelkrem et al., (2017) who showed that dry periods during germination (March to April) contribute to high DON contamination of oats, and warm, rainy and humid weather around flowering contributed to high DON accumulation in oat. Marzec-Schmidt et al. (2021) also confirmed that high relative humidity and precipitation around flowering correlated with high DON contamination levels in oat. Interestingly, the site-specific characteristics associated with high DON contamination levels in our study, high elevation and sandy soils, is related to dry conditions which may indicate that draught stress might have been important in the data set from 2016–2017.
A previous study applied different models, including statistical analysis and machine learning techniques, for DON prediction in oats, resulting in different model performances (Lindblad et al., 2012). Their results showed that very little of the variation in DON levels could be explained by agronomical or weather factors, and it was not possible to predict DON levels based on these variables. This low model performance could have been caused by the unbalanced data related to DON contamination, meaning only few records were related to high DON values and most of the records were related to low DON values. Poor model performance for predicting high mycotoxin contamination levels due to unbalanced datasets have also been encountered in other studies (Liu et al., 2018; Liu et al., 2021). Their results showed that the developed models have higher performance for predicting the samples with low-level contamination than for the samples with high-level contamination. Our study applied a machine learning technique (the random forest algorithm) to handle unbalanced data, resulting in a relatively balanced classification accuracy in each DON contamination level (high, medium, low). Using both datasets 1 and 2, the model showed good classification accuracy of at least 0.7. It means that a rather large proportion of the sites for each of the three DON contamination levels were correctly classified. However, for medium and high contamination levels, still a portion of the sites (< 40%) was wrongly predicted. One reason could be that the data are very skewed, with a large number of sites belonging to the low level and much fewer sites belonging to the two other levels (mid, high) also in our data. A higher total classification accuracy than 0.7 could be reached for the predictive model if sacrificing the prediction accuracy of high and medium-contamination levels. There are usually trade-offs between the prediction accuracy of each contamination level when using Machine Learning predictive algorithms. If a model was able to predict the high and medium-contamination levels with high accuracy, the accuracy in predicting the low-contamination class was sacrificed, and vice visa. In our study, most of regions with high and medium contamination level were correctly predicted while some of the low-contamination regions were also predicted as high and medium contamination level. The models were designed to predict the three contamination levels and at the same time reduce the number of false negatives, i.e. regions that are predicted as having low contamination level but that in fact have medium or high contamination level. This is done at the expense of more false positives, i.e. regions that are predicted as having medium or high contamination level, but in fact have a low contamination level. For oat supply chain stakeholders, it is more important not to miss a region with high contamination, than to erroneously regard a non-contaminated region as contaminated. In addition, instead of using three class setting of DON contamination level, a better prediction result could be obtained by setting two class level (e.g. when using 500 ppb as the threshold value, the total prediction accuracy reached 0.84 for both internal and external validation, results not presented). The three models we designed could easily be adapted to achieve a higher prediction accuracy for the low contamination class or the higher total classification accuracy, at the expense of lower prediction accuracy on the high and medium contamination levels, depending on what the stakeholders prioritize.
The SS, MS, and FS models were developed for different oat growing periods, and the results of these models provided several insights. First, on the internal validation using dataset 1 (20% of the data 2012–2019 except 2016), results showed a total classification accuracy of > 0.7 for each of the three models (relatively good performance for all levels). This indicated that good predictions could be made already by June (SS-model). This result is in line with the study from Hjelkrem et al., (2017). Their study showed that the prediction model using only pre-flowering weather data could adequately forecast the DON contamination in oat. Although good predictions could be made already by June, it is recommended to use weather variables in the full season when implementing the model in practice whenever possible. This is because DON contamination was mostly associated with the weather features around flowering as well as close to harvest (Hjelkrem et al., 2017).
The SS, MS, and FS models resulted in relatively good performance for three contamination levels, but low performance for all three contamination levels with the external validation (dataset 1, 2016). The reason for the poor model performance with the external validation (dataset 1, 2016) could be the large variation in DON contamination distribution over the years (Table 1). The trained model assigned weights (e.g., opposite to the rates of three contamination levels) to different DON contamination levels. When the distribution of DON contamination levels of the external validation dataset was very different from the trained dataset, and the model used the pre-assigned weights, it could have resulted in a poor performance in the external validation. In this study, data were split randomly into a training set (80%) for model learning where five-fold cross-validation was performed for model hyperparameter tuning. Then, the models were validated on “new” data using the internal and external validation dataset (Fig. 2). Other cross validation methods such as leave-one-year out could be applied (Hjelkrem et al., 2017). However, DON contamination distribution varied greatly between the years, and there was a clear overall trend with reduced contamination over the years (Table 1). Leave-one-year out validation is still problematic in achieving a high validation accuracy due the large changes of DON distribution in the three contamination levels. When considering the proposed implementation of the models in practice, the validation results are perhaps not truly reflecting the prediction performance of a prediction for a new coming year. The prediction performance for a coming year is highly related to the distribution of contamination levels. To further improve the model performance, adding data in more years to extend the training dataset to train the model for learning as many patterns as possible could be one possibility. In addition, future studies could use binary levels instead of multi-levels for DON contamination levels to increase the model classification accuracy.
One limitation of our study was that we did not consider all biological relevant factors for DON prediction in our model due to lack of related data. Other relevant factors could include crop management practices, such as fertilization, irrigation and pest control (Munkvold, 2014), the use of fungicides against Fusarium spp. around flowering (Van der Fels-Klerx et al., 2021; Liu et al., 2018; Torelli et al., 2012) and the harvest conditions (such as timely harvest). This information needs to be collected via field surveys with oat farmers. We have been able to get a large amount of data in this study at the expense of detailled information on agronomical features. To collect data from incividual farmers is another concept, which takes time is prone to introduce faults and that the dataset will be smaller. In future studies, it might be of interest to determine whether the inclusion of other variables could further improve the performance of the models. Variables from open sources could also be included, such as satellite data, which could provide a great asset to further improve the prediction models (Yudarwati et al., 2020). Another limitation with our study is the aggregate data, as some of the variation in the additional site-specific information might be smoothed out, reducing possible more local relationships. Our study only considers DON contamination of oats, not other mycotoxins, and future studies could predict the multi-mycotoxin contamination when data related to other mycotoxins were available as well. Measures that could be taken to limit the contamination could also be of interest to add, as decision making tool in the next step.
It can be concluded that the use of machine learning algorithms for DON prediction in oats, using contamination levels at the regional level in Sweden provides good prediction results when considering several years. Unfortunately, the models were not general enough to manage to predict DON-levels from individual years not included in the training of the model, i.e., model performance did not as high as internal validation when do external validation using leave one year out approach. One reason could be the DON contamination levels change as much as they do in the investigated years. Under such circumstances, it seems to be a better strategy to only use two risk-levels, above or below a certain level. However, this strategy has not been tested adequately in this investigation. Such models could be used as regional risk-assessment tools for farmers, crop collectors, and food safety inspectors for logistics in the oats supply chain, improved mycotoxin control, and risk-based testing. Given EC regulation 2017/625, food safety authorities need to apply risk-based control. Regions with a medium of high DON contamination level can be sampled and tested for the presence of DON more intensively than regions with low predicted contamination class. Collectors and food safety authorities of oats can also use the model predictions for deciding on testing frequencies, and they can use the predictions for routing and logistics in their oats supply chain.