FruitPhenoBox – a device for rapid and automated fruit phenotyping of small sample sizes

doi:10.21203/rs.3.rs-2632704/v1

Download PDF

Method Article

FruitPhenoBox – a device for rapid and automated fruit phenotyping of small sample sizes

https://doi.org/10.21203/rs.3.rs-2632704/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background: Fruit appearance of apples is cultivar-specific and one of the main criteria for consumer choice. Consequently, appearance is an important selection criterion in the breeding of new cultivars. It is also used for the description of older varieties or landraces. In commercial apple production, sorting devices are used to classify a high number of fruits from a few cultivars. In contrast, the description of fruits from germplasm collections or for breeding is based on a few fruits from many cultivars and is mostly performed visually by pomology experts. Therefore, such visual ratings are laborious, often difficult to compare and remain subjective.

Results: Here we report on a morphometric device, called the FruitPhenoBox, for automated fruit weighting and appearance description using computer-based analysis of five images per fruit. The recording of approximately 100 fruits each from 15 apple cultivars using the FruitPhenoBox was achieved rapidly, with an average handling and recording time below eleven seconds per fruit. The comparison of the fruits from the 15 apple cultivars identified significant differences in shape index, fruit width, height and weight. Fruit shape was characteristic for each cultivar, while fruit coloration showed larger variation within sample sets. Assessing a subset of 20 randomly selected fruits per cultivar, fruit height and width were described with a relative margin of error of 2.6% and 2.1% and 6.2%, respectively, calculated from the mean value of all available fruits.

Conclusions: The FruitPhenoBox allows for the rapid and consistent description of fruits of a single apple genotype. By relating the relative margin of error for fruit width, height and weight description with different sample sizes, it was possible to determine an appropriate fruit sample size to efficiently and accurately describe the recorded traits. Therefore, the FruitPhenoBox is a useful tool for breeding and the description of apple germplasm collections.

Malus x domestica

imaging

apple

fruit shape

fruit weight

fruit color

pomology

breeding

genetic resources

The appearance of apples for fresh consumption is one of the first drivers of the consumers' choice in supermarket [1], as the judgment of inner properties like taste and firmness requires fruit dissection. Fruit appearance, defined as fruit shape, coloration and size, is cultivar-specific [2] and an important trait when selecting novel cultivars in breeding programs [3, 4]. It is also an important trait for the description of older cultivars or landraces [5–7]. Fruit appearance is assessed using pomological descriptors, requiring the visual assessment of single fruits. In commercial production, appearance is exploited for classifying fruits into different categories based on weight, caliber and coloration. Nowadays, modern devices allow for assessing also defects and storage diseases [8]. Such information is extracted by analyzing several pictures of the same fruit, and sorting devices are designed for handling tons of fruits from a few cultivars. However, in apple breeding programs and for the description of germplasm collections, the requirements in terms of sample size are different: the appearance of a few fruits from many diverse cultivars, accessions or landraces, sometimes even from single trees needs to be described. The efforts required for using a commercial sorting device for this purpose may result large and the raw data used for fruit sorting (e.g. pictures) may not be available. Further, the space required as well as the acquisition costs of a sorting device can be prohibitive if purchased only for the description of a limited number of fruits per genotype. Therefore, the aim of this study was the development of a compact device enabling automated, rapid and consistent phenotypic description of apple fruits. In particular, we aimed at i) the weighting and imaging of individual fruits from different angles; ii) the development of a script for automated image analysis to extract fruit shape features for each investigated cultivar; iii) the generation of a factsheet for summarizing the appearance of each apple cultivar; iv) the assessment of the performance of the FruitPhenoBox; and v) the determination of the number of fruits per sample required for a confident fruit appearance description.

Performance assessment

Images and weights of approximatively 1,500 apple fruits from 15 cultivars were recorded over two sessions by a single operator with an average elapsed time of 10.58 seconds (median 9 s) per recorded fruit (replacing the fruit in the FruitPhenoBox, followed by image and weight recording). The side views of all fruits per cultivar generated an average fruit shape (including standard deviation) that was distinctive for each cultivar (Fig. 1). Average color distribution of the hue value per cultivar showed large deviations within cultivars (Fig. 2). One-way ANOVAs on height, width, weight and shape index revealed that there are significant (p < 2e-16) differences between cultivars for all traits. Post hoc Tukey test generated up to ten significance groups (p < 0.05) per investigated trait (Fig. 3).

Determination Of The Optimal Sample Size

Testing the effect of different sample sizes in the 15 apple cultivars on the relative margin of error of the fruit description indicated a general improvement with increasing sample sizes, particularly between n = 5 and n = 20 (Fig. 4). The average fruit height, width and weight calculated from 20 randomly selected fruits was described with a relative margin of error of 2.6%, 2.2% and 6.2%, respectively, of the mean values calculated using all available fruits of a cultivar. The largest margins of error were observed for fruit height of the cultivar 'Braeburn' (3.5%), and for both fruit width and weight of the cultivar 'Golden Reinders' (2.9% and 8.3%, respectively). The cultivars showing the best lowest margins of error were 'Bonita' for fruit heigth (1.6%), and 'Mariella' for fruit width and weight (1.4% and 3.9%, respectively).

The FruitPhenoBox is a compact device enabling rapid and consistent description of apple fruits using a system based on a combination of Matlab and R scripts. Compared to existing industrial apple sorting devices, this system is inexpensive, more compact, is operable by a single user and can be implemented for the description of fruits also in small breeding programs, as well as in projects aiming at the description of samples consisting in a relatively small number of fruits from a large number of cultivars, accession or landraces.

The use of a barcode-based labeling of the fruit samples avoids naming mistakes. A total of approximatively 1,500 fruits have been processed within a few hours, achieving an automated, fast and objective pomological description. The FruitPhenoBox offers the unique feature of storing the fruit images as high-quality TIF image files, ensuring the possibility of documenting the description process, and to allow a later re-analysis in case additional information needs to be extracted using novel algorithms, methods, or novel reference shape descriptors. The images generated could be used for training machine learning algorithms for an automated cultivar recognition based on fruit features.

Our analysis performed on approximately 100 fruits from 15 cultivars each confirmed that fruit shape is characteristic for each cultivar [2]. Color distribution showed larger variation within cultivars, confirming the common knowledge that despite being cultivar characteristic, fruit coloration can be affected by external factors as exposition to sun and showing therefore within-tree variability [9–11].

Furthermore, our investigation showed that the fruit sample size had stronger influence on the fruit weight description confidence compared to fruit width or height. This reflects the variety inherent uncertainty of shapes and therefore weights. We propose a minimal sample size of 20 fruits per cultivar for consistent fruit description. This number of fruits can be harvested on a single tree. The minimal sample size can be reduced for cultivars known to produce homogenous fruits, while it should be increased for cultivars producing more inhomogeneous fruits.

The factsheets generated with this data are valuable when describing apple genetic resources as it rapidly allows to generate a catalog providing a simple overview of the most important fruit descriptors. With some adaptation, the FruitPhenoBox and the analysis scripts could be rapidly adjusted to capture also other fruits or vegetables, for which the appearance is also a highly relevant trait that needs to be considered while breeding novel cultivars.

The FruitPhenoBox allows the morphometric description of apple fruits. It allows the rapid phenotyping of a large number of samples consisting in few fruits each, as done for the description of germplasm collections or breeding material at early stages, where single trees per genotype are generally assessed. It combines low hardware acquisition costs (below 5,000 USD), a compact design that allows single-user operation and avoidance of user bias by an automated analysis. It generates consistent data for describing cultivars fruit appearance and stores images that can be reanalyzed. If the collected shape features are connected with genotypic information for a genome-wide association study, this could help unraveling the genetic architecture of fruit appearance traits. Its use is beneficial to apple breeding and conservation programs.

Hardware description

The FruitPhenoBox (Figure 5) consists of an external frame of 850x850x660 mm built of aluminum profiles (30x30mm) mounted on a wooden base plate. Five rgb cameras, one from each side and one from the top (DFK 33UX273, 1296 x 1080 pixel + TCL 0814, www.theimagingsource.com) are mounted on the aluminum profiles and oriented toward the center, where a weight scale with serial connection port (Kern PCB 1000-1, www.kern-sohn.com) and a pedestal is placed. Cameras and scale are connected to a common personal computer. Two LED-rings (YONGNUO YN308, www.yongnuo.eu) are providing homogeneous lightning from the bottom and from the top. A barcode scanner (QuickScan Lite QW2400, www.datalogic.com) is used for registering unique ID/Cultivar name/accession number/QR code and a USB pedal (6210 - 0084 USB Footswitch, www.herga.com) is used as trigger. White PVC panels (3mm) with camera holes (39 mm diameter fitting camera lens) shield the FruitPhenoBox from external light disturbance.

Image acquisition and analysis

Image acquisition and analysis are steered by two different Matlab [12] scripts that were compiled to run in the Matlab Compiler Runtime 2018b environment. The first script controls the hardware as follows: Prior photograph shooting, camera parameters are defined to fix white balance parameters at first run and reload them afterwards. Prior to the measurement process, a unique barcode is assigned to the cultivar of each single tree name (ID) and scanned with a 2D-scanner. A fruit is placed on a pedestal on the scale in the center of the FruitPhenoBox, and the foot pedal triggers the snapshot from the five cameras as well as the recording of the fruit weight. This latter is recorded in the image metadata. Image names are generated concatenating ID, camera number, an incremental number starting from one, the shooting date and time, and are stored in 48-bit TIF format in a folder "images" and distributed in subfolders according to the barcode and the camera used for shooting (e.g. images/ID/cam1). Each snapshot generated five lossless compressed (packbits) TIF-images (1296x1080, 48-bit, each sizing approximatively 8 Mbytes each). Once all fruits are imaged, each single image is then segmented using a second Matlab script to remove the image background, define fruit position and outline its contours. Segmentation is done using hsv-color space in several steps: the ratio of saturation and value was thresholded with 0.2 and intersected with all values of saturation above 0.19 resulting in the first mask m_1. Additionally for each camera an image of the empty FruitPhenoBox (taken without apple) is subtracted from the actual image and the difference segmented with Otsu’s method [13] resulting in a second mask m_2. The set union of both masks was calculated as apple area in each image. Distribution of pixels color of the fruit area was analyzed and used to generate a histogram. Width and height were calculated from each single image utilizing the real-world pixel size of calibration images.

Fruit contours were converted from cartesian to polar coordinates (around their centroid) and interpolated to integer degree values for simple further analysis. For side views the contour was corrected to maximize symmetry to the y-axis by rotation around its centroid, the fruit mask was subsequently rotated by the determined angle. As an indicator of fruit symmetry, the fruit masks of side views were mirrored vertically at the axis parallel to the y-axis through their centroid. The area of the differences of original mask and mirrored mask was divided by the double area of the original mask yielding 0 for perfect symmetric shapes and 1 as maximum for this indicator for very unsymmetric fruit shapes. The radius of the contours was compared to the shapes provided in pomological description manuals [7], assigning fruit shape to one of the 13 available categories based on the highest correlation. Top view contours coordinates were transformed into polar coordinates with origin in the center of mass of the apple area. In the following these coordinates were used for determination of circumcircle, mean radius and incircle as well as the deviations of the radius to the mean circle. The standard deviation of these deviations was calculated and normalized to radius 1. Then the single image analyses were grouped per fruit, and then a text file containing the information of the variables listed in Table 1 per fruit of each single cultivar was generated.

Table 1: Apple fruit output variables of the second Matlab script assessed with the FruitPhenoBox.

Variable name	Description
ShapeClassCorr_side	normalized cross-correlation coefficient of side shape with reference shapes
ShapeClass_top	id of best-fitting reference shape, top view
weight_g	mass of apple [g]
diameter_deviation_cm	maximal deviation of diameter from mean diameter [cm]
diameter_cm	mean diameter [cm]
diameter_min_cm	minimum diameter [cm]
diameter_max_cm	maximum diameter [cm]
width_cm	apple width of all side views [cm]
width_mean_cm	mean apple width [cm]
height cm	apple height of all side views [cm]
height_mean_cm	mean apple height [cm]
symmetry	symmetry value: difference area with mirrored mask / 2 / mask area
symmetry_angle_deg	rotation angle of mask to achieve the minimal symmetry value
colhistheader	column names of color histograms
colhist	color histogram of hue and saturation of all views

R-Script for results visualization

The output files of the second Matlab script are then used as input for an R script [14] to summarize and visualize the data. The script creates summary factsheets (Supplementary data) per tree or cultivar depending on user input. For this purpose, a text file with the field plan of the experimental site including tree name/ID, cultivar name, row and tree position is required as an additional input file. The resulting factsheet contains information about fruit weight, width, height. Each top shape, an average side shape per fruit and an average side shape per single tree or cultivar (in case several trees of the same cultivar are investigated) is displayed. The average side shapes are determined using the polar coordinates; for this purpose, the average radius is calculated for each degree. Furthermore, the factsheet shows five photos, one from each side, of the first fruit per tree or cultivar, a color histogram of the mean values and additional information about the shape and data collection. Overall, the factsheets give an overview of the data and a first impression of the cultivar specific properties. Additionally, the R script exports polar coordinates, Cartesian coordinates, raw data of color histograms, correlation to predefined shapes and fruit weight and caliber in different text files. Thus, all the data collected is available in a compact and machine-friendly format for further analysis. The R script requires the packages dplyr [15] and tidyr [16] for data wrangling. The packages ggplot2 [17], gridExtra [18], gtable [19], qrcode [20] and magick [21] are required for image processing, data plotting and creating the fact sheet.

Performance assessment

To assess the performance of the FruitPhenoBox, approximatively 100 fruits from 15 apple cultivars each ('Ariane', 'Bonita', 'Braeburn', 'Milwa' (Diwa®), 'Gala Galaxy', 'Gala SchniCo Schniga', 'CH 101' (Galiwa®), 'Golden Reinders', 'Ladina', 'Mariella', 'SQ159' (Natyra®), 'Cripp's Pink' (Pink Lady®), 'PremA96' (Rockit®), 'Rustica' and 'Topaz') were photographed by a single user. Time stamps were used to estimate the average time elapsed between the snapshot of two different fruits, including the time required for replacing the fruit in the chamber. Average shapes per cultivar were generated, together with a boxplot representation of the distribution of weight, width, height, and shape index (height/weight). One-way ANOVAs were performed in R [14] to analyze differences of the four parameters between cultivars. For each parameter, letters indicating significance groups (p < 0.05) were assigned according to a post hoc Tukey test.

Optimal sample size determination

To determine the optimal sample size required to obtain confident results, we tested sample sizes (n) between five and 50 apples per cultivar. For each sample size, a random subset of n apples per cultivar was randomly selected from approximately 100 apples and the mean of their weight, width and height was calculated (200 times for each sample size). From the resulting mean sample values, the 2.5% and 97.5% quantiles and the means were calculated for each trait and cultivar. Description 95% confidence interval is the range between the lower and upper quantile divided by the mean value per cultivar and its half correspond to the relative margin of error.

Acknowledgments

The authors acknowledge the Molecular Plant Breeding and Crop Science groups at ETH Zurich for partial financial support, the Agroscope Fruit Genetic Resources group as well as the Product Quality and Innovation group for providing fruit samples. Furthermore, we thank Patrick Flütsch (Institute of Agricultural Sciences, ETH Zurich) and Martin Dosch (carpenter, ETH Zurich) for technical support in building the frame and the support, respectively. We thank the Institute of Mechanobiology (IBI 2) of the Forschungszentrum Jülich GmbH, Germany for permission to use the MATLIB Package.

Availability of Data and Materials

Supplementary files for this article, which include raw images in tif format, datasets extracted from the images and R scripts used in this study are available from the ETH Research collection under following doi: 10.3929/ethz-b-000590509. The Matlab scripts are available via gitlab (add final link)

Contributions

NK: hardware, software, data analysis, writing. MH: software, data analysis. BS: review & editing, AP: project administration, writing review & editing, GB: project administration, methodology, hardware, data analysis, writing. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Musacchi S, Serra S: Apple fruit quality: Overview on pre-harvest factors. Scientia Horticulturae 2018, 234:409-430.
Brown AG: The inheritance of shape, size and season of ripening in progenies of the cultivated apple. Euphytica 1960, 9(3):327-337.
Sansavini S, Donati F, Costa F, Tartarini S: Advances in apple breeding for enhanced fruit quality and resistance to biotic stresses: new varieties for the European market. Journal of fruit and ornamental plant research 2004, 12(Spec. ed. 2):13-52.
Laurens F: Review of the current apple breeding programmes in the world: objectives for scion cultivar improvement. In: Eucarpia Symposium on Fruit Breeding and Genetics 484: 1996. 163-170.
Jemrić T, Babojelić MS, Goran F, Šindrak Z: Fruit quality of nine old apple cultivars. Notulae Botanicae Horti Agrobotanici Cluj-Napoca 2013, 41(2):504-509.
Miranda C, Dapena E, Urbina V, Pereira-Lorenzo S, Errea P, Moreno M, Urrestarazu J, Fernandez M, Ramos-Cabrer AM, Diaz-Hernandez M: Development of a standardized methodology for phenotypical characterizations in apple. In: XIV EUCARPIA Symposium on Fruit Breeding and Genetics 1172: 2015. 367-370.
Szalatnay D, Bauermeister R: Obst-Deskriptoren NAP. In.; 2006.
Sofu MM, Er O, Kayacan MC, Cetişli B: Design of an automatic apple sorting system using machine vision. Computers and Electronics in Agriculture 2016, 127:395-405.
Awada MA, Wagenmakers PS, de Jager A: Effects of light on flavonoid and chlorogenic acid levels in the skin of 'Jonagold' apples. Scientia Horticulturae 2001, 88(4):289-298.
Magness J: Observations on color development in apples. In: Proc Amer Soc Hort Sci: 1928. 289-292.
Siegelman HW, Hendricks SB: Photocontrol of Anthocyanin Synthesis in Apple Skin. Plant Physiol 1958, 33(3):185-190.
Matlab. In., 9.5 (2018b) edn. Natick, Massachusetts: The MathWorks Inc.; 2018.
Otsu N: A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics 1979, 9(1):62-66.
Team RC: R: A language and environment for statistical computing. In. Edited by Computing RFfS. Vienna, Austria; 2021.
Wickham H, François R, Henry L, Müller K: dplyr: A Grammar of Data Manipulation. In., 1.0.6 edn; 2021.
Wickham H: tidyr: Tidy Messy Data. In., 1.1.3 edn; 2021.
Wickham H: ggplot2: Elegant Graphics for Data Analysis. In., 3.3.3 edn. New York: Springer-Verlag; 2016.
Auguie B: gridExtra: Miscellaneous Functions for "Grid" Graphics. In., 2.3 edn; 2019.
Wickham H, Pedersen TL: gtable: Arrange 'Grobs' in Tables. In., 0.3.0 edn; 2019.
Teh V, Onkelinx T: qrcode: Generate QRcodes with R. In., 0.1.1 edn; 2021.
Ooms J: magick: Advanced Graphics and Image-Processing in R. In., version 2.7.2 edn; 2021.

No competing interests reported.

Factsheet.pdf

Download PDF

Editorial decision: Major revision
16 Aug, 2023
Reviews received at journal
15 Aug, 2023
Reviews received at journal
25 Jul, 2023
Reviewers agreed at journal
22 Jun, 2023
Reviewers agreed at journal
07 Mar, 2023
Reviewers agreed at journal
07 Mar, 2023
Reviewers invited by journal
07 Mar, 2023
Editor assigned by journal
27 Feb, 2023
Submission checks completed at journal
27 Feb, 2023
First submitted to journal
27 Feb, 2023

You are reading this latest preprint version

FruitPhenoBox – a device for rapid and automated fruit phenotyping of small sample sizes

Status:

Version 1

Abstract

Figures

Background

Results

Performance assessment

Determination Of The Optimal Sample Size

Discussion

Conclusions

Methods

Declarations

Contributions

References

Additional Declarations

Supplementary Files

Status:

Version 1