Three-dimensional leaf edge reconstruction using a combination of two- and three-dimensional phenotyping approaches

doi:10.21203/rs.3.rs-3347414/v1

Download PDF

Research Article

Three-dimensional leaf edge reconstruction using a combination of two- and three-dimensional phenotyping approaches

https://doi.org/10.21203/rs.3.rs-3347414/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: The physiological functions of plants are carried out by leaves, which are important organs. The morphological traits of leaves serve multiple functional requirements and demands of plants. Traditional techniques for quantifying leaf morphology rely largely on two-dimensional (2D) methods, resulting in a limited understanding of the three-dimensional (3D) functionalities of leaves. Notably, recent advancements in surveying technologies have improved 3D data acquisition processes. However, there are still challenges in producing accurate 3D-representations of leaf morphologies, particularly leaf edges. Therefore, in this study, we propose a method for reconstructing 3D leaf edges using a combination of 2D image instance segmentation and curve-based 3D reconstruction.

Results: The proposed method reconstructed 3D leaf edges from multi-view images based on deep neural network-based instance segmentation for 2D edge detection, SfM for estimating camera positions and orientations, leaf correspondence identification for matching leaves among multi-view images, curve-based 3D reconstruction for estimating leaf edges as 3D curve fragments, and B-spline curve fitting for integrating curve fragments into a 3D leaf edge. The method was demonstrated on both virtual and actual plant leaves. On the virtually generated leaves, we evaluated the accuracy of the 3D reconstruction by calculating standardized Fréchet distance, which reveals that small leaves and high camera noise pose greater challenges to reconstruction. To balance the number and precision of 3D curve fragments, we proposed guidelines for setting the threshold for how only reliable curve fragments are reconstructed based on simulated data. These guidelines suggested that the threshold becomes lower with greater occlusions, larger leaf size, and camera positional error greater than a certain level. We also found the number of images does not affect the optimal threshold except in very few cases. Moreover, the proposed method succeeded in reconstructing holes in the leaf when the number of holes is three or less.

Conclusions: In this study, a nondestructive method for 3D leaf edge reconstruction was developed to address the 3D morphological properties of plants, which have been challenging to evaluate quantitatively. It is a promising way to capture whole plant architecture by combining 2D and 3D phenotyping approaches adapted to the target anatomical structures.

Phenomics

Leaf

Image analysis

Photogrammetry

Machine learning

Soybean

Leaves are highly important organs for plants since they are the sites of fundamental physiological processes, including photosynthesis, transpiration, and respiration. The phenotypic diversity of leaves underlies the various functional demands associated with their habitats [1–4]. Furthermore, their morphological properties are essential in balancing the multiple functional demands of individual plants and canopies [5–7], such as light interception [8, 9], heat transfer [10, 11], hydraulic conductivity [12, 13], mechanical constraints [14, 15], and growth efficiency [16, 17]. Quantifying the morphological traits of leaves provides a quantitative understanding of the relationships between the morphological traits and genetics of plants, morphogenesis, and environmental conditions, providing valuable insights into plant growth and development, improving crop yields, and enhancing plant productivity.

Leaves have complex three-dimensional (3D) shapes. Despite this, traditional measurement, quantification, and evaluation techniques rely on two-dimensional (2D) methods because they are simple and more feasible to use, especially considering existing technical limitations. In many cases, botanical specimens are preserved two-dimensionally [18] and undergo morphological changes upon drying [19, 20]. Quantitative evaluations are based on 2D imaging (e.g., flatbed scanners) and image analysis (e.g., [21, 22]). Leaves exhibit a wide range of patterns in 3D shapes [23], and their functionality is highly dependent on their configuration in the 3D space [24–27]. Moreover, some shapes cannot be adequately projected two-dimensionally (e.g., twisted leaves of Codiaeum variegatum ‘Spirale’). Consequently, many leaf characteristics have not been appropriately evaluated through 2D methods, inspiring interest in 3D evaluations.

High-resolution 3D morphological data can be acquired efficiently and cost-effectively using light detection and ranging (LiDAR) sensors, depth cameras, and photogrammetry techniques [28–30]. A pipeline utilizing structure from motion (SfM) and multi-view stereo (MVS), which reconstructs a 3D surface as point cloud data from a series of 2D images captured from different angles, has been implemented in several libraries and software products (e.g., [31, 32]). Several devices and techniques for acquiring the structures of plants in 3D have been developed to facilitate 3D evaluation in plant phenotyping studies [33, 34]. However, point cloud data produced by point-based 3D reconstruction methods, such as the commonly used SfM/MVS pipeline, may not be ideal for representing 3D leaf morphologies because of unclear leaf edges [35] and uncertainties regarding whether the holes in point cloud data are actually real or results of reconstruction errors [36]. Point-cloud data reconstructed using the point-based reconstruction method often include points representing both leaves and artifacts owing to the keypoints detected on the background (Fig. 1a). The holes in the output point-cloud data comprise reconstruction deficiencies and actual holes; it is difficult to distinguish between them solely based on point-cloud data (Fig. 1b, c). It is preferable to establish phenotyping methods that enable the direct estimation of leaf edges.

In this study, we proposed a method to reconstruct leaf edges from multi-view images using deep learning-based instance segmentation for 2D edge detection (Fig. 2a, b), SfM for estimating camera positions and orientations (Fig. 2c), leaf correspondence identification for matching leaves among multi-view images (Fig. 2d), curve-based 3D reconstruction for estimating leaf edges as curve fragments in 3D spaces (Fig. 2e), and B-spline curve fitting for integrating curved fragments into 3D leaf outlines (Fig. 2f). The applicability and limitations of the proposed method were examined using both simulated data and actual multi-view images of soybean plants. Our analysis revealed that leaf size, error in camera parameter estimation, and mask estimation error had significantly impacted accuracy. The proposed method is expected to be a valuable tool for clarifying the morphological characteristics of 3D leaf edges, which are difficult to quantitatively evaluate.

3D leaf edge reconstruction on virtually generated leaf models

The key idea of the proposed method is directly estimating 3D leaf edges using curve-based 3D reconstruction. In this study, we used a 3D curve sketch [37] that reconstructed a set of 3D curve fragments from the 2D edges of a target object in multi-view images (Fig. 3 and Methods). Based on the principle of epipolar geometry, candidates for corresponding edge pairs were identified along the epilines (Fig. 3a). All candidate edge pairs, which were called pair hypotheses, were reconstructed in 3D space; and these 3D curve fragments were reprojected onto other images to evaluate how closely the reconstructed curve generated the true projection (Fig. 3b). If the reprojected curve was sufficiently close to the edges in the reprojected image, it indicated that the pair hypothesis had been supported by the reprojected image. Only 3D curve fragments supported by a number of images greater than the support threshold (τ_t) were reconstructed (Fig. 3c). The proposed method was first demonstrated on virtual data generated from single and multiple leaves created using Blender [38] (see Methods for the details).

On single virtual leaves, true mask images and camera parameters are known. Based on this assumption, 3D leaf edges were reconstructed by extracting the 2D leaf edges from true mask images and adopting a 3D curve sketch (Fig. 4, upper row); the reconstructed leaf edges appeared along the edges. Notably, the support threshold τ_t strongly affected the performance of curve-based reconstruction; low τ_t values resulted in highly inaccurate 3D curve fragments, and high values resulted in 3D curve fragments that did not completely cover the leaf edges (Additional file 1: Fig. S1). Details regarding τ_t adjustment are discussed later (see the subsection "Guidelines for setting support thresholds in 3D edge reconstruction with occlusions").

Regarding the 3D edges of multiple virtual leaves of a single plant, they were reconstructed after identifying the correspondences between individual leaves across the mask images, resembling reconstruction in the single-leaf case in all aspects except for considering the influence of occlusion (Additional file 1: Fig. S2a). However, the correspondence of leaves among mask images is nontrivial in actual multi-view images because the mask image is estimated for each individual image. Thus, we precisely estimated the 3D leaf edges of multiple leaves in a single scene by incorporating a leaf correspondence identification step that prevented the generation of pair hypotheses between non-corresponding leaves across views (Fig. 4, lower row, Methods, and Additional file 2). In the absence of leaf correspondence identification, the number of reconstructed curve fragments decreased, and the vertical reconstruction error increased (Additional file 1: Fig. S2b). In the leaf correspondence identification step, the point cloud data obtained from SfM (Fig. 2c) was clustered into each leaf, and each cluster corresponding to a single leaf was reprojected onto the mask images (Fig. 5a). To preclude the leakage of points in backside into the front during reprojection, hidden point removal [39] was applied from each view (Fig. 5b). Leaf correspondences were identified by counting the number of reprojected points belonging to each cluster in each image (Fig. 5c).

Comparing the accuracy of 3D leaf edge reconstruction under different conditions

We evaluated the accuracy of the 3D leaf edge reconstruction method for different leaf areas, image numbers, occlusion levels, and noise levels. This evaluation was performed on the simulated multiple leaf data using the standardized Fréchet distance [40] (SFD) divided by the square root of the leaf area. The SFD was calculated for three individual plants with 8 different- sized leaves ($312{ \text{m}\text{m}}^{2}\le A\le \text{3,366} {\text{m}\text{m}}^{2}$) in several simulation scenarios, including different levels of occlusion (no, thin, and thick pillars), different numbers of multi-view images (32, 64, and 128 images), and different degrees of positional noise affecting the camera parameters ($\sigma =$ 0, 1, and 3 mm) (Fig. 6).

The SFD decreased with the increase in leaf area; the small leaves were more challenging to reconstruct than the larger leaves were. Small leaves had larger curvatures even if they had the same shapes, making it difficult for the 3D curve sketch to reconstruct the correct curve fragments because 2D curve fragments had been frequently generated through splitting by a tangential epipolar line (see Methods for details). The SFD increased with increases in the degree of noise at the camera positions. Although a less accurate camera extrinsic parameter estimation would increase the SFD, the effect might be limited under low noise. However, the SFD was less sensitive to the number of images and level of occlusion, considering that even if a leaf edge was obscured in an image, it could be complemented if it had appeared in other images [41].

Generation of mask images from actual multi-view images using Mask R-CNN

To generate a mask image for each leaf from multi-view images of actual plants, we used Mask R-CNN [42], which is a deep neural network (DNN) model used for instance segmentation. The performance of the model was evaluated on 1039 images of four individuals with group four-fold cross-validation; the images belonging to one individual were excluded as test images, and the other images were randomly divided into training (80%) and validation (20%) datasets (Fig. 7a and Methods). The model weights were adopted at epoch 8000, and the validation loss did not improve thereafter on the learning curve until epoch 10000 (Additional file 1: Fig. S3). Individual leaf masks were generated using the trained model (Fig. 7b). The average precision (AP) values of the test data are listed in Table 1. Regarding the values, AP was 49.8, and AP large (APl) was 76.9, indicating that inference had been successful in a large region, resembling the trend in a previous study on generic object recognition [42]. On the other hand, AP middle (APm) and AP small (APs) were smaller than APl, suggesting that generating masks for small leaves had been challenging. The model weights trained on all images of the four individuals were used for the subsequent analysis.

Guidelines for setting support thresholds in 3D edge reconstruction

To obtain accurate 3D leaf edges, the support threshold (τ_t) should be set appropriately to balance the trade-off between the number and precision of the reconstructed 3D curve fragments. We attempted to propose optimal support thresholds against occlusion indices (OIs) based on simulated virtual leaves by evaluating the precision-recall (PR) curve of the reconstructed 3D edges (i.e., the optimal support threshold was defined as the highest support threshold that resulted in the highest recall when the precision had exceeded 0.99) (Fig. 8 and Methods). In this study, the occlusion index of a target leaf was defined based on the sparse point cloud data of the target, as follows:

$$\text{O}\text{I}= \frac{1}{m}\sum _{i=1}^{m}\left(1-\frac{{n}_{i}}{n}\right)$$

where m is the number of images; n is the number of points of a target instance; and ${n}_{i}$ is the number of points of a target instance reprojected onto the i-th image; and OI is the occlusion index, where OI = 0 indicates no occlusion, and OI = 1 indicates complete occlusion.

Large support threshold values were observed for less occluded leaves (OI < 0.75), which had appeared in many images, because they had achieved both high precision and recall values by filtering inaccurate 3D curve fragments. Highly occluded leaves (OI > 0.75) tended to have lower optimal support thresholds at increased OI values, with the optimal values exhibiting large variations (Fig. 8a), which were attributed to differences in leaf areas, with larger leaves showing steeper trends. Furthermore, the optimal support thresholds decreased when the degree of camera positional error increased (i.e., low positional accuracies prevented precise filtering) (Fig. 8b) and increased slightly when there were more cameras (Fig. 8c). These trends were clearly observed in leaves with low to intermediate levels of occlusions (0.75–0.80) (Additional file 1: Fig. S4).

Herein, we propose a guideline for determining the support threshold based on simulated data. We conducted Bayesian ridge regression on the optimal support thresholds against the OI based on the simulated data of 128 images with no camera positional noise (Fig. 8d). Moreover, we have provided the following qualitative guidelines: the slope of the linear regression models should be made a downward revision for large leaves (i.e., the trend becomes steeper when the leaf area becomes larger) (Fig. 8a); the camera positional error should be suppressed under a certain value (Fig. 8b); and the number of images should not be increased unnecessarily because improvements in estimation precision reduce when there are more images (Fig. 8c).

Application of proposed method to actual soybean data

We demonstrated the performance of the proposed method by applying it to individual soybeans (Fukuyutaka) at three growth stages (Fig. 9; Additional files 3–5). The 3D leaf edges were reconstructed using the support threshold proposed in the guidelines (Fig. 8d). At the high support threshold, inaccurate reconstructions were suppressed, but some leaves disappeared (Fig. 9, mean). In contrast, at the low support threshold, there was inaccurate reconstruction and the occurrence of artifacts, but almost all the leaves had been reconstructed (Fig. 9, mean – 0.5 SD). It was more challenging to reconstruct all the leaves at a later growth stage because the number of occlusions caused by the number of leaves had increased. Several failed cases were observed: (1) single leaves were reconstructed as multiple leaves because point cloud segmentation had failed in the leaf correspondence step (Fig. 10a); (2) leaves were not reconstructed because the small leaves had disappeared at the mask generation step (Fig. 10b); and (3) reconstructed leaf edges differed markedly from their original shapes owing to B-spline fitting in the cases where they had not been covered by 3D curve fragments (Fig. 10c).

Applicability of the proposed method to holes on a leaf

Using the proposed method, we attempted to reconstruct holes in leaves, which was challenging to do using point-based 3D reconstruction. To demonstrate this, we applied the proposed method to simulate leaves with one to six holes. The edges and holes were reconstructed well when the number of holes was three or fewer (Fig. 11a-c), and the precision of the reconstruction decreased with an increasing number of holes (Fig. 11d-f). Particularly, for leaves with five and six holes, the holes had been reconstructed perpendicular to them (Fig. 11e, f), because the precision of the 3D curve fragment reconstruction had decreased, and the holes had not been separated in a clustering step, in which DBSCAN is used to separate the leaf edge and holes.

The proposed phenotyping approach, which includes the instance segmentation of 2D images and curve-based 3D reconstruction that integrates the information into a 3D space, successfully reconstructed 3D leaf edges from multi-view images of both virtual and actual plants (Figs. 4 and 9). Owing to the inclusion of the leaf correspondence identification step, our approach is applicable not only to a single leaf but also to multiple leaves in the same scene (Fig. 5). The direct 3D reconstruction of leaf edges does not require the removal of artifacts from the background (Fig. 2) and allows the robust estimation of leaf edges in a 1D closed curve in 3D space. The simulation results showed that as long as the camera positional errors were not too large (~ 1 mm), the precision in estimating the leaf edges could be maintained (Fig. 6), even when the number of cameras had been reduced or the degree of occlusion had been changed. Moreover, the proposed method paves the way for solving the problem of point-based 3D reconstruction methods such as SfM/MVS in struggling to distinguish real holes from artifacts (e.g., [36]). The proposed method correctly performs 3D reconstruction only for the holes in leaves recognized in 2D images instead of incorrectly recognizing these holes as the “negative” of the point cloud data. In our simulation, the holes were reconstructed well when the number of holes was less than four. Although the estimation was poor when the number of holes was greater than four, the results could be improved by recognizing each hole as an individual instance in the instance segmentation step, similar to the approach in leaf correspondence identification (Additional file 1: Fig. S2).

To improve the accuracy of 3D edge reconstruction, the following points should be considered: (1) tuning the hyperparameters, (2) improving the camera parameter estimation, and (3) improving the instance segmentation model. These points are elaborated as follows. (1) The hyperparameters used in the proposed method were tuned. We provided the guidelines for setting the support thresholds (${\tau }_{t}$) against the target leaf area (A), OI, the degree of positional error ($\sigma$), and the number of images (Fig. 8); however, other parameters also played crucial roles in 3D edge reconstruction. For example, the fragment length (l_fragment) potentially played a primary role in improving the accuracy of small leaves. In this study, we used a fragmentation length appropriate for larger structures (${40<l}_{\text{f}\text{r}\text{a}\text{g}\text{m}\text{e}\text{n}\text{t}}<100$ and ${80<l}_{\text{f}\text{r}\text{a}\text{g}\text{m}\text{e}\text{n}\text{t}}<200$ for the simulation and actual data, respectively) in the 2D edge extraction, which reduced the number of 2D curve fragments for smaller structures. In postprocessing using B-spline curve fitting, the number of knots should be tuned to capture high-curvature leaf edges (e.g., [43]). (2) Accurate camera parameter estimation improved the accuracy of 3D reconstruction. We robustly estimated camera parameters in SfM using coded and non-coded markers. Fabbri and Kimia [37] proposed a camera parameter calibration approach that minimized the curve-based reprojection error. (3) Improving the AP values of the instance segmentation model improved the performance of 3D reconstruction (Fig. 10b). The mask R-CNN model trained on our dataset showed that APs that had been considered to account for most of the mask generation accuracy of small leaves were smaller than APl and APm and were unsuitable for reconstructing immature leaves (Table 1). Therefore, it is desirable to expand the dataset, especially for small leaves. The use of pre-training models with large datasets, such as the segment thing model (SAM) [44], is also promising. Alternatively, a model capable of directly recognizing anatomical structures of interest in plants may be useful (e.g., [45], and [46]).

In this study, we proposed guidelines for setting the support threshold when applying the proposed method to actual plants. These guidelines mainly depend on the level of occlusion and noise and the number of images (Fig. 8). We investigated the advantages of the curve-based approach, learning that a limited number of images were sufficient for estimating 3D leaf edges. Reconstruction was successfully performed following these guidelines and verified using actual individual soybean data (Fig. 9). The guidelines helped us determine the configuration of the experimental designs and data acquisition scenarios, including the hyperparameters.

The proposed method is an essential technique for assessing the 3D morphological properties of leaves, which are challenging to quantitatively evaluate. These properties play a central role in balancing the multiple functional demands of individual plants and canopies [5–7], with traditional evaluations mostly being performed using 2D approaches (e.g., [21, 22]). The proposed method obtained 3D leaf edges, including their 3D positions, orientations, and sizes, relative to the configurations of organs in individual plants in a non-destructive manner (Fig. 9). It is a promising method to capture whole-plant architecture combined with a method for estimating branches [47, 48], other plant organs [49, 50], and leaf anatomical structures including leaf veins [51, 52], textures, and holes. Furthermore, functional structural plant models (FSPMs), which couple the 3D morphologies of plants with their physiological dynamics, can be improved and validated using such morphological data. For example, the optimal morphologies and movements (e.g., optimal canopy structure [53], diurnal leaf movement [54], and leaf phototropism [55]) predicted using FSPMs in previous studies were tested to determine how they fit the experimental data and vice versa. Our proposed method contributes to filling this gap by successfully integrating hierarchical morphological properties into 3D spaces.

We have proposed a method for the 3D reconstruction of leaf edges using DNN-based instance segmentation and a curve-based approach from multi-view images to address existing problems in point-based methods. The proposed method reconstructed the 3D leaf edges for both simulated and actual data even in the presence of multiple leaves in the same scene. The accuracy of leaf edge reconstruction was evaluated using simulated virtual plants under different conditions as the ground truth. We inferred that smaller leaves were more challenging to reconstruct than larger ones and that the camera positional error and image numbers did not affect the results that much since they had satisfied certain criteria. Based on the simulation, we have proposed qualitative and quantitative guidelines for setting several parameters used in the proposed method. The proposed method is applicable not only to leaf edges, but also to holes in leaves. We believe that by integrating hierarchical morphological properties, which are difficult to quantitatively evaluate, the proposed method will be a valuable tool for advancing our understanding of plant physiology, functional ecology, and cultivation management.

Structure from Motion

The SfM technique was utilized to obtain the projection matrix for each camera and the sparse point cloud from a multi-view image. Regarding SfM, it is a photogrammetric method for simultaneously estimating the camera parameters and the depth of corresponding points (i.e., sparse 3D point clouds) from multi-view images. In this study, we used Metashape (Agisoft, St. Petersburg, Russia), which is commercial photogrammetry software that includes SfM. The projection matrices, including the optical center, focal length, orientation, and position of the cameras, were exported as Extensible Markup Language (XML) files. Markers were placed on the image to optimize image placement and thereby make it easier to obtain the corresponding points.

Instance segmentation of leaves using Mask R-CNN

To extract the 2D edges of leaves, mask images of multi-view images were obtained using Mask R-CNN, a deep neural network model for segmentation [42]. We used Detectoron2 [56], a library for detection and segmentation tasks, to utilize the Mask R-CNN model with the backbone ImageNet and the model weights pretrained on the COCO dataset. The model was trained on a training dataset that comprised 80% of the dataset consisting of multi-view images of three individual plants, and the remaining 20% of images were used for validation (validation dataset). The model performance was evaluated using a test dataset consisting of another individual plant. To calculate the accuracy of instance segmentation using the Mask R-CNN, we performed four-fold cross-validation, which assigned a different individual as the test dataset.

Leaf edges were extracted from the predicted mask images using OpenCV library [31]. The extracted edges were divided according to the condition [37] under which the contours had random lengths. The 2D edges were divided into fragments with an overlap ≥ τ_overlap and with a certain length l_fragment. In this study, we used ${\tau }_{\text{o}\text{v}\text{e}\text{r}\text{l}\text{a}\text{p}}=15$ and ${40<l}_{\text{f}\text{r}\text{a}\text{g}\text{m}\text{e}\text{n}\text{t}}<100$ for the simulation data and ${\tau }_{\text{o}\text{v}\text{e}\text{r}\text{l}\text{a}\text{p}}=30$ and ${80<l}_{\text{f}\text{r}\text{a}\text{g}\text{m}\text{e}\text{n}\text{t}}<200$ for the real data.

Leaf correspondence identification across multi-view images

To individually process and reconstruct the leaves, we determined the correspondence of the leaves between the images. First, the point cloud was clustered, and the points in the backside was removed and projected onto the mask image. Then, the point cloud was associated with the mask on which most of the points had been located. If this was performed for all the mask images, the correspondence between the leaves of the images could be obtained via a point cloud. In this study, density-based spatial clustering of applications with noise (DBSCAN) [57], a density-dependent clustering method, was used for clustering on simulated data; and region-growing segmentation implemented in the Point Cloud Library [58], where the angle between normals determines whether a region is grown or not was used on real data. Hidden points removal [39], where the angle between the normals determines whether a region is grown, was used on the real data. Hidden point removal [39], which determines the visible points in a point cloud from a given viewpoint using a sphere and a spherical inversion operator, was used for background removal. Figure 6a shows an overview of the method, and Fig. 6b shows the practical application of the method.

Curve-based 3D reconstruction of leaf edges

A 3D curve sketch [37], which is a curve-based 3D reconstruction, was obtained using the projection matrix obtained by SfM, the multi-view mask image obtained by Mask R-CNN, and the correspondence of leaves among the images. All the subsequent processes were applied to each leaf. Obtaining a 3D curve sketch involves the following steps: (1) camera pair definition; (2) pair hypothesis generation; and (3) 3D curve fragment reconstruction and filtering by reprojection.

(1) Camera pair definition

To perform curve-based 3D reconstruction, camera pairs were defined based on the relative positions of the cameras in the scene. Angle b_ij, which is the angle between cameras i and j from the average positions of all the cameras (), was calculated for all the camera combinations. The camera pairs were defined as the combinations that satisfied. Since the cameras had been assumed to be equally spaced to cover the plants, b corresponded to the baseline in [37]. In this study, we used angles of 30, 40, and 60° on the simulated data of 128, 64, and 32 multi-view images, respectively. For the real data, b_max was set to 30°, regardless of the number of images.

(2) Pair hypothesis generation

Let be the p-th 2D curve fragment in the i-th image. Pair hypothesis, a potentially corresponding pair of 2D curve fragments, is defined as a pair of 2D curve fragments. In epipolar geometry, a fundamental matrix F_ij computed from the projection matrices corresponding to images ( and ) maps a point in the i-th image to a line in the j-th image. The line mapped by the fundamental matrix is called the epipolar line (or epiline), and any existing corresponding points along the line are found. By extending this concept to a 2D curve fragment, F_ij maps a 2D curve fragment in the i-th image to a band (a set of epipolar lines) in the j-th image. Pair hypotheses were generated based on the 2D curve fragments overlapping the bands (Fig. 3a). For a robust reconstruction, 2D curved fragments tangential to the epipolar line were excluded from the process (see [37] for details). The number of pairs of hypotheses per band was set to a maximum of only 10 to account for the limited the computational resources.

(3) 3D curve fragment reconstruction and filtered by reprojection

Then, 3D curve fragments (), which correspond to the pair hypotheses, were reconstructed using projection matrixes in 3D Euclidean space. Each reconstructed 3D curve fragment was reprojected onto multi-view images, excluding the i- and j-th images, to evaluate how closely the reconstructed curve fragments generated the true projection. The reconstructed curve fragments were supported by reprojections if the reprojected curve fragments had been located close to the edges of the target object (i.e., leaf) on the image; i.e., a reprojected curve fragment was supported if at least τ_v (%) of the curve fragment was located within τ_d pixels of the edges in τ_t images. Only curves supported with a sufficient number of images (i.e., greater than the support threshold τ_t) were reconstructed. A τ_v of 80% was used for all cases, and τ_d was 11 and 39 pixels for the simulated leaves and actual soybean specimens, respectively.

B-spline closed curve fitting

The 3D curve fragments were integrated into a closed 3D curve by using B-spline fitting. In B-spline fitting, a continuous periodic function is approximated by a piecewise polynomial function, which is a linear combination of the order j B-spline basis over i-th interval ${b}_{i,j}\left(l\right)$ as follows:

$$f\left(l\right)=\mathbf{w} \mathbf{b}\left(l\right)=\left(\begin{array}{ccc}{w}_{1}& \cdots & {w}_{n-1}\end{array}\right)\left(\begin{array}{c}{b}_{1,j}\left(l\right)\\ ⋮\\ {b}_{n-k-1,j}\left(l\right)\\ {b}_{n-k,j}\left(l\right)+{b}_{1-k,j}\left(l\right)\\ ⋮\\ {b}_{n-1,j}\left(l\right)+{b}_{0,j}\left(l\right)\end{array}\right)$$

where w_i denotes the coefficient of the i-th B-spline. Based on the coordinate values of the reconstructed 3D curve fragments, the B-spline coefficients were estimated for x-, y-, and z-coordinate values using the "curve_fit" function in SciPy [59].

Calculation of optimal support thresholds

For τ_t, the PR curves in the ground-truth mesh and the reconstructed curve fragment were calculated in the simulation data, which the optimal support threshold is the highest τ_t, with the highest recall when the precision had exceeded 0.99. The precision is the percentage of ground truths for which the reconstructed curved fragments are within 30 mm, and the recall is the percentage of curved fragments for which the ground truths are within 30 mm. We also excluded points supported by more than τ_p on a well-supported curve based on 3D drawings [60], which are modified versions of 3D curve sketches.

The simulation data were comprehensively tested for different precision and recall values with respect to the support threshold, which is defined as the number of images whose ratio to the total number of images (128, 64, and 32 images) falls into 24 equal parts (i.e., between 1 and 0.125). The precision and recall values were recorded; if the precision did not reach one, the minimum value was used as the optimal support threshold.

Virtual plant models

Virtual plant models (single, multiple, and single leaves with holes) were created using Blender (Blender Foundation, Amsterdam, Netherlands). Multi-view images of the virtual plants were generated using Unity (Unity Technologies, San Francisco, CA, US) from 128, 64, and 32 cylindrically arranged views. Three individuals were created based on the multiple-leaf model; each leaf was translated randomly—horizontally from − 33.33 to 33.33% and vertically from − 14.28–14.29% of the bounding box dimensions—and rotated randomly from − 10 to 10°.

Multi-view images of soybean plants

Multi-view images were obtained from four individual soybeans (Glycine max), including four cultivars (Enrei, Zairai 51 − 2, Aoakimame, and Saga zairai), to train the Mask R-CNN model and evaluate its performance. These individuals were captured at different growth stages: 1) Enrei: 34 days after sowing (DAS); 2) Zairai 51 − 2: 56 DAS, 3) Aoakimame: 24 DAS; and 4) Saga zairai: 48 DAS. To demonstrate the applicability of the proposed method, multi-view images of another cultivar, Fukuyutaka, at different growth stages of 21, 28, and 42 DAS, were obtained. Each set of multi-view images included 264 images, and approximately 130 images were subsampled. We used a simple fixed photogrammetry system consisting of digital cameras (EOS Kiss X7; Canon, Tokyo, Japan), a turntable (MT320RL40; ComXim, Shenzhen, China), and a camera control application (CaptureGRID4; Kuvacode, Kerava, Finland) (Additional file 1, Fig. S5) to obtain multi-view images.

3D three-dimensional

2D two-dimensional

SfM Structure from Motion

MVS Multi-View Stereo

LiDAR Light Detection And Ranging

SFD standardized Fréchet distance

AP average precision

APl AP large

APm AP middle

APs AP small

OI occlusion index

PR Precision-Recall

SD standard deviation

SAM Segment Anything Model

DAS days after sowing

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed during the current study are available in the Zenodo repository, [DOI] and GitHub repository [URL].

Competing interests

The authors declare that they have no competing interests.

Funding

This study was supported by Japan Society for the Promotion of Science (JSPS) KAKENHI Grant Numbers 20H01381, 21K14947, 22H04727, (to K.N.), Japan Science and Technology Agency (JST) PRESTO Grant Number JPMJPR16O5 (to K.N.), JST MIRAI Grant Number JPMJMI20G6 (to K.N.); Moonshot R&D Grant Number JPMJMS2021 (to K.N.), and Bio-oriented technology Research Advancement InstitusioN (BRAIN) Moonshot R&D Grant Number JPJ009237 (to K.N.).

Authors' contributions

KN conceived and designed this study. HM and KN performed the implementation and analyzed the results. HM and KN were major contributors to writing the manuscript. All authors read and approved the final manuscript.

Acknowledgements

We thank Horiguchi, R., Inbe, N., and Suzuki, M. for their assistance in making multi-view image datasets.

Wright IJ, Reich PB, Westoby M, Ackerly DD, Baruch Z, Bongers F et al. The worldwide leaf economics spectrum. Nature [Internet]. 2004;428:821–7. Available from: https://idp.nature.com/authorize/casa?redirect_uri= https://www.nature.com/articles/nature02403&casa_token =CtYdn6u63BcAAAAA:cNAkoHi669d547DoKaM3keUixX15dhKLnmyUD_oX2GBEB1VTe9mtWcMdFoXauJPFpb4iQItw5D0oAvb3.
Midolo G, De Frenne P, Hölzel N, Wellstein C. Global patterns of intraspecific leaf trait responses to elevation. Glob Chang Biol. 2019;25:2485–98.
Luo X, Keenan TF, Chen JM, Croft H, Colin Prentice I, Smith NG et al. Global variation in the fraction of leaf nitrogen allocated to photosynthesis. Nat Commun [Internet]. 2021;12:1–10. http://dx.doi.org/10.1038/s41467-021-25163-9.
Onoda Y, Wright IJ, Evans JR, Hikosaka K, Kitajima K, Niinemets Ü, et al. Physiological and structural tradeoffs underlying the leaf economics spectrum. New Phytol. 2017;214:1447–63.
Smith VC, Ennos AR. The effects of air flow and stem flexure on the mechanical and hydraulic properties of the stems of sunflowers Helianthus annuus l. J Exp Bot. 2003;54:845–9.
Díaz S, Kattge J, Cornelissen JHC, Wright IJ, Lavorel S, Dray S et al. The global spectrum of plant form and function. Nature [Internet]. 2016;529:167–71. http://dx.doi.org/10.1038/nature16489.
GIVNISH TJ. Comparative Studies of Leaf Form: Assessing the Relative Roles of Selective Pressures and Phylogenetic Constraints. New Phytologist [Internet]. 1987;106:131–60. Available from: https://onlinelibrary.wiley.com/doi/10.1111/j.1469-8137.1987.tb04687.x.
Onoda Y, Saluñga JB, Akutsu K, Aiba S ichiro, Yahara T, Anten NPR. Trade-off between light interception efficiency and light use efficiency: Implications for species coexistence in one-sided light competition. Journal of Ecology. 2014;102:167–75.
Niinemets Ü. A review of light interception in plant stands from leaf to canopy in different plant functional types and in species with varying shade tolerance. Ecol Res. 2010;25:693–714.
Roth-Nebelsick A. Computer-based analysis of steady-state and transient heat transfer of small-size leaves by free mixed convection. Plant Cell Environ. 2001;24:631–40.
Vogel S. Convective cooling at low airspeeds and the shapes of broad leaves. J Exp Bot. 1970;21:91–101.
Ding J, Johnson EA, Martin YE. Optimization of leaf morphology in relation to leaf water status: A theory. Ecol Evol. 2020;10:1510–25.
Sack L, Holbrook NM. Leaf hydraulics. Annu Rev Plant Biol. 2006;57:361–81.
Vogel S. Drag and reconfiguration of broad leaves in high winds. J Exp Bot. 1989;40:941–8.
Niklas KJ. A mechanical perspective on foliage leaf form and function. New Phytol. 1999;143:19–31.
Poorter H, Niinemets Ü, Ntagkas N, Siebenkäs A, Mäenpää M, Matsubara S, et al. A meta-analysis of plant responses to light intensity for 70 traits ranging from molecules to whole plant performance. New Phytol. 2019;223:1073–105.
Anjum SA, Ashraf U, Zohaib A, Tanveer M, Naeem M, Ali I et al. Growth and developmental responses of crop plants under drought stress: a review. Zemdirbyste-Agriculture [Internet]. 2017;104:267–76. Available from: http://www.zemdirbyste-agriculture.lt/1043_str34/.
Agarwal G, Belhumeur P, Feiner S, Jacobs D, Kress WJ, Ramamoorthi R, et al. First steps toward an electronic field guide for plants. Taxon. 2006;55:597–610.
Tomaszewski D, Górzkowska A. Is shape of a fresh and dried leaf the same? PLoS ONE. 2016;11:1–14.
Babu AK, Kumaresan G, Raj VAA, Velraj R. Review of leaf drying: Mechanism and influencing parameters, drying methods, nutrient preservation, and mathematical models. Renewable and Sustainable Energy Reviews [Internet]. 2018;90:536–56. https://doi.org/10.1016/j.rser.2018.04.002.
Černý J, Pokorný R, Haninec P, Bednář P. Leaf area index estimation using three distinct methods in pure deciduous stands. J Visualized Experiments. 2019;2019:1–14.
Webb N. WinDIAS User Manual 3.3. Cambridge. UK: Delta-T Devices Ltd.; 2019. pp. 3–6.
Huang C, Wang Z, Quinn D, Suresh S, Jimmy Hsia K. Differential growth and shape formation in plant organs. Proc Natl Acad Sci U S A [Internet]. 2018;115:12359–64. Available from: http://www.pnas.org/lookup/doi/10.1073/pnas.1811296115.
Shultis JK, Myneni RB. Radiative transfer in vegetation canopies with anisotropic scattering. J Quant Spectrosc Radiat Transf. 1988;39:115–29.
Li X, Strahler AH, Friedl MA. A conceptual model for effective directional emissivity from nonisothermal surfaces. IEEE Trans Geosci Remote Sens. 1999;37:2508–17.
Widlowski JL, Mio C, Disney M, Adams J, Andredakis I, Atzberger C, et al. The fourth phase of the radiative transfer model intercomparison (RAMI) exercise: Actual canopy scenarios and conformity testing. Remote Sens Environ. 2015;169:418–37.
Pinty B, Gobron N, Widlowski JL, Gerstl SAW, Verstraete MM, Antunes M, et al. Radiation transfer model intercomparison (RAMI) exercise. J Geophys Res Atmos. 2001;106:11937–56.
Panjvani K, Dinh AV, Wahid KA. LiDARPheno – A low-cost LiDAR-based 3D scanning system for leaf morphological trait extraction. Front Plant Sci. 2019;10.
Paulus S, Behmann J, Mahlein AK, Plümer L, Kuhlmann H. Low-cost 3D systems: Suitable tools for plant phenotyping. Sens (Switzerland). 2014;14:3001–18.
Lu X, Ono E, Lu S, Zhang Y, Teng P, Aono M et al. Reconstruction method and optimum range of camera-shooting angle for 3D plant modeling using a multi-camera photography system. Plant Methods. 2020;16.
Bradski G. The OpenCV Library. Dr Dobb’s Journal of Software Tools. 2000.
Cernea D. OpenMVS: open Multi-View Stereo reconstruction library [Internet]. GitHub repository. 2020. Available from: https://cdcseacave.github.io/openMVS.
Rossi R, Costafreda-Aumedes S, Leolini L, Leolini C, Bindi M, Moriondo M. Implementation of an algorithm for automated phenotyping through plant 3D-modeling: A practical application on the early detection of water stress. Comput Electron Agric. 2022;197.
Li Y, Wen W, Miao T, Wu S, Yu Z, Wang X et al. Automatic organ-level point cloud segmentation of maize shoots by integrating high-throughput data acquisition and deep learning. Comput Electron Agric. 2022;193.
Wu S, Wen W, Wang Y, Fan J, Wang C, Gou W et al. MVS-Pheno: A portable and low-cost phenotyping platform for maize shoots using multiview stereo 3D reconstruction. Plant Phenomics. 2020;2020.
Boukhana M, Ravaglia J, Hétroy-Wheeler F, De Solan B. Geometric models for plant leaf area estimation from 3D point clouds: A comparative study. Graphics and Visual Computing. 2022;7:200057.
Fabbri R, Kimia B. 3D curve sketch: Flexible curve-based stereo reconstruction and calibration. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2010;1538–45.
Community BO. Blender - a 3D modelling and rendering package [Internet]. Stichting Blender Foundation, Amsterdam; 2018. Available from: http://www.blender.org.
Katz S, Tal A, Basri R. Direct visibility of point sets. ACM Trans Graph. 2007;26:24.
ALT H, GODAU M. Computing the Fréchet Distance Between Two Polygonal Curves. Int J Comput Geom Appl [Internet]. 1995;05:75–91. Available from: https://www.worldscientific.com/doi/abs/10.1142/S0218195995000064.
Li S, Yao Y, Fang T, Quan L. Reconstructing Thin Structures of Manifold Surfaces by Integrating Spatial Curves. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2018;2887–96.
He K, Gkioxari G, Dollar P, Girshick R, Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV) [Internet]. IEEE; 2017. p. 2980–8. Available from: http://ieeexplore.ieee.org/document/8237584/.
Mörwald T, Balzer J, Vincze M. Modeling connected regions in arbitrary planar point clouds by robust B-spline approximation. Rob Auton Syst [Internet]. 2016;76:141–51. http://dx.doi.org/10.1016/j.robot.2015.11.006.
Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L et al. Segment Anything. ArXiv [Internet]. 2023; Available from: http://arxiv.org/abs/2304.02643.
Yu Z, Feng C, Liu M-Y, Ramalingam S. CASENet: Deep Category-Aware Semantic Edge Detection.
Cheng T, Wang X, Huang L, Liu W, Boundary-Preserving Mask R-CNN. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) [Internet]. 2020. p. 660–76. Available from: https://link.springer.com/10.1007/978-3-030-58568-6_39.
Bucksch A, Lindenbergh R, Menenti M. Robust skeleton extraction from imperfect point clouds. Visual Comput. 2010;26:1283–300.
Isokane T, Okura F, Ide A, Matsushita Y, Yagi Y. Probabilistic Plant Modeling via Multi-view Image-to-Image Translation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2018;2906–15.
Gené-Mola J, Sanz-Cortiella R, Rosell-Polo JR, Morros JR, Ruiz-Hidalgo J, Vilaplana V et al. Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Comput Electron Agric [Internet]. 2020;169:105165. https://doi.org/10.1016/j.compag.2019.105165.
Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Comput Electron Agric [Internet]. 2020;171:105302. https://doi.org/10.1016/j.compag.2020.105302.
Zhang W, Hansen MF, Smith M, Smith L, Grieve B. Photometric stereo for three-dimensional leaf venation extraction. Comput Ind [Internet]. 2018;98:56–67. https://doi.org/10.1016/j.compind.2018.02.006.
Wen W, Li B, Li BJ, Guo X. A leaf modeling and multi-scale remeshing method for visual computation via hierarchical parametric vein and margin representation. Front Plant Sci. 2018;9:1–14.
Sarlikioti V, De Visser PHB, Buck-Sorlin GH, Marcelis LFM. How plant architecture affects light absorption and photosynthesis in tomato: Towards an ideotype for plant architecture using a functionalstructural plant model. Ann Bot. 2011;108:1065–73.
KAO W -Y, FORSETH IN. Dirunal leaf movement, chlorophyll fluorescence and carbon assimilation in soybean grown under different nitrogen and water availabilities. Plant Cell Environ [Internet]. 1992;15:703–10. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-3040.1992.tb01012.x.
Kahlen K, Wiechers D, Stützel H. Modelling leaf phototropism in a cucumber canopy. Funct Plant Biol. 2008;35:876–84.
Yuxin W, Alexander K, Francisco M, Wan-Yen L, Ross G. Detectron2 [Internet]. 2019 [cited 2022 Nov 8]. Available from: https://github.com/facebookresearch/detectron2.
Ester M, Kriegel HP, Sander J, Xiaowei X. A density-based algorithm for discovering clusters in large spatial databases with noise. 1996 [cited 2023 Feb 17]; Available from: https://www.osti.gov/biblio/421283.
Rusu RB, Cousins S. 3D is here: Point Cloud Library (PCL). IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China: IEEE; 2011.
Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, et al. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nat Methods. 2020;17:261–72.
Usumezbas A, Fabbri R, Kimia BB. From Multiview Image Curves to 3D Drawings. In: Leibe B, Matas J, Sebe N, Welling M, editors. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) [Internet]. Cham: Springer International Publishing; 2016. p. 70–87. Available from: http://link.springer.com/10.1007/978-3-319-46493-0.

Table 1 is available in the Supplementary Files section.

No competing interests reported.

Table1.xlsx
Table1. AP values of Mask R-CNN. Evaluated AP, AP50, AP75, APs, APm, and APl values on four individuals (Enrei, Zairai 51-2, Aoakimame, Saga zairai) as the test dataset.
Additionalfile1.pdf
Additional file 1. Figure S1. Support threshold balances the number of images and the accuracy of reconstruction. More 3D curve fragments, including inaccurate ones, have been reconstructed with a lower support threshold (left). Less, accurate 3D curve fragments with a higher support threshold have been reconstructed (right). Fig. S2. Leaf correspondence on multiple leaves in the same scene. T The 3D leaf edges are reconstructed if the leaf correspondences are known (left).Reconstructed leaf edges (curves with different colors) are reconstructed along the edges of the original mesh (gray). However, the 3D leaf edge reconstruction fails when the leaf correspondences are not known, and they are attempted to be simultaneously reconstructed (right). Purple curves indicate 3D curve fragments. Figure S3. Learning curve of Mask R-CNN using Enrei as test data. Figure S4. Scatter diagrams of the optimal support thresholds. Each diagram shows the optimal support thresholds to the OI with different values for camera positional noise (row) and the number of images (column). Figure S5. Photogrammetry system. A soybean individual is placed on a turntable table. Multi-view images of the plant have been taken by 8 digital cameras controlled by digital photography software.
Additionalfile2.gif
Additional file 2. Movie of the reconstructed 3D leaf edges of the simulation data.
Additionalfile3.mp4
Additional file 3. Movie of the reconstructed 3D leaf edges of the actual soybean plant (Fukuyutaka, 21 DAS, mean – 0.25 SD).
Additionalfile4.mp4
Additional file 4. Movie of the reconstructed 3D leaf edges of the actual soybean plant (Fukuyutaka, 28 DAS, mean – 0.5 SD).
Additionalfile5.mp4
Additional file 5. Movie of the reconstructed 3D leaf edges of the actual soybean plant (Fukuyutaka, 42 DAS, mean – 0.5 SD).
Additionalfile6.xlsx
Additional file 6. Table S1. A list of variables used in this study.

Download PDF

Version 1

posted

You are reading this latest preprint version

Three-dimensional leaf edge reconstruction using a combination of two- and three-dimensional phenotyping approaches

Status:

Version 1

Abstract

Figures

Background

Results

Discussion

Conclusions

Methods

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1