Three-dimensional photogrammetry with deep learning instance segmentation to extract berry fruit harvestability traits

Authors: NI, XUEPING - Zhejiang University; LI, CHANGYING - University Of Georgia; JIANG, HUANYU - Zhejiang University; Takeda, Fumiomi

Submitted to: Journal of Photogrammetry and Remote Sensing
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 11/13/2020
Publication Date: 12/10/2020
Citation: Ni, X., Li, C., Jiang, H., Takeda, F. 2020. Three-dimensional photogrammetry with deep learning instance segmentation to extract berry fruit harvestability traits. Journal of Photogrammetry and Remote Sensing. 171:297-309. https://doi.org/10.1016/j.isprsjprs.2020.11.010.
DOI: https://doi.org/10.1016/j.isprsjprs.2020.11.010

Interpretive Summary: Making an informed decision on a variety’s yield potential, its suitability for machine harvesting and predicting harvest dates based on fruit development stage early in the season is important for blueberry producers and breeders. However, quantifying blueberry traits such as cluster size, compactness, and other fruit maturity variables is laborious. In this study, we developed a data processing pipeline to count berries, to measure maturity, and to evaluate cluster tightness automatically using a deep learning image segmentation of three-dimensional images of individual blueberry clusters captured with a digital camera. We used a strategy of translating the digital informatin through a series of mathematical equations with computer programs that assisted human operators to derive detailed information on the cluster size, fruit number on each cluster, maturity of each berry, and berry size. The computer-based translation of physical parameters of blueberry clusters reduced the data processing time. The resulting deep learning image segmentation techniques will be a valuable and efficient tool for blueberry producers and breeders for evaluating blueberry genotypes for machine harvestability, monitoring fruit development, fruit size, and estimating crop potenital.

Technical Abstract: To monitor blueberry development and improve blueberry management, it is necessary to extract blueberry cluster traits. Cluster traits, such as compactness, maturity, berry number, and berry size, are four important traits associated with harvestability and yield. In this study, an image-capturing system was developed to capture blueberry images from three different viewpoints to facilitate 3D reconstruction. The reconstruction was performed for four blueberry cultivars (‘Emerald’, ‘Farthing’, ‘Meadowlark’ and ‘Star’) with 10 cluster samples for each cultivar. A minimum bounding box was created to surround a 3D blueberry cluster bunch to calculate compactness as the ratio of berry volume and minimum bounding box volume. Mask R-CNN was used to segment individual blueberries with mature properties in the captured 2D images. To obtain 3D-2D mask correspondence, 3D blueberries were projected to 2D blueberry masks. A trait extraction algorithm was developed to segment individual 3D blueberries to obtain berry number, individual berry volume, and berry maturity. Berry maturity was used to calculate cluster maturity as the ratio of the mature berry (blue) number and the total berry (blue, reddish, and green) number comprising the cluster. The accuracy for determining the fruit number in a cluster was 97.3%. The linear regression for cluster maturity had an R2 of 0.908 with an RMSE of 0.068. The statistical analysis for the four cultivars showed that(a) only ‘Farthing’ and ‘Meadowlark’ had significant differences in cluster maturity in the middle of April; (b) ‘Emerald’ and ‘Farthing’ were more compact than ‘Meadowlark’ and ‘Star’ in the middle of April; and (c) the mature berry volume of ‘Farthing’ is larger than ‘Emerald’ and ‘Meadowlark’, while ‘Star’ had the smallest mature berry number. This study developed an efficient method for extracting 3D blueberry cluster traits for monitoring fruit development. This blueberry trait extraction method based on 3D reconstruction and deep learning can accurately and rapidly determine the cluster compactness, fruit size, estimated yield, and harvest time. This method provides a rapid and reliable technique for extracting fruit phenotypic traits from a large sample size.