Active laser-camera scanning for high-precision fruit localization in robotic harvesting: system design and calibration

Authors: ZHANG, KAIXIANG - Michigan State University; CHU, PENGYU - Michigan State University; LAMMERS, KYLE - Michigan State University; LI, ZHAOJIAN - Michigan State University; Lu, Renfu

Submitted to: Horticulturae
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 12/29/2023
Publication Date: 12/31/2023
Citation: Zhang, K., Chu, P., Lammers, K., Li, Z., Lu, R. 2023. Active laser-camera scanning for high-precision fruit localization in robotic harvesting: system design and calibration. Horticulturae. 10(1):40. https://doi.org/10.3390/horticulturae10010040.
DOI: https://doi.org/10.3390/horticulturae10010040

Interpretive Summary: Harvest automation is urgently needed to address the growing labor shortage and increasing labor cost issues facing the U.S. specialty crop industry. Fruit detection and localization is a critical first step in robotic harvesting of apples. Fruit localization is a procedure of determining the three-dimensional coordinates (or spatial positions) of target fruits on trees, so that the robot arm can reach the fruits for detachment. Currently, several sensing techniques, such as stereo vision and color-depth imaging (or RGB-D imaging), have been used for fruit localization. However, these sensors have not performed satisfactorily in localizing fruits, due to varying natural lighting conditions and fruit occlusions by leaves and branches. To address the shortcomings of the current techniques, we have designed and assembled a new active laser-camera scanning (ALACS) system, for high-precision fruit localization. The ALACS mainly consists of a red line laser, a color camera and a linear motorized stage. A localization model coupled with a robust calibration scheme was developed to enable high-precision computation of the spatial positions of target objects. Laboratory calibrations and evaluations showed that the calibrated ALACS system was able to achieve accurate localization of target objects with the mean localization errors of less than 1 mm and the maximum error of less than 4 mm, when the objects were within the robot’s working distance between 60 cm and 120 cm. The ALACS has been incorporated into a new version of the apple harvesting robot, which showed superior localization performance in orchard testing, compared to the conventional RGB-D imaging technique.

Technical Abstract: Robust and effective fruit detection and localization is essential for robotic harvesting systems. While extensive research efforts have been devoted to improving fruit detection, less emphasis has been placed on the fruit localization aspect, which is a crucial yet challenging task due to limited depth accuracy from existing sensor measurements in the orchard environment with variable lighting conditions and foliage/branch occlusions. In this paper, we present the system design and calibration of an active laser-camera scanner (ALACS), a novel perception module for robust and high-precision fruit localization. The hardware of ALACS mainly consists of a red line laser, an RGB camera, and a linear motion slide, which are seamlessly integrated into an active scanning scheme where a dynamic-targeting laser-triangulation principle is employed. A high-fidelity extrinsic model was developed to pair the laser illumination and the RGB camera, enabling precise depth computation when the target is captured by both sensors. A random sample consensus-based robust calibration scheme was then designed to calibrate the model parameters based on the collected data. Comprehensive evaluations were conducted to validate the system model and calibration scheme. The results showed that the proposed calibration method can detect and remove data outliers to achieve robust parameter computations, and the calibrated ALACS system is able to achieve high-precision localization with millimeter-level accuracy.