System design and control of an apple harvesting robot

Authors: ZHANG, KAIXIANG - Michigan State University; LAMMERS, KYLE - Michigan State University; CHU, PENGYU - Michigan State University; LI, ZHAOJIAN - Michigan State University; Lu, Renfu

Submitted to: Mechatronics
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 8/4/2021
Publication Date: 11/1/2021
Citation: Zhang, K., Lammers, K., Chu, P., Li, Z., Lu, R. 2021. System design and control of an apple harvesting robot. Mechatronics. 79. Article 102644. https://doi.org/10.1016/j.mechatronics.2021.102644.
DOI: https://doi.org/10.1016/j.mechatronics.2021.102644

Interpretive Summary: Apple harvest is a labor intensive operation and has become a major cost component in apple production in U.S. Hence, there is a growing need for robotic apple harvesting to address the critical issues of increasing cost and decreased availability in labor for the apple industry. While much effort on the development of robotic harvesting technology has been made in recent years, existing robotic harvesting systems are still short of meeting performance expectations, lack of robustness, and inefficient or too complex for commercial applications. In this paper, we report on the development of a new robotic apple harvesting system. The system consists of a low-cost 3-dimensional color camera for detecting and localizing fruit on trees, a simple pan-and-tilt mechanism which allows the robot to move quickly in the horizontal and vertical directions, and a pneumatic actuator for fast movements in and out the tree canopy. Instead of using a conventional mechanical gripper, the vacuum-based robot end-effector enables easy, fast grasping and detachment of fruit from trees. A control algorithm was developed for controlling the robot end-effector for reaching to target fruit. The three-dimensional color vision system based on a deep learning algorithm achieved 92.7% detection rate for 1,243 images of ‘Gala’ and ‘Blondee’ apples collected from a commercial orchard in 2019. Laboratory tests showed that the apple harvesting robot was able to reach to the target fruit with the overall distance error of less than 2 cm, which has met our expectations of picking fruit from trees. This work has laid an important foundation for further development of a commercially viable apple harvesting technology.

Technical Abstract: There is a growing need for robotic apple harvesting due to decreased availability and rising cost in labor. Towards the goal of developing a viable robotic system for apple harvesting, this paper presents a synergistic approach to mechatronic design and motion control of a robotic apple harvesting prototype. Specifically, we developed a deep learning-based fruit detection and localization system using an RGB-D camera. A three degree-of-freedom manipulator was designed with a hybrid pneumatic/motor actuation mechanism to achieve fast and dexterous movements. A vacuum-based end-effector was used for apple detaching. These three components are integrated into a robotic apple harvesting prototype with simplicity, compactness, and robustness. Moreover, a nonlinear velocity-based control scheme was developed for the manipulator to achieve accurate and agile motion control. The apple harvesting robot achieved satisfactory performance in laboratory tests.