Location: Sugarbeet and Bean Research
2022 Annual Report
Objectives
1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples.
2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables.
Approach
Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss.
Progress Report
Objective 1:
Major improvements have been made to enhance the overall performance of the harvesting robot for picking apples. Specifically, the robot’s manipulation system, including the pan/tilt and rotation mechanisms, was upgraded or redesigned for faster and more accurate movement of the robot’s arm to target fruit. A new, fully automated fruit dropping mechanism was constructed and incorporated into the robot, which allows quick dropping of harvested apples to the fruit catching module. An optimization scheme was implemented in the robot’s planning and control algorithm to determine the best picking and dropping strategies to reduce the fruit picking cycle. In addition, a modified camera configuration was used in the perception system to improve fruit detection caused by leave occlusion and hence fruit localization. A new catching module was constructed, which allows the robot to release picked apples in the shortest distance and time possible.
Field testing and evaluation of the improved version of the robot was conducted in the research orchards of Michigan State University’s Horticultural Teaching and Research Center at Holt, Michigan during 2021 harvest season. Results showed that the robot was able to perform continuous picking at an average speed of 3.6 s per fruit, which represents a significant improvement over the previous version of the robot and is also much faster than that reported by other research groups. Field tests also showed that a modified camera configuration has resulted in about a 3 percent fruit detection rate improvement, but their performance for fruit localization was still short of meeting our expectations.
To improve fruit localization by the robot’s perception system, a sensor fusion approach was proposed by integrating a Lidar sensor with our existing perception system. Calibration and evaluation of the lidar sensor was conducted in the laboratory to determine its ability to locate target fruit. Results showed that the lidar sensor was unable to meet the fruit localization accuracy requirement. Hence, research is being conducted to design and construct a new perception system for more accurate fruit localization.
Although the end effector tested in 2021 field tests performed much better than the original version, it was still short of meeting our requirement of picking 99% fruit after the end effector engages with target fruit. Hence, further effort was made to design a new version of the end effector along with the selection of a more efficient vacuum system. After several rounds of trial, a new end effector was designed. In laboratory evaluation, this new effector has exceeded our expectation in generating suction forces needed for gripping and detaching fruit from trees. An invention disclosure was submitted for the new end effector in May 2022.
Objective 2:
Two versions of the structured-light imaging (SLI) system, one for laboratory use and the other for future online real-time inspection of food products, have been designed. The laboratory version system integrated with an inhouse developed computer program has been tested and evaluated for taking pattern images from moving samples, while the online version system with a different optical configuration is being assembled for simultaneous acquisition of multiple pattern images from moving samples. Preliminary evaluation of the two system designs indicated that they would meet the imaging speed requirements for laboratory and future online inspection applications. Major components needed for the online SLI system have been acquired and ready for testing and integration with an existing laboratory platform.
Accomplishments
1. An integrated robotic system for efficient apple picking. Automated harvesting technology is urgently needed to reduce the fruit industry’s reliance on manual labor for fruit harvesting and overall production cost. However, there still exist major technological hurdles in developing cost effective and fast harvesting robots. A team of researchers from ARS and Michigan State University in East Lansing, Michigan has developed an integrated robotic system with new, improved fruit picking, dropping and catching mechanisms and by optimizing the robot’s planning and control strategies, to enhance its overall harvest efficiency. The improved fruit picking mechanism has had superior performance, while the optimized planning and control algorithms coupled with the fruit dropping and catching mechanisms have greatly improved the robot’s harvesting speed. The optimized robotic system was able to pick apples at an average rate of 3.6 s per fruit during 2021 field harvest testing, which is significantly better than the previous version of the robotic system and is also much faster than that reported by other research groups. With further improvements, the technology will help U.S. apple growers mitigate harvesting labor shortage and achieve overall production cost savings
Review Publications
Lu, Y., Zhang, Z., Lu, R. 2022. Development and preliminary evaluation of a new apple harvest assist and in-field sorting machine. Applied Engineering in Agriculture. 38(1):23-35. https://doi.org/10.13031/aea.14522.
Zhang, K., Lammers, K., Chu, P., Li, Z., Lu, R. 2021. System design and control of an apple harvesting robot. Mechatronics. 79. Article 102644. https://doi.org/10.1016/j.mechatronics.2021.102644.
