Location: Sugarbeet and Bean Research
2021 Annual Report
Objectives
1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples.
2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables.
Approach
Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss.
Progress Report
Objective 1: Based on the preliminary work in a prior year, a new version of robotic apple harvester prototype was built. Improvements were made to the design of the tilting and panning mechanisms for more accurate and precise movements of the robot arm in the vertical and horizontal directions. In addition, the rotation mechanism was redesigned, which resulted in a smoother detachment of fruit from trees. Three control strategies for the robot arm movement, including open-loop velocity control, position control, and close-loop velocity control, were evaluated in laboratory with fruit hanging at different positions within the robot’s working space. Results showed that among the three control strategies, the close-loop velocity control had the best performance with the average error of less than 20 mm in reaching the target fruit, which has met the initial expectations for the robot. In addition, preliminary research was conducted to optimize robot path planning for fruit picking and obstacle (tree branches) avoidance.
To improve the fruit gripping capability of the robot, three new end effectors were designed and fabricated using a silicon material. The new end effectors were tested for generating fruit-sucking pressure or force. Further, field tests were also conducted for three varieties of apple to determine the forces needed to detach fruit from trees by pulling or twisting movement. The collected picking force information was then used as a guidance in the design of the new end effectors. During the 2020 harvest season, the three new end effectors were evaluated for detaching fruit from trees, and the ‘straight’ end-effector design had overall better performance with 87% picking success rate.
New computer algorithms, based on deep learning technique, were developed for detecting fruit from trees. The algorithms were trained and tested with more than 1,500 images collected using a color and depth camera from a commercial orchard in Sparta, Michigan during 2019 harvest season under different natural light conditions (i.e., direct, back light, and overcast). The new algorithm, called suppression mask region-based convolutional neural network (or suppression Mask R-CNN), achieved superior performance in detecting fruit of ‘Gala’ and ‘Blondee’ cultivars, with the overall detection rate of over 90%. The algorithm was integrated with the robot hardware, and it showed promising performance for picking apples in orchard. However, field tests showed that the new algorithm had difficulty detecting those fruit that were heavily occluded by leaves. Moreover, it was found that heavy leaf occlusions could also cause large errors in determining the position of target fruit on trees, thus affecting the successful picking rate. Hence, further efforts are being made to improve the accuracy and efficiency of detecting and localizing target fruit.
To facilitate the development, testing and evaluation of a new robotic apple harvester, an artificial orchard environment was constructed to simulate the actual apple orchard. The new artificial orchard consists of artificial trees installed with different sizes of branches coupled with specially-designed artificial fruit stems for hanging real apples. A light system has been installed in the artificial orchard that simulates different sky conditions (i.e., clear sky, overcast and cloudy) for different times of day with adjustable light incident angles. The new artificial orchard environment is also installed with a commercial object localization system composed of eight cameras, which provides quick and accurate information on the three-dimensional position of objects within the working space. This new artificial orchard environment provides a useful platform to test and evaluate how a robotic harvester would operate under the real orchard environment.
Objective 2: Data analyses were completed for detection of subsurface bruising in fresh pickling cucumbers, using structured-light imaging (SLI) technique. The original SLI images for 240 ‘Vlaspick’ pickling cucumbers were first processed to obtain two sets of new images, i.e., direct component (DC) which are equivalent to the images that would be acquired by a conventional imaging system under uniform, diffuse illumination, and alternating component (AC) which are unique to SLI and may reveal some hidden features that would otherwise be difficult to detect by DC images. Image enhancement was then performed for the DC and AC images, using a newly developed technique, called fast bi-dimensional empirical mode decomposition (BEMD). Image features were extracted from the enhanced DC and AC images, and top 50 features were selected for establishing classification models based on a machine learning algorithm, called support vector machines. Superior classification results for bruised and normal cucumbers were obtained with the overall classification accuracy of 91%, when DC and AC images were combined. This study demonstrated that SLI can provide an effective means for detecting subsurface bruising of pickling cucumbers. A manuscript from this study was submitted for journal publication.
Preliminary tests were conducted to evaluate the performance of a high-speed imaging sensor for real-time acquisition of images from samples under structured lighting with different frame rates (i.e., images per second). These tests showed that this imaging setup would meet our need for building a laboratory SLI-based system for online inspection of horticultural products, such as tomato and cucumber.
Accomplishments
1. Development of a new robotic apple harvesting technology. Automated harvesting technology is urgently needed to address labor shortage and increasing labor cost issues facing the multi-billion dollar U.S. apple industry and other tree fruits. While considerable research in robotic harvesting has been reported in recent years, there still exist critical technical challenges of picking fruit from clusters and/or occluded by leaves and branches. In collaboration with researchers at Michigan State University, an ARS researcher at East Lansing, Michigan has developed a new robotic apple harvesting technology, which utilizes an innovative concept of vacuum sucking and rotation, coupled with a simple and effective robot arm movement mechanism, for picking fruit from trees. The new harvesting robot has demonstrated its effectiveness and dexterity in picking fruit from clusters and/or deep in canopy during field testing in 2020, and a patent application for the technology has been filed. With further research, the technology has the potential to change the way of how apples are being harvested and reduce the apple industry’s reliance on manual labor for fruit harvesting.
Review Publications
Lu, Y., Saeys, W., Kim, M.S., Peng, Y., Lu, R. 2020. Hyperspectral imaging technology for quality and safety evaluation of horticultural products: A review and celebration of the past 20-year progress. Postharvest Biology and Technology. 170. Article 111318. https://doi.org/10.1016/j.postharvbio.2020.111318.
Sun, Y., Lu, R., Pan, L., Tu, K., Wang, X. 2020. Assessment of the optical properties of peaches with fungal infection using spatially-resolved diffuse reflectance technique and their relationships with tissue structural and biochemical properties. Food Chemistry. 321. Article 126704. https://doi.org/10.1016/j.foodchem.2020.126704.
Zhang, Z., Lu, R., Cannayen, I. 2020. A time and motion study for evaluation of apple harvest processes with different harvest methods. Transactions of the ASABE. 63(6):1957-1967. https://doi.org/10.13031/trans.14144.
Chu, P., Li, Z., Lammers, K., Lu, R., Liu, X. 2021. Deep learning-based apple detection using a suppression mask R-CNN. Pattern Recognition Letters. 147:206-211. https://doi.org/10.1016/j.patrec.2021.04.022.
Zhang, Z., Lu, Y., Lu, R. 2021. Development and evaluation of an apple infield grading and sorting system. Postharvest Biology and Technology. 180. Article 111588. https://doi.org/10.1016/j.postharvbio.2021.111588.
Lu, Y., Lu, R., Zhang, Z. 2021. Detection of subsurface bruising in fresh pickling cucumbers using structured-illumination reflectance imaging. Postharvest Biology and Technology. 180. Article 111624. https://doi.org/10.1016/j.postharvbio.2021.111624.
Lu, Y., Lu, R. 2021. Detection of chilling injury in pickling cucumbers using dual-band chlorophyll fluorescence imaging. Foods. 10(5). Article 1094. https://doi.org/10.3390/foods10051094.
