Deep learning-based apple detection using a suppression mask R-CNN

Authors: CHU, PENGYU - Michigan State University; LI, ZHAOJIAN - Michigan State University; LAMMERS, KYLE - Michigan State University; Lu, Renfu; LIU, XIAOMING - Michigan State University

Submitted to: Pattern Recognition Letters
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 4/19/2021
Publication Date: 7/1/2021
Citation: Chu, P., Li, Z., Lammers, K., Lu, R., Liu, X. 2021. Deep learning-based apple detection using a suppression mask R-CNN. Pattern Recognition Letters. 147:206-211. https://doi.org/10.1016/j.patrec.2021.04.022.
DOI: https://doi.org/10.1016/j.patrec.2021.04.022

Interpretive Summary: Apple harvesting is a labor-intensive operation, which accounts for about 15% of the total U.S. apple production cost. Development of robotic harvesting technology is urgently needed to address labor cost and availability concerns, reduce occupational health risks for workers, and ultimately improve the sustainability and profitability of the U.S. fruit industry. Fruit detection, which identifies fruits and provides targets for the robot to perform subsequent actions, is the first and foremost task in robotic harvesting. Computer vision has been widely used in robotic harvesting systems. However, accurate detection of apples on trees are still challenging due to variable lighting conditions and occlusion of fruits by leaves and/or branches. While many computer vision algorithms have been reported recently for fruit detection, the success rates are still short of meeting expectations. In this paper, we report on the development of a novel deep learning-based apple detection framework. A comprehensive dataset with 4,795 color images was collected for two varieties of ‘Gala’ and ‘Blondee’ apples under different lighting conditions (i. e., sunny versus overcast and direct lighting versus back lighting) from commercial orchards in Sparta, Michigan during the 2019 harvest season. A novel deep learning-based algorithm, called DeepApple, was proposed for apple detection. DeepApple outperformed state-of-the-art models with a higher detection accuracy of above 90% and a detection time of 0.25 second per image when the algorithm was implemented on a desktop computer, which can meet the need for robotic harvesting of apples. The developed algorithm has been incorporated into a new apple harvest robot that is being developed.

Technical Abstract: Robotic apple harvesting has received much research attention recently due to growing shortage and rising cost in labor. One key enabling technology towards automated harvesting is accurate and robust apple detection, which poses great challenges as a result of the complex orchard environment that involves varying lighting conditions and foliage/branch occlusions. This paper reports on the development of a novel deep learning-based apple detection framework named suppression Mask R-CNN (also called DeepApple). Specifically, we first collected a comprehensive apple orchard dataset for ‘Gala’ and ‘Blondee’ apples, using a color camera, under different lighting conditions (sunny vs. overcast and front lighting vs. back lighting). We then developed a novel suppression Mask R-CNN for apple detection, in which a suppression branch is added to the standard Mask R-CNN to suppress non-apple features generated by the original network. Comprehensive evaluations were performed, which showed that the developed suppression Mask R-CNN network outperformed state-of-the-art models with a higher F1-score of 0.905 and a detection time of 0.25 second per frame on a standard desktop computer.