O2RNet: Occluder-occludee relational network for robust apple detection of clustered orchard environments

Authors: CHU, PENGYU - Michigan State University; LI, ZHAOJIAN - Michigan State University; ZHANG, KAIXIANG - Michigan State University; CHEN, DONG - Michigan State University; LAMMERS, KYLE - Michigan State University; Lu, Renfu

Submitted to: Smart Agricultural Technology
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 7/1/2023
Publication Date: 7/11/2023
Citation: Chu, P., Li, Z., Zhang, K., Chen, D., Lammers, K., Lu, R. 2023. O2RNet: Occluder-occludee relational network for robust apple detection of clustered orchard environments. Smart Agricultural Technology. 5. Article 100284. https://doi.org/10.1016/j.atech.2023.100284.
DOI: https://doi.org/10.1016/j.atech.2023.100284

Interpretive Summary: Harvest automation is urgently needed to address the issues of labor shortage and rising labor costs. Fruit detection is a first important step in robotic harvesting. Accurate and robust detection of apples on trees is challenging, due to varying lighting conditions, fruit clustering and foliage/branch occlusions. Current computer vision algorithms are prone to errors in identifying individual fruit in clusters, which would cause issues in subsequent robotic harvesting operations. This research was therefore aimed at developing a new computer vision algorithm for enhanced detection of clustered apples under complex canopy structures and varying natural lighting conditions. A total of 900 color images were collected for two varieties of apple in two orchards in 2019 and 2021. These images were annotated and made available to the research community. A new artificial neural network, called the occluder-occludee relational network (O2RNet), was developed to enhance the detection of clustered apples. The new model was evaluated using the collected image data against 12 other state-of-the-art neural network models. It consistently outperformed the other models with an overall detection accuracy of 94%. The new model provides an effective method for robust detection of clustered apples, which is critical to robotic harvesting of apples.

Technical Abstract: Automated apple harvesting has attracted significant research interest in recent years because of its great potential to address the issues of labor shortage and rising labor costs. One key challenge to automated harvesting is accurate and robust apple detection, due to complex orchard environments that involve varying lighting conditions, fruit clustering and foliage/branch occlusions. Apples are often grown in clusters on trees, which may be mis-identified as a single apple and thus causes issues in fruit localization for subsequent robotic harvesting operations. In this paper, we present the development of a novel deep learning-based apple detection framework, called the Occluder-Occludee Relational Network (O2RNet), for robust detection of apples in clustered situations. A comprehensive dataset of RGB images were collected for two apple varieties under different lighting conditions (overcast, direct lighting, and back lighting) with varying degrees of apple occlusions, and the images were annotated and made available to the public. A novel occlusion-aware network was developed for apple detection, in which a feature expansion structure is incorporated into the convolutional neural networks to extract additional features generated by the original network for occluded apples. Comprehensive evaluations of the developed O2RNet were performed using the collected images, which outperformed 12 other state-of-the-art models with a higher accuracy of 94% and a higher F1-score of 0.88 on apple detection. O2RNet provides an enhanced method for robust detection of clustered apples, which is critical to accurate fruit localization for robotic harvesting.