Modeling realistic 3D agricultural vegetations using photometric-based approach and its application to weed detection

Authors: HU, CHENGSONG - Texas A&M University; THOMASSON, J. ALEX - Mississippi State University; REBERG-HORTON, S. CHRIS - North Carolina State University; Mirsky, Steven; BAGAVATHIANNAN, MUTHUKUMAR - Texas A&M University

Submitted to: Computers and Electronics in Agriculture
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 4/28/2022
Publication Date: 5/18/2022
Citation: Hu, C., Thomasson, J., Reberg-Horton, S., Mirsky, S.B., Bagavathiannan, M.V. 2022. Modeling realistic 3D agricultural vegetations using photometric-based approach and its application to weed detection. Computers and Electronics in Agriculture. https://doi.org/10.1016/j.compag.2022.107020.
DOI: https://doi.org/10.1016/j.compag.2022.107020

Interpretive Summary: The advent of precision agriculture technologies, such as the use of GPS-guided tractors and variable-rate fertilizer applications, has the potential to increase the efficiency and sustainability of US cropping systems. However, the use of precision agriculture for weed control (one of the most time- and input- intensive field activities) has been limited due to the difficulty in using sensors or cameras to tell the difference between weeds and crops. This article reports on a new method for using basic camera equipment and algorithms to develop 3D plant image data sets that were then used to train neural networks for weed detection. This work makes possible the development of publicly available 3D crop and weed photo databases, which is key to enabling precision weed control methods. Results from this work will inform a framework for building a national image repository of weeds for researchers and industry to build precision technologies with.

Technical Abstract: 3D computer graphics is one of the major approaches to create synthetic images to train, evaluate, or validate computer vision systems. Not only can it generate photorealistic RGB images, but also ground truth information such as pixel-wise classification and scene depth, which is expensive to generate from real images. Attempts have been made to utilize computer graphics in agriculture, showing great potential to improve the performance of computer vision algorithms based on convolutional neural networks. However, the complexity of agricultural vegetations has been largely hindering the development of 3D plant image datasets for agricultural applications. Here, a framework is developed to facilitate the synthesis of 3D agricultural vegetation scenes that are both visually appealing and physically plausible with sufficient geometric and optical details. The framework fully relies on photometric approaches, thus requiring no sophisticated equipment. It starts with an efficient method to acquire dual-faced leaf models with details of leaf geometry, light reflectance and light transmittance. A parametric L-system template that can be easily modified for different plant species is used to organize leaf models in a geometric arrangement that resembles real plants. Finally, a ray tracing approach is adopted to produce high levels of visual realism. This framework produces both visually appealing individual plants and large agricultural scenes. To show the robustness of the proposed framework, we used the rendered images to train neural networks for detection of weeds, which are major pest organisms threatening crop production across the world. A considerable boost of detection performance is granted by the rendered images, as well as the ability for instance segmentation. The promising results obtained here open up several interesting areas for future work, one of which being the development of publicly available 3D crop and weed databases.