Skip to main content
ARS Home » Research » Publications at this Location » Publication #371682
Title: Multimodal AI to improve agriculture

Author
item Parr, Cynthia - Cyndy
item Lemay, Danielle
item Owen, Christopher
item Woodward-Greene, Jennifer
item SUN, JIAYANG - George Mason University

Submitted to: IEEE IT Professional
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 4/3/2020
Publication Date: 6/24/2021
Citation: Parr, C.J, Lemay, D.G., Owen, C.L., Woodward Greene, M.J., Sun, J. 2021. Multimodal AI to improve agriculture. IEEE IT Professional. 23(3):53-57. https://doi.org/10.1109/MITP.2020.2986122.
DOI: https://doi.org/10.1109/MITP.2020.2986122

Interpretive Summary: Advances in natural language processing (NLP) and computer vision are now being applied to many agricultural problems. These techniques take advantage of non-traditional (or non-numeric) data sources such as text in libraries and images from field operations. However, these techniques could be more powerful if combined with AI and numeric sources of data in multi-modal pipelines. We present several recent examples where USDA-ARS researchers and collaborators are using AI methods with text and images to improve core scientific knowledge, the management of agricultural research, and agricultural practice. NLP enables automated indexing, clustering and classification for agricultural research project management. We explore two case studies where combining techniques and data sources in new ways could accelerate progress in personalized nutrition and invasive pest detection. One challenge in applying these techniques is the difficulty in obtaining high quality training data. Other challenges are a lack of machine learning (ML) techniques customized for use and ML skills or experience among researchers and other stakeholders. Initiatives are underway at USDA-ARS to address these challenges.

Technical Abstract: Advances in natural language processing (NLP) and computer vision are now being applied to many agricultural problems. These techniques take advantage of non-traditional (or non-numeric) data sources such as text in libraries and images from field operations. However, these techniques could be more powerful if combined with AI and numeric sources of data in multi-modal pipelines. We present several recent examples where USDA-ARS researchers and collaborators are using AI methods with text and images to improve core scientific knowledge, the management of agricultural research, and agricultural practice. NLP enables automated indexing, clustering and classification for agricultural research project management. We explore two case studies where combining techniques and data sources in new ways could accelerate progress in personalized nutrition and invasive pest detection. One challenge in applying these techniques is the difficulty in obtaining high quality training data. Other challenges are a lack of machine learning (ML) techniques customized for use and ML skills or experience among researchers and other stakeholders. Initiatives are underway at USDA-ARS to address these challenges.