Food/non-food classification of real-life egocentric images in low- and middle-income countries based on image tagging features

Authors: CHEN, GUANGZONG - University Of Pittsburgh; JIA, WENYAN - University Of Pittsburgh; ZHAO, YIFAN - University Of Pittsburgh; MAO, ZHI - University Of Pittsburgh; LO, BENNY - Imperial College; ANDERSON, ALEX - University Of Georgia; FROST, GARY - Imperial College; JOBARTEH, MODOU - Imperial College; MCCRORY, MEGAN - Boston University; SAZONOV, EDWARD - University Of Alabama; STEINER-ASIEDU, MATILDA - University Of Ghana; ANSONG, RICHARD - University Of Ghana; BARANOWSKI, TOM - Children'S Nutrition Research Center (CNRC); BURKE, LORA - University Of Pittsburgh; SUN, MINGUI - University Of Pittsburgh

Submitted to: Frontiers in Artificial Intelligence
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 2/26/2021
Publication Date: 4/1/2021
Citation: Chen, G., Jia, W., Zhao, Y., Mao, Z.H., Lo, B., Anderson, A.K., Frost, G., Jobarteh, M.L., McCrory, M.A., Sazonov, E., Steiner-Asiedu, M., Ansong, R.S., Baranowski, T., Burke, L., Sun, M. 2021. Food/non-food classification of real-life egocentric images in low- and middle-income countries based on image tagging features. Frontiers in Artificial Intelligence. 4:644712. https://doi.org/10.3389/frai.2021.644712.
DOI: https://doi.org/10.3389/frai.2021.644712

Interpretive Summary: Undernutrition and obesity both reflect dietary intake and represent significant problems in low- and middle-income countries (LMICs). To study food intake and thereby develop effective intervention strategies, it must be studied at the individual, household, and community levels. In a multinational research project, a wearable camera, the eButton worn on the chest, is used to conduct objective dietary assessment in sub-Saharan Africa. The assessment includes multiple diet-related activities in urban and rural families, including food sources (e.g., shopping, harvesting, and gathering), preservation/storage, preparation, cooking, and consumption (e.g., portion size and nutrition analysis). The recorded images (tens of thousands per day) are post-processed to obtain the diet related information. The first step in this process is to use Artificial Intelligence (AI) technology to separate the acquired images into two binary classes: images with (Class 1) and without (Class 0) edible items. A combination of AI algorithms achieved high levels of sensitivity and specificity. Thereby, researchers need only to study Class-1 images, reducing their image processing workload significantly.

Technical Abstract: Malnutrition, including both undernutrition and obesity, is a significant problem in low- and middle-income countries (LMICs). In order to study malnutrition and develop effective intervention strategies, it is crucial to evaluate nutritional status in LMICs at the individual, household, and community levels. In a multinational research project supported by the Bill & Melinda Gates Foundation, we have been using a wearable technology to conduct objective dietary assessment in sub-Saharan Africa. Our assessment includes multiple diet-related activities in urban and rural families, including food sources (e.g., shopping, harvesting, and gathering), preservation/storage, preparation, cooking, and consumption (e.g., portion size and nutrition analysis). Our wearable device ("eButton" worn on the chest) acquires real-life images automatically during wake hours at preset time intervals. The recorded images, in amounts of tens of thousands per day, are post-processed to obtain the information of interest. Although we expect future Artificial Intelligence (AI) technology to extract the information automatically, at present we utilize AI to separate the acquired images into two binary classes: images with (Class 1) and without (Class 0) edible items. As a result, researchers need only to study Class-1 images, reducing their workload significantly. In this paper, we present a composite machine learning method to perform this classification, meeting the specific challenges of high complexity and diversity in the real-world LMIC data. Our method consists of a deep neural network (DNN) and a shallow learning network (SLN) connected by a novel probabilistic network interface layer. After presenting the details of our method, an image dataset acquired from Ghana is utilized to train and evaluate the machine learning system. Our comparative experiment indicates that the new composite method performs better than the conventional deep learning method assessed by integrated measures of sensitivity, specificity, and burden index, as indicated by the Receiver Operating Characteristic (ROC) curve.