I2N: Image to Nutrients, a sensor guided semi-automated tool for annotation of images for nutrition analysis of eating episodes

Authors: GHOSH, TONMOY - UNIVERSITY OF ALABAMA; MCCRORY, MEGAN - BOSTON UNIVERSITY; MARDEN, TYSON - UNIVERSITY OF COLORADO; HIGGINS, JANINE - UNIVERSITY OF COLORADO; ANDERSON, ALEX - UNIVERSITY OF GEORGIA; DOMFE, CHRISTABEL - UNIVERSITY OF GEORGIA; JIA, WENYAN - UNIVERSITY OF PITTSBURGH; LO, BENNY - IMPERIAL COLLEGE; FROST, GARY - IMPERIAL COLLEGE; STEINER-ASIEDU, MATILDA - UNIVERSITY OF GHANA; BARANOWSKI, TOM - CHILDREN'S NUTRITION RESEARCH CENTER (CNRC); SUN, MINGUI - UNIVERSITY OF PITTSBURGH; SAZONOV, EDWARD - UNIVERSITY OF ALABAMA

Submitted to: Frontiers in Nutrition
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 7/5/2023
Publication Date: 7/27/2023
Citation: Ghosh, T., McCrory, M.A., Marden, T.J., Higgins, J., Anderson, A.K., Domfe, C.A., Jia, W., Lo, B.P., Frost, G., Steiner-Asiedu, M., Baranowski, T., Sun, M., Sazonov, E. 2023. I2N: Image to Nutrients, a sensor guided semi-automated tool for annotation of images for nutrition analysis of eating episodes. Frontiers in Nutrition. 10. Article 1191962. https://doi.org/10.3389/fnut.2023.1191962.
DOI: https://doi.org/10.3389/fnut.2023.1191962

Interpretive Summary: Traditional methods of monitoring food intake through self-report is error prone and can be time-consuming for the user. This paper presents a semi-automatic dietary assessment tool called Image to Nutrients (I2N). I2N estimates energy intake, nutritional content, and the amount consumed. It incorporates several nutrient databases (e.g., USDA-SR, FNDDS, USDA Global Branded Food Products Database) for annotating eating episodes and food items. Two studies were used to evaluate the feasibility and usefulness of I2N: 1) a US-based study with 30 participants and a total of 60 days of data and 2) a Ghana-based study with 41 participants and a total of 41 days of data). In both studies, a total of 314 eating episodes were annotated using at least three food databases. Using I2N's sensor-guided image review, the number of images that needed to be reviewed was reduced by 93% and 85% for the two studies, respectively, compared to reviewing all the images. I2N is a unique tool that allows for simultaneous viewing of food images, sensor-guided image review, and access to multiple databases in one tool, making nutritional analysis of food images efficient.

Technical Abstract: Dietary assessment is important for understanding nutritional status. Traditional methods of monitoring food intake through self-report such as diet diaries, 24-hour dietary recall, and food frequency questionnaires may be subject to errors and can be time-consuming for the user. This paper presents a semi-automatic dietary assessment tool we developed - a desktop application called Image to Nutrients (I2N) - to process sensor-detected eating events and images captured during these eating events by a wearable sensor. I2N has the capacity to offer multiple food and nutrient databases (e.g., USDA-SR, FNDDS, USDA Global Branded Food Products Database) for annotating eating episodes and food items. I2N estimates energy intake, nutritional content, and the amount consumed. The components of I2N are three-fold: 1) sensor-guided image review, 2) annotation of food images for nutritional analysis, and 3) access to multiple food databases. Two studies were used to evaluate the feasibility and usefulness of I2N: 1) a US-based study with 30 participants and a total of 60 days of data and 2) a Ghana-based study with 41 participants and a total of 41 days of data). In both studies, a total of 314 eating episodes were annotated using at least three food databases. Using I2N's sensor-guided image review, the number of images that needed to be reviewed was reduced by 93% and 85% for the two studies, respectively, compared to reviewing all the images. I2N is a unique tool that allows for simultaneous viewing of food images, sensor-guided image review, and access to multiple databases in one tool, making nutritional analysis of food images efficient. The tool is flexible, allowing for nutritional analysis of images if sensor signals aren't available.