Location: Diet, Microbiome and Immunity Research
Project Number: 2032-10700-002-006-S
Project Type: Non-Assistance Cooperative Agreement
Start Date: Jul 1, 2022
End Date: Dec 31, 2024
Objective:
The objective of the current project is adapt deep learning algorithms for prediction of ingredients from real-world food photos used for dietary assessment and evaluate using the benchmark SNAPMe dataset.
Approach:
Aim 1) Compare/evaluate existing deep learning algorithms for ingredient predictions from benchmark food images (the SNAPMe dataset). The SNAPMe dataset includes food and beverage cell phone photos with labels, paired with a traditional dietary assessment data in the form of food records collected using the Automated Self-Administered 24-hour Dietary Assessment Tool (ASA24). The ASA24 text data will be used as the ground truth for ingredient labels. The types of foods in the SNAPMe dataset include both mixed dishes (multiple ingredient, e.g. mac and cheese) and core (single ingredient, e.g. apple) foods.
Aim 2) Adapt existing algorithms or develop new algorithms for improved predictions. The data used to train previous models include almost exclusively mixed (multiple ingredient) foods sourced from recipe websites commonly used in the U.S. Therefore, we anticipate that even the best performing model will struggle with single-ingredient core foods and multi-cultural foods. Datasets for retraining will include core food images from our curated Glycopedia Food Image DB, a database mostly consisting of single-ingredient foods, and ethnically diverse foods that fared poorly. For example, if predictions from Chinese cuisine fared poorly, we’d include images from VireoFood-172 food image dataset.
For all aims, the accuracy of the ingredient predictions/volume estimation will be evaluated using standard metrics. Utility for dietary assessment will be evaluated by fuzzy-string matching ingredients to USDA Food Data Central.
traditional dietary assessment data in the form of food records collected using the Automated Self-Administered 24-hour Dietary Assessment Tool (ASA24). The ASA24 text data will be used as the ground truth for ingredient labels. The types of foods in the SNAPMe dataset include both mixed dishes (multiple ingredient, e.g. mac and cheese) and core (single ingredient, e.g. apple) foods.
