Surveying nutrient assessment with photographs of meals (SNAPMe): a benchmark dataset of food photos for dietary assessment

Authors: Larke, Jules; CHIN, ELIZABETH - University Of California, Davis; BOUZID, YASMINE - University Of California, Davis; Nguyen, Tu; VAINBERG, YAEL - University Of California, Davis; LEE, DONGHEE - University Of California, Davis; PIRSIAVASH, HAMED - University Of California, Davis; SMILOWITZ, JENNIFER - University Of California, Davis; Lemay, Danielle

Submitted to: The American Journal of Clinical Nutrition
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 11/22/2023
Publication Date: 11/30/2023
Citation: Larke, J.A., Chin, E.L., Bouzid, Y.Y., Nguyen, T.T., Vainberg, Y., Lee, D., Pirsiavash, H., Smilowitz, J.T., Lemay, D.G. 2023. Surveying nutrient assessment with photographs of meals (SNAPMe): a benchmark dataset of food photos for dietary assessment. The American Journal of Clinical Nutrition. 15(23) 4972. https://doi.org/10.3390/nu15234972.
DOI: https://doi.org/10.3390/nu15234972

Interpretive Summary: Photo-based dietary assessment methods are becoming more feasible as artificial intelligence methods improve. However, advancement of these methods to the level usable in nutrition studies has been hindered by the lack of a dataset against which to benchmark algorithm performance. We conducted the Surveying Nutrient Assessment with Photographs of Meals (SNAPMe) Study (ClinicalTrials ID: NCT05008653) to develop a benchmark dataset of food photographs paired with traditional food records. The SNAPMe database contains 3,311 unique food photos linked with 275 food records from 95 participants who photographed all foods consumed and recorded food records in parallel for up to 3 study days each. We then used the SNAPMe benchmark to evaluate publicly available algorithms for ingredient prediction for food photos and found that existing algorithms performed poorly, especially for single-ingredient foods and beverages. The SNAPMe database will provide nutrition and computer science researchers with a benchmark to test future algorithms for the improvement of photo-based dietary assessment.

Technical Abstract: Background: Photo-based dietary assessment is becoming more feasible as artificial intelligence methods improve. However, advancement of these methods for dietary assessment in research settings has been hindered by the lack of an appropriate dataset against which to benchmark algorithm performance. Objective: The objective of the current study is to develop a benchmark data set for the evaluation of computer vision algorithms for the use of food photos in dietary intake assessment. Methods: We conducted the Surveying Nutrient Assessment with Photographs of Meals (SNAPMe) Study (ClinicalTrials ID: NCT05008653) to pair meal photographs with traditional food records. Participants were recruited nationally and 110 completed enrollment meetings via web-based video conferencing. Participants uploaded and annotated their meal photos using a mobile phone app called Bitesnap and completed food records using the Automated Self-Administered 24-hour Dietary Assessment Tool (ASA24®) on the same day. Participants included photos before and after eating non-packaged and multi-serving packaged meals, as well as photos of the front package label and ingredient label for single-serving packaged foods. Results: The SNAPMe Database (DB) contains 3,311 unique food photos linked with 275 ASA24 food records from 95 participants who photographed all foods consumed and recorded food records in parallel for up to 3 study days each. Each line item of each ASA24 food record is linked to a food photo. Use of the SNAPMe DB to evaluate ingredient prediction with the publicly available Facebook Inverse Cooking (F1 score = 0.23) and Im2Recipe (F1 score = 0.13) algorithms demonstrates a domain gap for single ingredient foods. Correlations between nutrient estimates common to BiteSnap and ASA24 indicated a range in predictive capacity across nutrients (Cholesterol, Adjusted R2 = 0.85, p < 1.3e-34; Food Folate, Adjusted R2 = 0.21, p < 2.9e-4). Conclusion: SNAPMe DB is a publicly available benchmark for photo-based dietary assessment in nutrition research. Its utility has been demonstrated, suggesting areas of improvement needed for publicly available models, especially for the prediction of single ingredient foods and beverages.