A novel approach to dining bowl reconstruction for image-based food volume estimation

Authors: JIA, WENYAN - University Of Pittsburgh; REN, YIQIU - University Of Pittsburgh; LI, BOYANG - University Of Pittsburgh; BEATRICE, BRITNEY - University Of Pittsburgh; QUE, JINGDA - University Of Pittsburgh; CAO, SHUNXIN - University Of Pittsburgh; WU, ZEKUN - University Of Pittsburgh; MAO, ZHI - University Of Pittsburgh; LO, BENNY - Imperial College; ANDERSON, ALEX - University Of Georgia; FROST, GARY - Imperial College; MCCRORY, MEGAN - Boston University; SAZONOV, EDWARD - University Of Alabama; STEINER-ASIEDU, MATILDA - University Of Ghana; BARANOWSKI, TOM - Children'S Nutrition Research Center (CNRC); BURKE, LORA - University Of Pittsburgh; SUN, MINGUI - University Of Pittsburgh

Submitted to: Sensors
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 2/8/2022
Publication Date: 2/15/2022
Citation: Jia, W., Ren, Y., Li, B., Beatrice, B., Que, J., Cao, S., Wu, Z., Mao, Z.H., Lo, B., Anderson, A.K., Frost, G., McCrory, M.A., Sazonov, E., Steiner-Asiedu, M., Baranowski, T., Burke, L.E., Sun, M. 2022. A novel approach to dining bowl reconstruction for image-based food volume estimation. Sensors. 22. Article 1493. https://doi.org/10.3390/s22041493.
DOI: https://doi.org/10.3390/s22041493

Interpretive Summary: Knowing the amounts of energy and nutrients in an individual's diet is important for understanding amounts needed for maintaining health and preventing chronic diseases. As dietary assessment can now be performed using food images obtained from a smartphone or a wearable device, a challenge is to measure the volume of food in a bowl from an image. A new method to measure the size and shape of a bowl involves adhering a paper ruler centrally across the bottom and sides of the bowl and taking an image. An algorithm was developed to reconstruct the three-dimensional bowl interior using observed distortions. Our experiments using nine bowls, colored liquids, and amorphous foods demonstrated high accuracy for food volume estimation in a bowl as compared to an independent human estimator. This method advances the use of technological analysis of images for dietary intake assessment.

Technical Abstract: Knowing the amounts of energy and nutrients in an individual's diet is important for maintaining health and preventing chronic diseases. As electronic and AI technologies advance rapidly, dietary assessment can now be performed using food images obtained from a smartphone or a wearable device. One of the challenges in this approach is to computationally measure the volume of food in a bowl from an image. This problem has not been studied systematically despite the bowl being the most utilized food container in many parts of the world, especially in Asia and Africa. In this paper, we present a new method to measure the size and shape of a bowl by adhering a paper ruler centrally across the bottom and sides of the bowl and then taking an image. When observed from the image, the distortions in the width of the paper ruler and the spacings between ruler markers completely encode the size and shape of the bowl. A computational algorithm is developed to reconstruct the three-dimensional bowl interior using the observed distortions. Our experiments using nine bowls, colored liquids, and amorphous foods demonstrate high accuracy of our method for food volume estimation involving round bowls as containers. A total of 228 images of amorphous foods were also used in a comparative experiment between our algorithm and an independent human estimator. The results showed that our algorithm overperformed the human estimator who utilized different types of reference information and two estimation methods, including direct volume estimation and indirect estimation through the fullness of the bowl.