Detection of surface and subsurface defects of apples using structured-illumination reflectance imaging with machine learning algorithms

Authors: LU, YUZHEN - Michigan State University; Lu, Renfu

Submitted to: Transactions of the ASABE
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 9/17/2018
Publication Date: 9/17/2018
Citation: Lu, Y., Lu, R. 2018. Detection of surface and subsurface defects of apples using structured-illumination reflectance imaging with machine learning algorithms. Transactions of the ASABE. 61(6):1831-1842. https://doi.org/10.13031/trans.12930.
DOI: https://doi.org/10.13031/trans.12930

Interpretive Summary: Computer vision technology is now widely used for automated detection of defects on apples; however, its performance still falls short of the industry’s expectations because some surface and/or subsurface defects are either not visible or can be confused with the normal tissue during image processing and classification. To overcome the shortcomings of conventional computer vision technology with uniform illumination for fruit defect detection, a new structured-illumination reflectance imaging (SIRI) system was developed for fruit defect detection. With the SIRI, fruit to be inspected is imaged under illumination of sinusoidal patterns, and the acquired images are then decomposed into two sets of images, i.e., direct component (DC), which corresponds to the image acquired under uniform illumination, and amplitude component (AC), which provides additional, new image features. In this research, three machine learning algorithms were developed for processing the DC and AC images and their combinations to detect surface and subsurface defects on ‘Delicious’ and ‘Golden Delicious’ apples. Results showed that DC images were better than AC images in detecting surface defects, while AC images were superior to DC images for subsurface defect (i.e., bruising) detection. Combination of DC and AC images improved defect detection results. Among the three image processing algorithms, the convolutional neural network performed the best with 98% detection accuracies for both varieties of apple. This research demonstrated that SIRI, coupled with the new image processing algorithm, can provide an effective means for enhancing defect detection of apples. With further improvement in imaging acquisition speed, the technique has potential for quality inspection of apples and other horticultural and food products.

Technical Abstract: Machine vision technology coupled with uniform illumination is now widely used for automatic sorting and grading of apples and other fruits, but it still does not have satisfactory performance for defect detection because there are a large variety of defects, some of which are difficult to detect under uniform illumination. Structured-illumination reflectance imaging (SIRI) offers a new modality by using sinusoidally-modulated structured illumination, to obtain two sets of independent images, i.e., direct component (DC), which corresponds to conventional uniform illumination, and amplitude component (AC), which is unique for structured illumination. The objective of this study was to develop machine learning classification algorithms using DC and AC images and their combinations for enhanced detection of surface and subsurface defects of apples. A multispectral SIRI system under illumination of two phase-shifted sinusoidal patterns was used to acquire images from ‘Delicious’ and ‘Golden Delicious’ apples with various types of surface and subsurface defects. DC and AC images were extracted through demodulation of the acquired images, and were then enhanced using bi-dimensional empirical mode decomposition and subsequent image reconstruction. Defect detection algorithms were developed, by using random forest (RF), support vector machine (SVM) and convolutional neural network (CNN), for DC, AC, and ratio (AC divided by DC) images and their combinations. Results showed that AC images were superior to DC images for detecting subsurface defects and DC images were overall better than AC for detecting surface defects, whereas ratio images were comparable to, or better than, DC images for defect detection. The ensemble of DC, AC and ratio images resulted in significantly better detection accuracies over using them individually. Among the three classifiers, CNN performed the best with 98% detection accuracies for both varieties of apple, followed by SVM and RF. This research demonstrated that SIRI, coupled with a machine learning algorithm, can provide a new, versatile and effective modality for fruit defect detection.