Accuracy, reliability, and timing of visual evaluations of decay in fresh-cut lettuce

Authors: Simko, Ivan; Hayes, Ryan

Submitted to: PLOS ONE
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 4/10/2018
Publication Date: 4/17/2018
Citation: Simko, I., Hayes, R.J. 2018. Accuracy, reliability, and timing of visual evaluations of decay in fresh-cut lettuce. PLoS One. 13(4):e0194635. https://doi.org/10.1371/journal.pone.0194635.
DOI: https://doi.org/10.1371/journal.pone.0194635

Interpretive Summary: Visual assessments are used for evaluating the quality of food products, such as fresh-cut lettuce packaged in bags with modified atmosphere. We have compared the accuracy and the reliability of visual evaluations of decay on fresh-cut lettuce performed with experienced and inexperienced raters. In addition, we have analyzed decay data from over 4.5 thousand bags to determine the optimum timing for evaluations to detect differences among accessions. We have detected high repeatability, reproducibility, and accuracy for experienced raters. Inexperienced raters did not perform as well and their ratings showed decreased repeatability, but even larger reduction in reproducibility and accuracy. Under- or overestimated ratings were predominantly found for bags with intermediate levels of decay, which corresponds to the middle of the rating scale. This occurs because intermediate amounts of decay are more difficult to discriminate than extremes. The frequencies of aberrant ratings for experienced raters ranged from 0.6% to 4.4% (mean = 2.1%), for inexperienced raters the frequencies were substantially higher, ranging from 6.1% to 15.6% (mean = 9.4%). Therefore, we recommend that new raters receive training that includes practical examples in this range of decay, use of standard area diagrams, and continuing interaction with experienced raters (consultation during actual rating). Very high agreement among experienced raters indicate that visual ratings can be successfully used for evaluations of decay, until a more objective, rapid, and affordable method is developed. We recommend evaluating samples at multiple time points until 42 days after processing (about 80% decay on average) and then combining these individual ratings into the area under the decay progress stairs (AUDPS) score. Applying this approach, trained evaluators can accurately detect difference among lettuce accessions and identify lettuce cultivars with reduced decay.

Technical Abstract: Visual assessments are used for evaluating the quality of food products, such as fresh-cut lettuce packaged in bags with modified atmosphere. We have compared the accuracy and the reliability of visual evaluations of decay on fresh-cut lettuce performed with experienced and inexperienced raters. In addition, we have analyzed decay data from over 4.5 thousand bags to determine the optimum timing for evaluations to detect differences among accessions. Lin’s concordance coefficient ('c) that takes into consideration both the closeness of the data and the conformance to the identity line showed high repeatability (intra-rater reliability, 'c = 0.97), reproducibility (inter-rater reliability, 'c = 0.92), and accuracy ('c = 0.96) for experienced raters. Inexperienced raters did not perform as well and their ratings showed decreased repeatability ('c = 0.93), but even larger reduction in reproducibility ('c = 0.80) and accuracy ('c = 0.90). We have detected that 5.3% of ratings were outside of the 95% limits of agreement. These under- or overestimates were predominantly found for bags with intermediate levels of decay, which corresponds to the middle of the rating scale. This occurs because intermediate amounts of decay are more difficult to discriminate than extremes. The frequencies of aberrant ratings for experienced raters ranged from 0.6% to 4.4% (mean = 2.1%), for inexperienced raters the frequencies were substantially higher, ranging from 6.1% to 15.6% (mean = 9.4%). Therefore, we recommend that new raters receive training that includes practical examples in this range of decay, use of standard area diagrams, and continuing interaction with experienced raters (consultation during actual rating). Very high agreement among experienced raters indicate that visual ratings can be successfully used for evaluations of decay, until a more objective, rapid, and affordable method is developed. We recommend evaluating samples at multiple time points until 42 days after processing (about 80% decay on average) and then combining these individual ratings into the area under the decay progress stairs (AUDPS) score. Applying this approach, trained evaluators can accurately detect difference among lettuce accessions and identify lettuce cultivars with reduced decay.