A discussion and evaluation of statistical procedures used by JIMB authors when comparing means

Authors: Klasson, K Thomas

Submitted to: Journal of Industrial Microbiology and Biotechnology
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 1/8/2024
Publication Date: 1/10/2024
Citation: Klasson, K.T. 2024. A discussion and evaluation of statistical procedures used by JIMB authors when comparing means. Journal of Industrial Microbiology and Biotechnology. 51. Article kuae001. https://doi.org/10.1093/jimb/kuae001.
DOI: https://doi.org/10.1093/jimb/kuae001

Interpretive Summary: Out of the 166 articles published in Journal of Industrial Microbiology and Biotechnology (JIMB) in 2019-2020 (not including special issues or review articles), 46 of them used a statistical test to compare two or more means. The most popular test was the (Standard) t-test, which often was used to compare several pair of means. Other statistical procedures used included Fisher’s Least Significant Difference (LSD), Tukey’s Honest Significant Difference (HSD), and Welch’s t-test; and to a lesser extent Bonferroni, Duncan’s Multiple Range Test, and Student-Newman-Keuls test. This manuscript looks at the performance of some of these tests with simulated experimental data, typical of those reported by JIMB authors. The results show that many of the most common procedures used by JIMB authors result in statistical conclusions that are prone to have large false positive (Type I) errors. In simulation studies of experiments with six treatments, 17-29% of the experiments contained significant differences the between treatments (as determined by the t-test, Welch’s t test, and Fisher’s LSD) that were incorrect. These comparisons procedures were compared with other alternatives (Tukey’s HSD, Bonferroni, and Fisher-Hayter) that did not have the same issues.

Technical Abstract: Out of the 166 articles published in Journal of Industrial Microbiology and Biotechnology (JIMB) in 2019-2020 (not including special issues or review articles), 46 of them used a statistical test to compare two or more means. The most popular test was the (Standard) t-test, which often was used to compare several pair of means. Other statistical procedures used included Fisher’s Least Significant Difference (LSD), Tukey’s Honest Significant Difference (HSD), and Welch’s t-test; and to a lesser extent Bonferroni, Duncan’s Multiple Range Test, and Student-Newman-Keuls test. This manuscript examines the performance of some of these tests with simulated experimental data, typical of those reported by JIMB authors. The results show that many of the most common procedures used by JIMB authors result in statistical conclusions that are prone to have large false positive (Type I) errors. In simulation studies of experiments with six treatments, 17-29% of the experiments contained significant differences the between treatments (as determined by the t-test, Welch’s t test, and Fisher’s LSD) that were incorrect. These comparisons procedures were compared with other alternatives (Tukey’s HSD, Bonferroni, and Fisher-Hayter) that did not have the same issues.