How modelers model: the overlooked social and human dimensions in model intercomparison studies

Authors: ALBANITO, FABRIZIO - University Of Aberdeen; MCBEY, DAVID - University Of Aberdeen; SMITH, PETE - University Of Aberdeen; EHRHARDT, FIONA - Inrae; HARRISON, MATTHEW - Tasmanian Institute Of Agricultural Research; BHATIA, ARTI - Indian Agricultural Research Institute; BELLOCCHI, GIANNI - Inrae; BRILLI, LORENZO - National Research Council - Italy; CAROZZI, MARCO - Inrae; CHRISTIE, KAREN - Tasmanian Institute Of Agricultural Research; Liebig, Mark

Submitted to: Environmental Science and Technology
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 9/2/2022
Publication Date: 9/2/2022
Citation: Albanito, F., Mcbey, D., Smith, P., Ehrhardt, F., Harrison, M.T., Bhatia, A., Bellocchi, G., Brilli, L., Carozzi, M., Christie, K., Liebig, M.A. 2022. How modelers model: the overlooked social and human dimensions in model intercomparison studies. Environmental Science and Technology. https://doi.org/10.1021/acs.est.2c02023.
DOI: https://doi.org/10.1021/acs.est.2c02023

Interpretive Summary: Analyzing results from multiple models has been used to research greenhouse gas emissions from agriculture and to develop mitigation options. Running different models and model versions with different sets of site conditions is a way to account for the uncertainty derived from single model simulations. However, differences in model limitations and how input data are treated make model comparisons difficult. Additionally, the complexity of model ensemble studies arises not only due to the models themselves, but also on the experience and approach used by the modelers to calibrate and validate results. There is little information on the choices made during model calibration, how many parameters are calibrated relative to the data available, and how models are validated when their outputs are compared against observed data. Given these concerns, modelers that contributed to a recent model ensemble study were surveyed. We analyzed the rationale used by the modelers where different model types were compared across five stages. Two conclusions were derived from this investigation: 1) modelers perceive datasets such as general site information, climate condition, and management practices as very important for modelling cropland and grassland systems, and 2) the framework of multi-model intercomparison studies needs to pay more attention to the structure of the models, while understanding interrelationships between different processes in the models. Moving forward, ensemble studies should include in their guidelines a quantified understanding of how data interpretations and model structures influence calibration and validation strategies.

Technical Abstract: Multi-model ensemble studies have become increasingly common to investigate a range of environmental processes and have been used to examine the impact of climate change on agroecosystems. A number of questions, however, continue to prompt discussion and debate of what exactly model ensemble studies tell us about the environmental process of interest. For agroecosystems, this includes questions about how model ensemble studies inform the current uncertainty surrounding the impact of the future climate on agriculture, and the effectiveness of climate mitigation strategies to reduce greenhouse gas emission from agriculture. There is a growing realization that the complexity of model ensemble studies is dependent not only the models used, but also on the experience and approach used by the modellers to calibrate and validate the results, which remain a source of uncertainty. In this study, we applied a multi-criteria decision-making method to investigate the rationale applied by the modellers in a model ensemble study where different model types were compared across five successive calibration stages (i.e. from partial to full calibration). Modellers shared a common understanding about the importance of the datasets, such as general site information, or data about climate and agricultural management practices measured during the experimental period, used in Stage 1 to initialize the models for calibration. Historical information about climate and management practices, used in Stage 2, were perceived as less important. The use of these input data alone, however, did not provide enough information for the modellers to obtain satisfactory ensemble simulations. In this context, by visualizing the structure of the relationships between distinct datasets used in modelling protocol, the opinion of the modellers was that datasets about general site information and climate have the capacity to influence input data on management practices, soil characteristics, and experimental data from sites. Access to these input data in the final stages of the modelling protocol allowed the improvement of the ensemble simulations. However, the importance of site-specific experimental data, used in the calibration and validation routines, depended on the experience of the modellers. The gradual access to input data across the five calibration stages may therefore have negatively influenced the consistency of the interpretations made by the modellers, with cognitive biases in “trial-and-error” model calibration routines. Future ensemble studies should include guidelines on recording how modeller data interpretations and model structures influence calibration and validation strategies.