KEY CRITERIA AND SELECTION OF SENSITIVITY ANALYSIS METHODS APPLIED TO NATURAL RESOURCE MODELS

Authors: Ascough Ii, James; Green, Timothy; Ma, Liwang; Ahuja, Lajpat

Submitted to: Symposium Proceedings
Publication Type: Proceedings
Publication Acceptance Date: 11/20/2005
Publication Date: 12/13/2005
Citation: Ascough Ii, J.C., Green, T.R., Ma, L., Ahuja, L.R. 2005. Key criteria and selection of sensitivity analysis methods applied to natural resource models. Modsim 2005 International Congress on Modeling and Simulation, Modeling and Simulation Society of Australia and New Zealand, Melbourne, Australia. December 12-15, 2005. p2463-2469.

Interpretive Summary: TEST

Technical Abstract: Integrated natural resource models (e.g., APSIM)are typically large and complex, thus, it can be difficult to prioritize parameters that are most promising with respect to system management goals. It is important to evaluate how a model responds to changes in its inputs as part of the process of model development, verification, and evaluation. There are several techniques for sensitivity analysis used by practitioners and analysts in numerous fields. For example, sensitivity analysis methods are commonly built-in features of a particular software tool, e.g., Crystal Ball or @Risk. However, there are other sensitivity analysis methods, including those used outside of the natural resources field, applicable to integrated system models. In this paper, we concentrate on qualitatively evaluating four sensitivity analysis methods: 1) Fourier Amplitude Sensitivity Test (FAST), 2) Response Surface Method (RSM), 3) Mutual Information Index (MII), and 4) the methods of Sobol'. For sensitivity analysis of natural resource models, the FAST and Sobol' methods are particularly attractive. These methods are capable of computing the so-called "Total Sensitivity Indices" (TSI), which measure parameter main effects and all of the interactions (of any order) involving that parameter. Additional recommendations resulting from our evaluation include: * Sensitivity analysis shuld be used prior to model development, during model development, and when the model is applied to a specific problem. * Sensitivity analysis provides useful risk insights, but alternative approaches are also needed to understand "which" parameters show up as important and "why" they show up as important. * Sensitivity analysis can be a valuable tool in building confidence in the model and in the embedded computer codes. * In spite of current advances, the state-of-the-art science has not matured to the point of quantitatively deriving significance from sensitivity analyses as input to final decision-making. * The use of global sensitivity methods is emphasized herein. Many methods currently in use have some sort of global aspect (though not explicitly recognized), in particular, variance-based sensitivity measures (e.g. FAST and Sobol') are concise and easy to understand and communicate. There is a concern that uncertainty and sensitivity analysis methods could be incorrectly used to make a case for or against a project. Therefore, there is a need to develop guidance documents (with expert involvement or endorsement) that will provide sensitivity analysis practitioners with knowledge of what is available, and the context of where the methods can be used (i.e., when to use them, and how to use them). Recent developments illustrate the tremendous need for implementing quantitative sensitivity analysis. Furthermore, a gap remains in public education of the utility and implementation of sensitivity analysis methods in the decision-making process. Issues addressed in this paper pertaining to the application of sensitivity analysis in natural resource modeling include: 1) criteria for sensitivity analysis methods applied to natural resource models, 2) identification of several promising sensitivity analysis methods for application to natural resource models, and 3) needs for implementation and demonstration of sensitivity analysis methods. As stated above, it is our goal that this paper will eventually lead to creation of a guidance document for assisting practitioners with regard to the selection of sensitivity analysis methods, and their application, interpretation, and reporting. The overall guidelines should not be too restrictive, but instead provide useful boundaries and principles for selecting, using, and interpreting results from sensitivity analysis methods.