Sensitivity Analysis

Role of a Sensitivity Analyst

A sensitivity analyst performs several diverse tasks: determining the range or distribution of each input parameter, devising hypotheses for possible system behavior and suitable experimental designs to test the hypotheses, carrying out experiments and analyzing results.

During the life cycle of a sensitivity analysis, goals change. At first, the analyst might stress test the system (typically a computer implementation of a complex mathematical model) to ensure that it behaves as expected. A suitable hypothesis would be that the system can be evaluated and that it will generate appropriate results, everywhere in the sample space. A traditional two-level fractional factorial design, using combinations of extreme parameter values, is particularly effective at rooting out anomalous behavior, exhibited as abnormal program termination or unrealistic outputs. A resolution IV design is ideal as it requires only 2N simulations (where N is the number of parameters) to ensure equal frequencies for all eight combinations of high and low values for any selection of three parameters. Where some parameters have infinite distributions (e.g., the normal distribution), a consistent policy is needed for truncation to yield extreme values.

Next, the analyst might screen parameters, hoping to identify a small group that dominate system variation. An appropriate hypothesis is that no individual parameter has any influence on the output(s) of interest. Influential parameters are those for which this hypothesis can be rejected at a high level of significance. If enough runs are performed that the effects of many parameters can be detected, influential parameters are those with the largest effects on system outputs. Initial stress testing runs might have identified parameters with large main effects. These experiments might also have shown the existence of large two-factor interactions, without uniquely identifying them. These effects, based on extreme simulations, might overwhelm less dramatic influences that play a significant role in more probable simulations. The analyst has several design choices to highlight other influential parameters.

Designs where only the value of one parameter changes at a time are particularly effective in identifying even small effects, provided that the experiments are deterministic. It is typically necessary to change the value of a single parameter several times at different locations in the sample space, as the parameter's influence might depend on the values of other parameters. The elementary effects method provides a scheme for varying one parameter at a time on a grid. For each parameter, this method computes an average of the absolute value of the output changes resulting from changing that [Page 1340]parameter alone, to provide a statistic to compare against other parameters.

...