Monte Carlo Simulation

A Monte Carlo simulation is a methodological technique used to evaluate the empirical properties of some quantitative method by generating random data from a population with known properties, fitting a particular model to the generated data, collecting relevant information of interest, and replicating the entire procedure a large number of times (e.g., 10,000) in order to obtain properties of the fitted model under the specified condition(s). Monte Carlo simulations are generally used when analytic properties of the model under the specified conditions are not known or are unattainable. Such is often the case when no closed-form solutions exist, either theoretically or given the current state of knowledge, for the particular method under the set of conditions of interest. When analytic properties are known for a particular set of conditions, Monte Carlo simulation is unnecessary. Due to the computational tediousness of Monte Carlo methods because of the large number of calculations necessary, in practice they are essentially always implemented with one or more computers.

A Monte Carlo simulation study is a systematic investigation of the properties of some quantitative method under a variety of conditions in which a set of Monte Carlo simulations is performed. Thus, a Monte Carlo simulation study consists of the findings from applying a Monte Carlo simulation to a variety of conditions. The goal of a Monte Carlo simulation study is often to make general statements about the various properties of the quantitative method under a wide range of situations. So as to discern the properties of the quantitative method generally, and to search for inter-action effects in particular, a fully crossed factorial design is often used, and a Monte Carlo simulation is performed for each combination of the situations in the factorial design. After the data have been collected from the Monte Carlo simulation study, analysis of the data is necessary so that the properties of the quantitative procedure can be discerned. Because such a large number of replications (e.g., 10,000) are performed for each condition, the summary findings from the Monte Carlo simulations are often regarded as essentially population values, although confidence intervals for the estimates is desirable.

The general rationale of Monte Carlo simulations is to assess various properties of estimators and/or procedures that are not otherwise mathematically tractable. A special case of this is comparing the nominal and empirical values (e.g., Type I error rate, statistical power, standard error) of a quantitative method. Nominal values are those that are specified by the analyst (i.e., they represent the desired), whereas empirical values are those observed (i.e., they represent the actual) from the Monte Carlo simulation study. Ideally, the nominal and empirical values are equivalent, but this is not always the case. Verification that the nominal and empirical values are consistent can be the primary motivation for using a Monte Carlo simulation study.

As an example, under certain assumptions the standardized mean difference follows a known distribution, which in this case allows for exact analytic confidence intervals to be constructed for the population standardized mean difference. One of the assumptions on which the analytic procedure is based is that in the population, the scores within each of the two groups distribute normally. In order to evaluate the effectiveness of the (analytic) [Page 831]approach to confidence interval formation when the normality assumption is not satisfied, Ken Kelley implemented a Monte Carlo simulation study and compared the nominal and empirical confidence interval coverage rates. Kelley also compared the analytic approach to confidence interval formation using two bootstrap approaches so as to determine whether the bootstrap performed better than the analytic approach under certain types of nonnormal data. Such comparisons require Monte Carlo simulation studies because no formula-based comparisons are available as the analytic procedure is based on the normality assumption, which was (purposely) not realized in the Monte Carlo simulation study.

...