A Priori Monte Carlo Simulation

An a priori Monte Carlo simulation is a special case of a Monte Carlo simulation that is used in the design of a research study, generally when analytic methods do not exist for the goal of interest for the specified model or are not convenient. A Monte Carlo simulation is generally used to evaluate empirical properties of some quantitative method by generating random data from a population with known properties, fitting a particular model to the generated data, collecting relevant information of interest, and replicating the entire procedure a large number of times (e.g., 10,000). In an a priori Monte Carlo simulation study, interest is generally in the effect of design factors on the inferences that can be made rather than a general attempt at describing the empirical properties of some quantitative method. Three common categories of design factors used in a priori Monte Carlo simulations are sample size, model misspecification, and unsatisfactory data conditions. As with Monte Carlo methods in general, the computational tediousness of a priori Monte Carlo simulation methods essentially requires one or more computers because of the large number of replications and thus the heavy computational load. Computational loads can be very great when the a priori Monte Carlo simulation is implemented for methods that are themselves computationally tedious (e.g., bootstrap, multilevel models, and Markov chain Monte Carlo methods).

For an example of when an a priori Monte Carlo simulation study would be useful, Ken Kelley and Scott Maxwell have discussed sample size planning for multiple regression when interest is in sufficiently narrow confidence intervals for standardized regression coefficients (i.e., the accuracy-in-parameter-estimation approach to sample size planning). Confidence intervals based on noncentral t distributions should be used for standardized regression coefficients. Currently, there is no analytic way to plan for the sample size so that the computed interval will be no larger than desired some specified percent of the time. However, Kelley and Maxwell suggested an a priori Monte Carlo simulation [Page 38]procedure when random data from the situation of interest are generated and a systematic search (e.g., a sequence) of different sample sizes is used until the minimum sample size is found at which the specified goal is satisfied.

As another example of when an application of a Monte Carlo simulation study would be useful, Linda Muthén and Bengt Muthén have discussed a general approach to planning appropriate sample size in a confirmatory factor analysis and structural equation modeling context by using an a priori Monte Carlo simulation study. In addition to models in which all the assumptions are satisfied, Muthén and Muthén suggested sample size planning using a priori Monte Carlo simulation methods when data are missing and when data are not normal—two conditions most sample size planning methods do not address.

Even when analytic methods do exist for designing studies, sensitivity analyses can be implemented within an a priori Monte Carlo simulation framework. Sensitivity analyses in an a priori Monte Carlo simulation study allow the effect of misspecified parameters, misspecified models, and/ or the validity of the assumptions on which the method is based to be evaluated. The generality of the a priori Monte Carlo simulation studies is its biggest advantage. As Maxwell, Kelley, and Joseph Rausch have stated, “Sample size can be planned for any research goal, on any statistical technique, in any situation with an a priori Monte Carlo simulation study” (2008, p. 553).

...