Significance Level, Interpretation and Construction

The Need of Level of Significance in Hypothesis Testing

In hypothesis testing, it is often impossible, too costly, or too time-consuming to obtain the entire population data on any variable to determine whether a null hypothesis is true. Decisions of hypothesis testing, thus, should be made using sample data. Whenever an experiment in collecting evidence is undertaken, despite how seriously care and controls are introduced by the researcher, the outcome is always subject to some variability of chance. Hence, falsely rejecting the null hypothesis or falsely not rejecting the null hypothesis always exists. The classic way to solve this dilemma is to confine the class of tests for consideration. A conventional setting for confining the tests is assigning the level of significance. Once a test is chosen to deal with a given null hypothesis, the calculated value of the test statistic is compared with tables of critical values at specific level of significance. If the calculated value exceeds the critical value, then the null hypothesis is rejected.

Setting the Level of Significance

A significance level α = .05 means that there is a 5% chance a researcher will accept the alternative hypothesis (reject the null hypothesis) when the null hypothesis is true. Then, in the long run, the proportion of times the researcher will make a Type I error will be .05. However, the selection of the level of significance should not be affected by the results of the data; the researcher should choose it before the data have been collected. If the level α is affected by the data, a bias on stated error probabilities should be entered to the study. In fact, for avoiding a bias of the study, a complete decision process involving the selection of test statistic, the choice of significance level a, for determining one-sided or two-sided test, and the cutoff points must be set up in advance of making any observation of the data.

Size, p Value, and Significance Level

People often do not make a clear distinction between the size and significance level of a test, and sometimes these two terms are used interchangeably. By letting null hypothesis H0: θ∊Θ e T0, a clear distinction might be made by saying that a test with power function π(θ) is a size α test if supθ∊Θ0π(θ) = α and is a level α test if supθ∊Θ0π(θ) ≤ α Constructing a size α test sometimes is so difficult for computation that the researcher might turn to setting the compromise of constructing a level α test. Once the level of significance has been set as α, the rule of hypothesis testing is to reject the null hypothesis when its size is smaller than α.

...