Statistical Inference, Bayesian

Basic Principles

It is well known that in classical, frequentist inference we cannot assign a probability that an unknown parameter µ takes a particular value or falls inside a particular interval. We can only talk about the probability of observing certain types of data, given a particular value of that parameter, and interpret this probability as a relative frequency. Bayesians break with this rule: They use (subjective) probabilities for expressing all kinds of statistical uncertainties, including the probability of an unknown parameter being in a particular interval.

Suppose we test a null hypothesis H0: µ ≤ 0 against an alternative hypothesis H1: µ > 0. A Bayesian assigns a prior probability distribution over µ (e.g., in the form of a probability density function). Suppose we observe some evidence E, for example, values of an observable whose probability distribution depends on the value of µ. The Bayesian calculates the posterior probability of H0 and H1 as the conditional probability of these hypotheses, given the observed evidence: pnew(H0) = p(H0 | E) and pnew(H1) = p(H1 | E). This updating principle is called Bayesian Conditionalization.

For calculating these conditional probabilities, the Bayesian statistician makes use of Bayes’s Theorem:

$$\begin{array}{c}p(H_0\mid E)=p(H_0)\frac{p(E\mid H_0)}{p\left(E\right)}\\ p(H_1\mid E)=p(H_1)\frac{p(E\mid H_1)}{p\left(E\right)}\end{array}$$

It has been argued that the posterior distribution of an unknown parameter is the real target of [Page 856]statistical inference, and Bayesian inference is the only school of statistics that can deliver it. Moreover, the posterior distribution can be used in a decision-theoretic calculus for inference and decision-making, in agreement with expected utility theory. Since the likelihoods p(E | H0) and p(E | H1) are based on the sampling distribution of the observed data under the competing hypotheses, the posterior distribution is the product of a distinctly subjective element—the prior distribution—and a more objective element, represented by the likelihoods.

Bayesian statistical inference has some important advantages over frequentist inference. Most importantly, the posterior takes the base rate of an event or hypothesis into account (i.e., p(H0) and p(H1)). Therefore, it avoids common fallacies in frequentist statistics, such as base rate neglect or the inverse probability fallacy (i.e., inferring from p(E | H) to p(H | E)). Bayesians also dispense with p values and do not need to bother about their correct interpretation.

Similar to frequentist tests, Bayesian statistical methods produce point values of parameter estimates (e.g., regression coefficients). Uncertainty around these estimates is quantified in terms of a credible interval. Point estimate and credible interval are the central tendency (mean, median, or mode) of the posterior distribution and its 95 central percentiles or 95% highest density interval, respectively. Unlike frequentist confidence intervals, credible intervals have a natural and intuitive interpretation: a 95% credible interval of parameter µ is an interval I in the parameter space such that p(μ ∊ I) = 95%. Credible intervals are also at the basis of modern Bayesian inference techniques, such as regions of highest posterior density.

...