Multivariate Normal Distribution

Probability Density Function

If $X={\left(X_1,\dots ,X_n\right)}^{\prime }$ is a multivariate normal random vector, denoted $X~N(\mu ,\Sigma )$ or$X~N_n\left(\mu ,\Sigma \right)$, then its density is given by

$$f_{\text{X}}\left(x\right)=\frac{1}{{\left|\Sigma \right|}^{1/12}{\left(2\pi \right)}^{n/2}}\exp \left\{-\frac{1}{2}\left({\left(x-\mu \right)}^{\prime }\stackrel{-1}{\Sigma }\left(x-\mu \right)\right)\right\},$$

where $\mu ={\left({\mu }_1,\dots ,{\mu }_n\right)}^{\prime }=E(X)$ is a vector whose components are the expectations $E(X_1),\dots ,E(X_n)$and S is the nonsingular variance–covariance matrix (n × n) whose diagonal terms are variances and off-diagonal terms are covariances:

$$\begin{array}{l}\Sigma =V(X)=\left[E\left(X-\mu \right)\left(X-\{\mu \right)\prime \right]\\ =\left[\begin{array}{cccc}{\sigma }_1^2& {\sigma }_{12}& \dots & {\sigma }_{1n}\\ {\sigma }_{21}& {\sigma }_2^2& \dots & {\sigma }_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{n1}& {\sigma }_{n2}& \dots & {\sigma }_n^2\end{array}\right].\end{array}$$

Note that the covariance matrix Σ is symmetric and positive definite. The (i,j)th element is given by ${\sigma }_{ij}=E\left[\left(X_i-{\mu }_i\right)\left(X_j-{\mu }_j\right)\right]$ and ${\sigma }_i^2\equiv {\sigma }_{ii}=V(X_i)$.

An important special case of the multivariate normal distribution is the bivariate normal. If $\left(\begin{array}{c}{X}_1\\ X_2\end{array}\right)~N_2\left(\mu ,\Sigma \right),$ where $\mu =\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right]\Sigma =\left[\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right]$ and $\rho \equiv Corr\left(X_1,X_2\right)=\frac{{\sigma }_{12}}{{\sigma }_1{\sigma }_2}$, then the bivariate density is given by

$$\begin{array}{l}{f}_{X_1,X_2}\left(x,y\right)=\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{\left(1-{\rho }^2\right)}}\\ \exp \left\{-\frac{{\left(\frac{x-{\mu }_1}{{\sigma }_1}\right)}^2-2\rho \left(\frac{x-{\mu }_1}{{\sigma }_1}\right)\left(\frac{y-{\mu }_2}{{\sigma }_2}\right)+{\left(\frac{y-{\mu }_2}{{\sigma }_2}\right)}^2}{2\left(1-{\rho }^2\right)}\right\}.\end{array}$$

Let $X={\left(X_1,X_2\right)}^{\prime }$; the joint density can be rewritten in matrix notation as

$$f_{\text{X}}\left(x\right)=\frac{1}{2\pi {\left|\Sigma \right|}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2}\left({\left(x-\mu \right)}^{\prime }\stackrel{-1}{\Sigma }\left(x-\mu \right)\right)\right\}.$$

Multivariate Normal Density Contours

[Page 997]The contour levels of fX(x), that is, the set of points in ${ℝ}^n$ for which fX(x) is constant, satisfy

$${\left(x-\mu \right)}^{\prime }\stackrel{-1}{\Sigma }\left(x-\mu \right)=c^2.$$

These surfaces are n-dimensional ellipsoids centered at $\mu$, whose axes of symmetry are given by the principal components (the eigenvectors) of Σ. Specifically, the length of the ellipsoid along the ith axis is $c\sqrt{{\lambda }_i}$, where λi is the ith eigenvalue associated with the eigenvector ei (recall that eigenvectors ei and eigenvalues λi are solutions to $\Sigma e_i={\lambda }_i{e}_i$ for i=1, . . . ,n).

Some Basic Properties

The following list presents some important properties involving the multivariate normal distribution.

• 1.
  The first two moments of a multivariate normal distribution, namely $\mu$ and ,Σ completely characterize the distribution. In other words, if X and Y are both multivariate normal with the same first two moments, then they are similarly distributed.
• 2.
  Let $X={\left(X_1,\dots ,X_n\right)}^{\prime }$ be a multivariate normal random vector with mean μ and covariance matrix Σ, and let . The linear combination $Y={\alpha }^{\prime }X={\alpha }_1{X}_1+\cdots +{\alpha }_n{X}_n$ is normal with mean $E(Y)={\alpha }^{\prime }\mu$ and variance $V(Y)={\alpha }^{\prime }\Sigma \alpha ={\displaystyle \sum _{i=1}^n}{\alpha }_i^{2}V(X_i)+{\displaystyle \sum _{i\ne j}}{\displaystyle \sum }{\alpha }_i{\alpha }_{j}Cov(X_i,X_j).$ Also if ${\alpha }^{\prime }X$ is normal with mean ${\alpha }^{\prime }\mu$ and variance ${\alpha }^{\prime }\Sigma \alpha$ for all possible α, then X must be a multivariate normal random vector with mean μ and covariance matrix $\Sigma \left(X~N_n\left(\mu ,\Sigma \right)\right)$.
• 3.
  More generally, let $X={\left(X_1,\dots ,X_n\right)}^{\prime }$ be a multivariate normal random vector with mean μ and covariance matrix Σ, and let $\text{A}\in {ℝ}^{m\times n}$ be a full rank matrix with m £ n, the set of linear combinations $Y={\left(Y_1,\dots ,Y_m\right)}^{\prime }=AX$ is multivariate normally distributed with mean Aμ and covariance matrix $A\Sigma A^{\prime }$. Also, if $Y=AX+b$ where b is a m × 1 vector of constants, then Y is multivariate normally distributed with mean Aμ + b and covariance matrix $A\Sigma A^{\prime }$.
• 4.
  If Xi and Yj are jointly normally distributed, then they are independent if and only if Cov(Yi Yj) = 0. Note that it is not necessarily true that uncorrelated univariate normal random variables are independent. Indeed, two random variables that are marginally normally distributed may fail to be jointly normally distributed.
• 5.
  Let $Z=(Z_1,\dots ,Z_n{)}^{\prime }$ where $Z_i\stackrel{i.i.d.}{~}N\left(0,\text{1}\right)$ (where i.i.d. = independent and identically distributed). Z is said to be standard multivariate normal, denoted Z ∼ N(0, In), and it can be shown that E [Z] = 0 and V(Z) = In, where In denotes the unit matrix of order n. The joint density of vector Z is given by

  $$f_Z(z)={\displaystyle \prod _{i=1}^n{f}_{Z_i}(z_i)={(2\pi )}^{-n/2}}\exp \left\{-\frac{1}{2}{z}^{\prime }z\right\}.$$

  ...