Stochastic Processes

Processes Discrete in Time and State Space

The simplest model of a stochastic process is discrete in time and state space, that is, the state space is countable, events happen at equidistant points of time, and for each pair of subsequent states Xs = i and Xt = j, a certain probability pij(s,t) holds (Markov chain in the strict sense). It is defined by its matrix of transition probabilities:

$$M_{\left(s,t\right)}=\left(\begin{array}{ccc}{p}_{11}\left(s,t\right)& \cdots & p_{1n}\left(s,t\right)\\ ⋮& p_{ij}\left(s,t\right)& ⋮\\ p_{n1}\left(s,t\right)& \cdots & p_{nn}\left(s,t\right)\end{array}\right),{\displaystyle \sum _{j=1}^n{p}_{ij}=1}.$$

The state of the process is described as a vector containing the probabilities of the entities being in each possible state such that for two times s < t the probabilities of finding the entity in any of its possible states is

$$\begin{array}{l}\left(p_1\left(t\right)\cdots p_i\left(t\right)\cdots p_n\left(t\right)\right)\\ =\left(p_1\left(s\right)\cdots p_i\left(s\right)\cdots p_n\left(s\right)\right)\\ \left(\begin{array}{ccc}{p}_{11}\left(s,t\right)& \cdots & p_{1n}\left(s,t\right)\\ ⋮& p_{ij}\left(s,t\right)& ⋮\\ p_{n1}\left(s,t\right)& \cdots & p_{nn}\left(s,t\right)\end{array}\right).\end{array}$$

Markov chains are called homogeneous if M(s, t) = M(s+u, t+u), that is, if M is always the same for t − s = Δ. The standard case of a homogeneous Markov chain is the so-called random walk between cumulative gains and losses of a simple game tossing coins whereby head wins one unit and tail loses one unit. The states of this process are all positive and negative integer numbers with a transition matrix that contains only zeros in the main diagonal, q in the diagonal below the main diagonal, p in the diagonal above the main diagonal, and zeros elsewhere such that the cumulative gains and losses change only between neighboring states but can reach every positive and negative numbers (with a fair coin, p = q = 0.5 holds).

Stochastic Processes in Continuous Time and State Space

Random walk can be generalized to the so-called Brownian motion in continuous time and space: If the difference between neighboring states is called η, and the time span between subsequent observations Δ and the limit Δ → 0, η → 0 is taken one can show that the probability density function (PDF) of the current state of Brownian motion is the PDF of the normal distribution whose mean is the state at the beginning of the process and whose variance is proportional to the time elapsed since the process started. Moreover, the PDF of the state change in a constant time interval is the PDF of the normal distribution with mean zero and constant variance.

...