Markov Processes and the Markov Property

A Markov process is a stochastic process with no memory: given the present state, the future is statistically independent of the entire past. A hopping particle whose next jump depends only on where it is — never on the route it took — is Markovian. This one property, deceptively plain, is the load-bearing beam of nonequilibrium statistical mechanics and of essentially all stochastic simulation, because it collapses the intractable infinite hierarchy of a process's joint distributions onto just two objects: an initial distribution $P(x_0)$ and a single-step transition kernel $P(x'\mid x)$ . The Markov property is best read as a property of the description, not of nature. Microscopic Hamiltonian dynamics is deterministic and trivially Markovian in the full phase-space point. Project onto a few slow collective variables and the reduced dynamics is generically non-Markovian — memory kernels appear, exactly the generalized Langevin equation. Markovianity of the reduced description is recovered when a clean separation of timescales lets the fast (bath) variables relax to local equilibrium before the slow ones move. The consistency of transition probabilities under composition is the Chapman–Kolmogorov equation. Its infinitesimal-time limit for jumps is the master equation, and for diffusions the Fokker–Planck equation; its discrete-time, finite-state version is a Markov chain governed by a stochastic matrix, whose long-time fate — a unique stationary distribution reached at a rate set by the spectral gap — follows from the Perron–Frobenius theorem. Computational physics then inverts the logic: rather than derive a Markov process, one engineers a chain whose stationary distribution is a prescribed Boltzmann target $\pi(x)\propto e^{-\beta E(x)}$ . That inversion is Markov chain Monte Carlo, the practical workhorse for sampling high-dimensional Gibbs measures.

Key takeaways

A process is Markovian when the future depends on the past only through the present: $P(x_{n+1}\mid x_n,\dots,x_0)=P(x_{n+1}\mid x_n)$ .
Memorylessness collapses the full joint distribution to $P(x_0)$ times a product of one-step kernels — the entire economy of the subject.
The Chapman–Kolmogorov equation is the consistency condition for transition probabilities; it is equivalent to the Markov property, and its short-time limit is the master equation (recalled by name here).
In continuous time, memorylessness forces exponentially distributed waiting times — the exponential is the unique memoryless continuous law — which is why master-equation kinetics use constant rates.
A finite, irreducible, aperiodic chain has a unique stationary distribution reached at rate set by the spectral gap $1-|\lambda_2|$ ; critical slowing down is that gap closing.
Key idea: memorylessness collapses an infinite hierarchy of joint distributions onto a single transition kernel — specify $P(x_0)$ and $P(x'\mid x)$ and every multi-time correlation is fixed. Markovianity lives in the chosen variables: enlarge the state to absorb memory, coarse-grain and you may lose it.

The core insight

Memorylessness collapses an infinite hierarchy of joint distributions onto a single transition kernel: give $P(x_0)$ and $P(x'\mid x)$ and every multi-time correlation of the process is determined. Markovianity is a property of the chosen state variables — enlarge the state to absorb memory, coarse-grain and you may lose it.

Where this lives today

The Markov property is the silent assumption behind almost every stochastic algorithm in physics. Markov chain Monte Carlo — Metropolis–Hastings, heat-bath, and cluster updates — draws samples from Boltzmann and Gibbs measures by engineering a chain whose stationary distribution is the target; its efficiency is governed entirely by the spectral gap, and critical slowing down near phase transitions is that gap closing. Kinetic Monte Carlo and the Gillespie algorithm exploit the exponential-waiting-time result to generate exact continuous-time trajectories of chemical and surface kinetics, indispensable when molecule numbers are small (gene expression, single-cell biology). And the failure of Markovianity under coarse-graining is a modern research theme in its own right: extracting valid few-state Markov-state models of protein folding from molecular-dynamics trajectories requires choosing states and lag times so that memory has decayed — a direct, quantitative use of the Chapman–Kolmogorov test.

The physical picture

Picture a particle hopping between sites. If, to predict its next move, you only need to know where it is — not how it got there — the process is Markov. The present is a sufficient statistic for the future; the past acts on the future only through the present. This is the entire content of the property, and everything else is consequence. The deepest consequence is timing. Memorylessness in continuous time forces the waiting time in any state to be exponential, because the exponential is the one continuous distribution for which "how long you have already waited" carries no information: $P(T>t+s\mid T>s)=P(T>t)$ . A memoryless process therefore cannot have a preferred residence time — it leaves at a constant rate, and that is precisely why the master equation uses rates rather than probabilities. Markovianity is a microscope setting, not an absolute. Watch a heavy Brownian particle stroboscopically on a timescale long compared to molecular collisions but short compared to its own drift, and successive displacements look independent — Markovian. Watch faster and the collisional correlations (velocity memory) reappear; the description must then be enlarged to include the velocity or the bath. Position alone is non-Markovian for an underdamped particle; the pair (position, velocity) is Markovian again. The figures below make both faces concrete: the Chapman–Kolmogorov composition that defines a Markov process, and the memory-carrying path that violates it.

Figure

Chapman–Kolmogorov: the transition from t₁ to t₃ equals the sum over the intermediate state x₂ — memorylessness as a composition law

Figure

Punchline: a Markov path forgets its history at t₂, but a memory-carrying path does not — the CK identity fails for the non-Markov process

The core equations

A process is Markovian when every conditional distribution collapses onto the immediately preceding state. This single algebraic statement is the seed from which the trajectory factorization, the Chapman–Kolmogorov consistency law, and (in the short-time limit) the master equation all grow.

P(x_{n+1}\mid x_n,x_{n-1},\dots,x_0)=P(x_{n+1}\mid x_n) \qquad (1)

where: $x_n$ — state at discrete step $n$ ; $P(\cdot\mid\cdot)$ — conditional probability of the future given the indicated past. Read it as: the present screens off the past. Applying the chain rule of probability and then this property, the full joint distribution of a trajectory factorizes into initial data times a product of one-step kernels — the defining economy of the subject.

P(x_n,\dots,x_0)=P(x_0)\prod_{k=1}^{n}P(x_k\mid x_{k-1}) \qquad (2)

where: $P(x_n,\dots,x_0)$ — the full $(n{+}1)$ -time joint probability; $P(x_k\mid x_{k-1})$ — the one-step transition kernel. Every multi-time correlation function is now determined by $P(x_0)$ and this kernel. The consistency condition that transition probabilities must satisfy under composition over successive intervals is the Chapman–Kolmogorov equation. For a continuous state space it reads:

P(x_3,t_3\mid x_1,t_1)=\int \mathrm{d}x_2\;P(x_3,t_3\mid x_2,t_2)\,P(x_2,t_2\mid x_1,t_1) \qquad (3)

where: $P(x',t'\mid x,t)$ — transition-probability density from $x$ at $t$ to $x'$ at $t'>t$ ; $\int\mathrm{d}x_2$ — marginalization over the intermediate state at any $t_1<t_2<t_3$ . The identity holds for any choice of the middle time — that arbitrariness is exactly what memorylessness buys. Composing over an infinitesimal step converts finite transition probabilities into transition rates and yields the master equation — the differential form of Chapman–Kolmogorov. The full derivation, the rate-matrix spectrum, and the Kramers–Moyal reduction to Fokker–Planck are owned by the master equation article; here it is recalled as the short-time limit of (3).

\frac{\partial P(x,t)}{\partial t}=\sum_{x'}\big[\,W_{x\leftarrow x'}\,P(x',t)-W_{x'\leftarrow x}\,P(x,t)\,\big] \qquad (4)

where: $W_{x\leftarrow x'}$ — transition rate (probability per unit time) from $x'$ to $x$ ; the gain–loss structure and its spectral consequences are established in the master-equation article, not re-derived here. Discrete-time, finite-state Markov processes are governed by a row-stochastic transition matrix, whose long-time behavior is fixed by the Perron–Frobenius theorem: a unique stationary distribution reached at a rate set by the second eigenvalue.

p_{n+1}=p_n\,W,\quad \sum_j W_{ij}=1,\quad \pi W=\pi,\quad \tau_{\mathrm{mix}}\sim -\frac{1}{\ln|\lambda_2|} \qquad (5)

where: $W_{ij}$ — probability of $i\to j$ in one step; $\pi$ — stationary distribution; $\lambda_2$ — second-largest eigenvalue modulus; $1-|\lambda_2|$ — the spectral gap controlling convergence.

Derivation AExponential waiting times from the Markov property

The goal is to prove that memorylessness in continuous time uniquely forces the residence time in a state to be exponentially distributed — the single fact that licenses constant transition rates in the master equation and underlies the Gillespie stochastic simulation algorithm. The strategy is to translate the memoryless condition on the residence time into a functional equation for its survival probability, recognize that equation as Cauchy's multiplicative functional equation, and solve it. Watch how the Markov property enters exactly once, in the first line, and everything else is forced.

Step 1. Let $T$ be the time the system remains in a given state before its next jump and let $S(t)=P(T>t)$ be the survival probability. Memorylessness of the Markov process demands $P(T>t+s\mid T>s)=P(T>t)$ . Why: the present state carries no record of how long it has already been occupied, so the extra waiting time cannot depend on the elapsed time $s$ .

Step 2. Rewrite the conditional probability using $P(T>t+s\mid T>s)=S(t+s)/S(s)$ , so the memoryless condition becomes $S(t+s)/S(s)=S(t)$ , i.e. $S(t+s)=S(t)\,S(s)$ . Why: this is Cauchy's multiplicative functional equation — a multiplicative-in-time survival probability is the precise mathematical fingerprint of "no memory."

Step 3. The unique continuous, monotone-decreasing solution with $S(0)=1$ is $S(t)=e^{-\lambda t}$ for some $\lambda\ge0$ , so the waiting-time density is $p(\tau)=\lambda e^{-\lambda\tau}$ with mean $\langle\tau\rangle=1/\lambda$ . Why: any nonexponential residence law would encode a preferred timescale and hence memory, contradicting the Markov property.

Step 4. Identify the constant: $\lambda=\sum_{x'\neq x}W_{x'\leftarrow x}$ , the total exit rate from the state — exactly the loss term of the master equation. Why: the rate at which the system leaves is the sum of the rates of all channels out of the current state, tying the residence law directly to the master-equation rates. *Verified: verify_aq-sm-markov-processes-and-markov-property.py confirms the exponential is the unique memoryless law (survival multiplicativity), $p(\tau)=\lambda e^{-\lambda\tau}$ , and $\langle\tau\rangle=1/\lambda$ numerically.*

Go deeper · Derivation B · 1 minChapman–Kolmogorov is equivalent to the Markov property, and the semigroup it generates

Derivation BChapman–Kolmogorov is equivalent to the Markov property, and the semigroup it generates

The goal is to show that the Chapman–Kolmogorov composition law is not an extra assumption bolted onto a Markov process but a restatement of memorylessness, and that in the time-homogeneous case the transition operators form a one-parameter semigroup $T_{t+s}=T_t T_s$ . The strategy is to derive Chapman–Kolmogorov from the Markov property by marginalizing a three-time joint distribution, then read off the semigroup law and its matrix realization for a finite chain. The subtlety to keep in view: Chapman–Kolmogorov is necessary for a Markov process but, on its own, not sufficient — memory can hide in higher-order joint distributions that a single three-time identity does not see.

Step 1. Start from the three-time joint distribution and factor it with the Markov property: $P(x_3,x_2,x_1)=P(x_3\mid x_2)\,P(x_2\mid x_1)\,P(x_1)$ . Marginalize over $x_2$ and divide by $P(x_1)$ . Why: summing out the middle state must reproduce the direct $x_1\to x_3$ transition, and the Markov factorization is what makes the middle conditional depend only on $x_2$ .

Step 2. The result is $P(x_3\mid x_1)=\sum_{x_2}P(x_3\mid x_2)P(x_2\mid x_1)$ — the Chapman–Kolmogorov equation. Conversely, if this holds for every triple of times it constrains the two-time kernels to compose consistently; the equivalence to memorylessness is at the level of the two-time transition structure. Why: composition of conditional kernels is precisely what "forget everything but the present" means for transition probabilities.

Step 3. For a time-homogeneous chain write the transition kernel over duration $t$ as an operator $T_t$ . Chapman–Kolmogorov becomes the semigroup law $T_{t+s}=T_t\,T_s$ , and for a finite state space this is matrix multiplication $W(t+s)=W(t)\,W(s)$ with $W(t)=e^{\mathcal L t}$ generated by the rate matrix $\mathcal L$ . Why: a consistent composition over successive intervals is exactly the algebraic statement that the evolution operators multiply.

Step 4. Stationarity is then an eigenvalue statement: $\pi$ is invariant iff $\pi W=\pi$ (discrete) or $\mathcal L\,\pi=0$ (continuous), i.e. $\pi$ is the left/right null structure of the generator. Why: an invariant distribution is a fixed point of the transition operator, which is the eigenvector at eigenvalue $1$ (discrete) or $0$ (continuous). *Verified: verify_aq-sm-markov-processes-and-markov-property.py checks Chapman–Kolmogorov by matrix multiplication $T(t_1)T(t_2)=T(t_1+t_2)$ for a two-state telegraph process, and the semigroup/stationarity conditions numerically.*

Limiting cases

How Markovianity emerges, breaks, and specializes across descriptions:

Case	Result	Meaning
Timescale separation (Markov limit)	$\tau_{\mathrm{fast}}\ll\tau_{\mathrm{slow}}$	Eliminated bath variables relax before the retained ones move; memory kernels collapse to $\delta$ -functions and the reduced dynamics becomes Markovian (overdamped Langevin / Fokker–Planck).
Non-Markovian memory	$m\dot v=-\int_0^t\Gamma(t-t')v(t')\,\mathrm{d}t'+\xi(t)$	The generalized Langevin equation: a finite-width kernel $\Gamma$ encodes history. Markovianity returns only as $\Gamma(t)\to2\gamma\,\delta(t)$ .
Discrete-time chain	$p_{n+1}=p_n W,\ \sum_j W_{ij}=1$	Finite state space, time-homogeneous: a stochastic matrix. Perron–Frobenius gives a unique $\pi=\pi W$ for irreducible, aperiodic $W$ .
State-space enlargement	$x\to(x,v)$ or $(x,\text{phase})$	A non-Markovian process in $x$ can often be embedded as Markovian in an enlarged state carrying the memory variables.

Worked examples

Example 1 — Chapman–Kolmogorov and the memoryless waiting time of a telegraph process. Take a two-state (telegraph) Markov process with rate $k_{\uparrow}=2.0\,\mathrm{s^{-1}}$ from state 1→2 and $k_{\downarrow}=0.5\,\mathrm{s^{-1}}$ from 2→1. The transition matrix over time $t$ is $T(t)=e^{\mathcal L t}$ with $\mathcal L=\begin{pmatrix}-k_\uparrow & k_\downarrow\\ k_\uparrow & -k_\downarrow\end{pmatrix}$ .

Step 1. Verify Chapman–Kolmogorov by composition. Compute $T(0.3)$ and $T(0.4)$ from the matrix exponential and check $T(0.3)\,T(0.4)=T(0.7)$ . Numerically the product reproduces $T(0.7)$ to machine precision — the semigroup law holds. This confirms the process composes consistently: the past is forgotten at the intermediate time.

Step 2. Compute the memoryless residence time in state 1. The exit rate is $\lambda_1=k_\uparrow=2.0\,\mathrm{s^{-1}}$ , so the waiting-time density is $p(\tau)=2.0\,e^{-2.0\tau}$ with mean $\langle\tau\rangle=1/\lambda_1=\mathbf{0.5\ s}$ . The probability of remaining longer than $2\langle\tau\rangle=1.0\,\mathrm s$ is $e^{-2}=\mathbf{0.135}$ , independent of how long the system has already been in state 1.

Step 3. Confirm the stationary distribution as the invariant of the kernel. Solving $\pi\mathcal L=0$ gives $\pi_1=k_\downarrow/(k_\uparrow+k_\downarrow)=0.5/2.5=\mathbf{0.20}$ and $\pi_2=0.80$ , and $\pi T(t)=\pi$ for all $t$ (numerically invariant). The chain forgets its initial condition and settles onto $\pi$ .

Example 2 — a two-time-slice non-Markov process violates Chapman–Kolmogorov. Consider a process on states $\{A,B\}$ whose next step depends on the previous two states (a memory-1 chain, order-2 process). Encode it as a first-order chain on the enlarged states $\{AA,AB,BA,BB\}$ ; the point is that on the original two states alone, the two-time kernels do not compose.

Step 1. Take a concrete order-2 rule: from $BB$ go to $A$ with probability $0.9$ , but from $AB$ go to $A$ with probability $0.1$ — so the probability of $A$ next depends on whether the state before the current $B$ was $A$ or $B$ . Build the reduced two-time kernel $\tilde P(x_{n+1}\mid x_n)$ by averaging over the (stationary) distribution of the hidden previous state.

Step 2. Form the reduced one-step matrix $\tilde T$ and test whether $\tilde T\cdot\tilde T$ equals the directly computed two-step kernel $\tilde T^{(2)}$ obtained by marginalizing the true order-2 dynamics over two steps. They differ: $\|\tilde T^2-\tilde T^{(2)}\|>0$ . The mismatch is the quantitative signature of broken Chapman–Kolmogorov — the reduced description is not Markovian.

Step 3. Restore Markovianity by enlarging the state. On $\{AA,AB,BA,BB\}$ the dynamics is first-order Markov and its $4\times4$ kernel composes exactly, $M(2\ \text{steps})=M^2$ . Memory was not destroyed — it was hidden; enlarging the state to carry it makes the process Markov again.

Origins · 1 minmarkov process and how this was discovered

markov process

memoryless stochastic dynamics, formalized 1906 · A Markov process is a stochastic process in which the future is conditionally independent of the past given the present. Andrey Andreyevich Markov introduced the concept in 1906 to prove that the law of large numbers holds for dependent variables — a direct response to Pavel Nekrasov's claim that statistical regularity required independence. In 1913 Markov illustrated it by counting the alternation of vowels and consonants in Pushkin's Eugene Onegin, the first empirical Markov chain. The property named after him became the organizing principle of continuous-time kinetics, diffusion theory, and — a century later — the Markov chain Monte Carlo methods that dominate computational physics.

History

1906

— Andrey Markov extends the law of large numbers to dependent variables, defining the chains that bear his name

1913

— Markov applies his chains to the vowel–consonant sequence of Pushkin's Eugene Onegin, the first empirical Markov chain

1928

— Sydney Chapman writes the composition relation for transition probabilities in kinetic theory

1931

— Andrei Kolmogorov gives the analytic theory of Markov processes, deriving the forward and backward equations

1940

— William Feller develops the rigorous semigroup theory of Markov transition functions

Going further

References. Pathria & Beale, Statistical Mechanics, 3rd ed., Ch. 15 (stochastic processes) · Kardar, Statistical Physics of Particles, Ch. 9 · Reif, Fundamentals of Statistical and Thermal Physics, Ch. 15 · Sethna, Statistical Mechanics: Entropy, Order Parameters, and Complexity, Ch. 8 · van Kampen, Stochastic Processes in Physics and Chemistry, Ch. IV–V · Gardiner, Handbook of Stochastic Methods, Ch. 3

Provenance. Definitions and equations cross-checked against Pathria & Beale Ch. 15, Kardar SPP Ch. 9, Reif Ch. 15, Sethna Ch. 8, van Kampen Ch. IV–V, and Gardiner Ch. 3. A verifier should re-derive: (i) the trajectory factorization from the chain rule plus the Markov property; (ii) the exponential waiting-time distribution from memorylessness (Cauchy functional equation); (iii) Chapman–Kolmogorov from the three-time joint distribution, and the semigroup law $T_{t+s}=T_tT_s$ ; (iv) the equivalence of stationarity to invariance of the transition kernel; (v) that the master equation is the short-time limit of Chapman–Kolmogorov (owned by the master-equation article). No empirical constants are quoted; all results are exact structural statements.

Builds on: Probability and conditional distributions · Phase Space and Liouville's Theorem · Ensembles and the Boltzmann distribution. Leads to: Chapman–Kolmogorov Equation · The Master Equation · Fokker–Planck Equation · Detailed Balance and Microscopic Reversibility · Metropolis–Hastings Algorithm · Langevin Equation and Stochastic Differential Equations.

What exactly does "memoryless" mean — surely the present state contains information about the past?

It does, but only that which is relevant to the future is retained. Memoryless means the present state is a sufficient statistic: conditioned on it, the future distribution does not change if you additionally reveal the past. The trajectory that led to the current state may have been long or short, direct or winding — none of that alters where the process goes next. Formally, $P(x_{n+1}\mid x_n,\dots,x_0)=P(x_{n+1}\mid x_n)$ . The past influences the future through the present, never around it.

How is the Markov property related to the master equation and the Fokker–Planck equation?

They are the continuous-time faces of the same idea. Take Chapman–Kolmogorov over an infinitesimal interval: to first order in $\Delta t$ the system either stays or jumps, and finite transition probabilities become transition rates. The gain–loss balance that results is the master equation. If the state variable is continuous and jumps are small, a further expansion (Kramers–Moyal, truncated at second order) gives the Fokker–Planck drift–diffusion equation. The master equation and that reduction are derived and owned in the master-equation article; here they are recalled as short-time limits of the Markov composition law.

If almost any coarse-grained description is non-Markovian, why is the Markov assumption so useful?

Because timescale separation is ubiquitous. Whenever the eliminated (fast, bath) degrees of freedom relax much faster than the retained (slow) ones, their memory kernel collapses to a delta function and the reduced dynamics is Markovian to excellent approximation — the overdamped Langevin and Smoluchowski equations are exactly this. The Markov assumption is not a claim that nature is memoryless; it is a claim about a chosen description with a clean gap in timescales. When that gap is absent, one enlarges the state (adding velocity, or a few slow bath modes) until Markovianity is restored.

What is the difference between global balance and detailed balance?

Global balance is stationarity: the net probability current into each state vanishes, $\sum_{x'}[W_{x\leftarrow x'}\pi(x')-W_{x'\leftarrow x}\pi(x)]=0$ , so $\dot\pi=0$ . Detailed balance is stronger: each individual pairwise current vanishes, $W_{x\leftarrow x'}\pi(x')=W_{x'\leftarrow x}\pi(x)$ . Detailed balance implies global balance but not conversely. A nonequilibrium steady state satisfies global balance while sustaining persistent circulating currents around cycles — a whirlpool that never drains. Detailed balance additionally encodes microscopic reversibility and is the condition Monte Carlo samplers usually impose to guarantee the correct equilibrium target.

Why does the spectral gap control convergence, and what is critical slowing down?

For a finite chain the transition matrix has largest eigenvalue $1$ (the stationary mode) and all others inside the unit circle. Writing $W^n$ in its spectral decomposition, every non-stationary mode decays as $|\lambda_k|^n$ , so the slowest is governed by the second-largest modulus $|\lambda_2|$ . The relaxation (mixing) time is $\tau\sim-1/\ln|\lambda_2|$ , and the gap $1-|\lambda_2|$ sets the convergence rate. Near a continuous phase transition the relevant eigenvalue approaches $1$ , the gap closes, and Monte Carlo autocorrelation times diverge — critical slowing down. Cluster algorithms are engineered precisely to enlarge that gap.

Can a deterministic system be a Markov process?

Yes — trivially, and instructively. Hamiltonian dynamics in the full phase-space point is a (degenerate) Markov process: the transition kernel is a delta function on the deterministic flow, and the future is fixed by the present with no randomness at all. The interesting non-Markovianity appears only when you project onto a subset of variables and average over the rest; the marginalized dynamics then generically acquires memory. So determinism and Markovianity are compatible at the microscopic level; stochasticity and memory are both artifacts of reduced description.