Bayesian Inference: An Introduction

Suppose a coin is tossed ten times and lands heads eight times. What is the probability that the coin is fair? A classical statistician refuses to answer this question as posed. On that view, probability describes long-run frequencies of repeatable events. "The coin is fair" is a fixed fact, not a random variable. So it has no probability at all. A Bayesian statistician answers directly. The unknown bias is treated as a random quantity with a probability distribution. This distribution gets updated by data. This article develops that second viewpoint rigorously. It shows how a prior distribution on an unknown parameter combines with a likelihood through Bayes' rule. Together they produce a posterior distribution. It also compares this machinery, both philosophically and mathematically, to the frequentist estimation theory built around Maximum Likelihood Estimation. The idea matters because it resolves a genuine ambiguity in what "inference" is for. Frequentist procedures — point estimators, confidence intervals, pp-values — are evaluated by their behavior under hypothetical repetition of the data-generating experiment. They hold the unknown parameter fixed. Bayesian procedures instead update a degree of belief about the parameter. They treat the observed data as fixed and the parameter as the random object of interest. These are different mathematical objects answering different questions. Confusing them is one of the most common errors in applied statistics. One example is reading a 95% confidence interval as "95% probability the parameter lies here." The big picture is a single equation. Given a prior density π(θ)\pi(\theta) encoding what is believed about a parameter θ\theta before seeing data, and a likelihood L(θ∣x)L(\theta \mid x) describing how probable the observed data xx would be under each value of θ\theta, Bayes' rule produces a posterior density π(θ∣x)∝L(θ∣x) π(θ)\pi(\theta \mid x) \propto L(\theta \mid x)\,\pi(\theta). This single line of algebra, iterated over an entire course, generates estimation, credible intervals, model comparison, and prediction. All of this comes from one coherent probabilistic calculus. It does not rely on a toolbox of separately-justified procedures. This article gives the formal definition of prior, likelihood, and posterior. It states and proves the fundamental theorem. This theorem shows the posterior is a genuine probability distribution obtained by conditioning. It works two fully asymmetric numerical examples: a Beta–Bernoulli update and a Normal–Normal update with unequal variances. It shows via counterexample why an improper prior can produce a non-existent posterior. It closes with a comparison table separating Bayesian from frequentist claims. It also includes real applications in statistics, machine learning, and law.

Key takeaways
  • A prior π(θ)\pi(\theta) encodes belief about parameter θ\theta before data; the likelihood L(θ∣x)L(\theta\mid x) is the probability (or density) of the observed data as a function of θ\theta.
  • Bayes' rule turns these into a posterior π(θ∣x)∝L(θ∣x)π(θ)\pi(\theta\mid x) \propto L(\theta\mid x)\pi(\theta), a full probability distribution over θ\theta given the data.
  • The posterior is not a heuristic: it is literally the conditional distribution of θ\theta given X=xX=x in the joint model (θ,X)(\theta, X), so its validity follows from ordinary conditional probability once θ\theta is given a distribution.
  • Bayesian point estimates (posterior mean/median/mode) and credible intervals answer "what do I believe about θ\theta given this data?", unlike frequentist confidence intervals which answer "how would this procedure behave over repeated sampling?".
  • Conjugate priors (Beta–Bernoulli, Normal–Normal, Gamma–Poisson) make posteriors computable in closed form and reveal the update as a weighted combination of prior belief and data.
  • Key idea: treating the unknown parameter itself as random converts inference into ordinary conditioning, replacing the frequentist's repeated-sampling guarantees with a single, internally consistent probability statement about θ\theta given the data actually observed.

Where this lives today

Bayesian inference underlies modern hierarchical models in medicine and social science, where partial pooling across groups (patients, schools, hospitals) is naturally expressed by placing a prior on group-level parameters that itself has hyperparameters. In machine learning, Bayesian methods appear as regularization: a Gaussian prior on regression weights combined with a Gaussian likelihood yields ridge regression as the posterior mode, while a Laplace prior yields the LASSO — turning penalized optimization into a special case of MAP estimation, echoing the corollary above. Spam filters and early Bayesian classifiers apply Bayes' rule directly to word-frequency likelihoods and class priors. In forensic and legal statistics, likelihood ratios update prior odds of guilt into posterior odds given DNA or other evidence, a direct application of Bayes' rule in odds form. A/B testing in industry increasingly uses posterior probabilities that one variant beats another, replacing pp-values with direct probability statements decision-makers find easier to interpret. In physics and cosmology, Bayesian model comparison via marginal likelihoods (Bayes factors) is standard for comparing competing cosmological models against the same data, and Markov chain Monte Carlo methods — built to sample posteriors that have no closed conjugate form — are now central computational tools across Statistical Inference broadly, extending the Beta–Bernoulli and Normal–Normal calculations above to essentially arbitrary models.

:::quiz[A 95% Bayesian credible interval [a,b] for θ given observed data x formally asserts which statement?]

  • P(a ≤ θ ≤ b) = 0.95 over repeated sampling of X with θ fixed
  • P(θ ∈ [a,b] | X = x) = 0.95, a posterior probability given the observed data ✓
  • 95% of all possible datasets would produce an interval containing the true θ
  • The MLE lies in [a,b] with probability 0.95

Ask the tutor: Can you show how to derive the Normal–Normal posterior formula by completing the square in the exponent, step by step?

DefinitionBayesian statistical model, prior, likelihood, posterior

A Bayesian statistical model consists of: 1. A parameter space Θ\Theta and a random variable (or vector) Θ\Theta taking values in Θ\Theta, with a specified prior distribution having density or mass function π(θ)\pi(\theta). 2. A family of conditional distributions for data XX given Θ=θ\Theta = \theta, with density/mass function f(x∣θ)f(x \mid \theta), defined for xx in a sample space X\mathcal{X}. Given an observed value X=xX = x, the likelihood function is

L(θ∣x)=f(x∣θ),θ∈Θ,L(\theta \mid x) = f(x \mid \theta), \qquad \theta \in \Theta,

regarded as a function of θ\theta for the fixed observed xx. The posterior distribution of θ\theta given X=xX = x is the conditional density/mass function

π(θ∣x)=f(x∣θ) π(θ)m(x),m(x)=∫Θf(x∣θ) π(θ) dθ,\pi(\theta \mid x) = \frac{f(x\mid \theta)\,\pi(\theta)}{m(x)}, \qquad m(x) = \int_{\Theta} f(x \mid \theta)\,\pi(\theta)\,d\theta,

(with the integral replaced by a sum if Θ\Theta is discrete), defined for every xx with m(x)∈(0,∞)m(x) \in (0,\infty). The normalizing constant m(x)m(x) is called the marginal likelihood or prior predictive density of XX. A prior is called proper if π\pi integrates (or sums) to 11 over Θ\Theta; it is called improper if ∫Θπ(θ) dθ=∞\int_\Theta \pi(\theta)\,d\theta = \infty, in which case π\pi is not itself a probability density but the formula for π(θ∣x)\pi(\theta\mid x) may still define a valid posterior provided m(x)<∞m(x) < \infty.

Intuition

Think of the prior as a landscape of belief spread over all possible parameter values before any data arrives — flat if you are ignorant, peaked if you have strong prior information. The likelihood, as a function of θ\theta for the fixed observed data, is a second landscape: it is tall wherever the observed data would have been probable, and near zero wherever the data would have been a surprise. Bayes' rule multiplies these two landscapes together, point by point in θ\theta, and then rescales the product so that it integrates to 11. The posterior is literally the shape you get by pinching the prior toward the region where the likelihood is large, in proportion to how much data-evidence exists there. This is exactly the same operation as computing a conditional probability of an event AA given BB: restrict attention to the world where BB happened, and renormalize. Here BB is "the data equal xx," and AA ranges over all possible values of θ\theta; the posterior is the conditional law of Θ\Theta given X=xX = x in the joint distribution built from prior times likelihood. Two forces determine how much the posterior moves from the prior toward the likelihood: how much data has been collected (more data sharpens and relocates the likelihood), and how informative the prior was to begin with (a very peaked, confident prior resists being moved).

Figure
Prior, likelihood, and posterior for a Beta–Bernoulli update: prior Beta(2,2), 8 heads in 10 tosses, posterior Beta(10,4)
θ density 0 0.5 1 prior Beta(2,2) likelihood (peak ≈0.8) posterior Beta(10,4)

Contrast with frequentist estimation

A frequentist estimator, such as the maximum likelihood estimator θ^MLE=arg⁡max⁡θL(θ∣x)\hat\theta_{\mathrm{MLE}} = \arg\max_\theta L(\theta\mid x), treats θ\theta as a fixed unknown constant and studies the sampling distribution of θ^\hat\theta over hypothetical repetitions of the experiment. It never assigns a probability distribution to θ\theta itself. The Bayesian instead builds a full distribution π(θ∣x)\pi(\theta \mid x), of which the MLE and the posterior mode coincide exactly when the prior is flat (constant) — a fact proved below — showing the two frameworks are not opposed so much as nested, with the frequentist point estimate emerging as a limiting Bayesian case.

TheoremBayes' rule for statistical models, and posterior propriety

Let Θ\Theta be a random variable with prior density π\pi on Θ⊆Rk\Theta \subseteq \mathbb{R}^k, and let X∣Θ=θX \mid \Theta = \theta have conditional density f(⋅∣θ)f(\cdot \mid \theta) on X⊆Rn\mathcal{X} \subseteq \mathbb{R}^n, jointly measurable in (θ,x)(\theta, x). Suppose: (i) π\pi is a proper density, i.e. ∫Θπ(θ) dθ=1\int_\Theta \pi(\theta)\,d\theta = 1; (ii) for the observed value xx, the marginal m(x)=∫Θf(x∣θ)π(θ) dθm(x) = \int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta satisfies 0<m(x)<∞0 < m(x) < \infty. Then: 1. The pair (Θ,X)(\Theta, X) has a well-defined joint density p(θ,x)=f(x∣θ)π(θ)p(\theta, x) = f(x\mid\theta)\pi(\theta), and XX has marginal density m(x)m(x). 2. For every xx with m(x)>0m(x) > 0, the conditional density of Θ\Theta given X=xX = x exists and equals

π(θ∣x)=f(x∣θ) π(θ)m(x).\pi(\theta \mid x) = \frac{f(x\mid \theta)\,\pi(\theta)}{m(x)}.

3. π(⋅∣x)\pi(\cdot \mid x) is a genuine probability density on Θ\Theta: it is nonnegative and integrates to 11.

Proof

We use the definition of joint and conditional densities directly; this is a rigorous instance of conditioning, not an ad hoc rule.

Step 1 (joint density). Define p(θ,x):=f(x∣θ) π(θ)p(\theta, x) := f(x \mid \theta)\,\pi(\theta) for (θ,x)∈Θ×X(\theta, x) \in \Theta \times \mathcal{X}. This is nonnegative since both factors are nonnegative densities. We check it integrates to 11 over the product space, using Tonelli's theorem (all integrands are nonnegative, so order of integration is free):

∫X∫Θp(θ,x) dθ dx=∫X∫Θf(x∣θ)π(θ) dθ dx=∫Θπ(θ)(∫Xf(x∣θ) dx)dθ=∫Θπ(θ)⋅1 dθ=1,\int_{\mathcal X}\int_\Theta p(\theta, x)\, d\theta\, dx = \int_{\mathcal X}\int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta\,dx = \int_\Theta \pi(\theta) \left(\int_{\mathcal X} f(x\mid\theta)\,dx\right) d\theta = \int_\Theta \pi(\theta)\cdot 1 \, d\theta = 1,

where the inner integral is 11 because f(⋅∣θ)f(\cdot\mid\theta) is a probability density for each fixed θ\theta, and the outer integral is 11 by hypothesis (i). Hence p(θ,x)p(\theta,x) is a legitimate joint density for the pair (Θ,X)(\Theta, X), meaning we may define the joint law of (Θ,X)(\Theta,X) to have this density — this is precisely the standard construction "prior times conditional likelihood equals joint," justified because Tonelli guarantees total mass 11.

Step 2 (marginal of XX). By definition of a marginal density from a joint density,

m(x):=∫Θp(θ,x) dθ=∫Θf(x∣θ)π(θ) dθ,m(x) := \int_\Theta p(\theta,x)\,d\theta = \int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta,

which is exactly the m(x)m(x) in the statement, and mm is itself a valid density for XX since ∫Xm(x) dx=∫X∫Θp(θ,x) dθ dx=1\int_{\mathcal X} m(x)\,dx = \int_{\mathcal X}\int_\Theta p(\theta,x)\,d\theta\,dx = 1 by Step 1.

Step 3 (conditional density). For real-valued random vectors with a jointly defined density p(θ,x)p(\theta,x) and marginal m(x)m(x), the conditional density of Θ\Theta given X=xX=x is defined, whenever m(x)>0m(x) > 0, by

π(θ∣x):=p(θ,x)m(x).\pi(\theta\mid x) := \frac{p(\theta,x)}{m(x)}.

This is precisely the density-level analogue of the elementary conditional probability formula P(A∣B)=P(A∩B)/P(B)P(A\mid B) = P(A\cap B)/P(B): for any Borel set C⊆ΘC \subseteq \Theta, one can verify directly that

∫Cπ(θ∣x) dθ\int_C \pi(\theta\mid x)\,d\theta

computes, via the disintegration of the joint law, the conditional probability P(Θ∈C∣X∈[x,x+dx])P(\Theta \in C \mid X \in [x, x+dx]) in the limiting sense that underlies conditional densities. Substituting p(θ,x)=f(x∣θ)π(θ)p(\theta,x) = f(x\mid\theta)\pi(\theta) gives exactly

π(θ∣x)=f(x∣θ)π(θ)m(x),\pi(\theta\mid x) = \frac{f(x\mid\theta)\pi(\theta)}{m(x)},

which is the claimed formula. Since hypothesis (ii) guarantees 0<m(x)<∞0 < m(x) < \infty for the observed xx, this conditional density is well-defined and finite-valued (as a function of θ\theta, up to the pointwise value of f(x∣θ)π(θ)f(x\mid\theta)\pi(\theta), which is finite for a.e. θ\theta since it is integrable).

Step 4 (posterior is a probability density). Nonnegativity of π(θ∣x)\pi(\theta\mid x) is immediate since it is a ratio of nonnegative quantities with positive denominator. For total mass,

∫Θπ(θ∣x) dθ=∫Θf(x∣θ)π(θ)m(x) dθ=1m(x)∫Θf(x∣θ)π(θ) dθ=m(x)m(x)=1,\int_\Theta \pi(\theta\mid x)\,d\theta = \int_\Theta \frac{f(x\mid\theta)\pi(\theta)}{m(x)}\,d\theta = \frac{1}{m(x)}\int_\Theta f(x\mid\theta)\pi(\theta)\,d\theta = \frac{m(x)}{m(x)} = 1,

using the definition of m(x)m(x) from Step 2. Hence π(⋅∣x)\pi(\cdot\mid x) is nonnegative and integrates to 11, i.e. it is a genuine probability density on Θ\Theta. ■\blacksquare

∎
CorollaryPosterior mode reduces to MLE under a flat prior

Let π(θ)=c\pi(\theta) = c be constant on a bounded Θ\Theta (a proper "flat" prior), and suppose L(θ∣x)=f(x∣θ)L(\theta\mid x) = f(x\mid\theta) has a unique maximizer θ^MLE(x)∈Θ\hat\theta_{\mathrm{MLE}}(x) \in \Theta. Then the posterior mode

θ^MAP(x):=arg⁡max⁡θ∈Θπ(θ∣x)\hat\theta_{\mathrm{MAP}}(x) := \arg\max_{\theta \in \Theta} \pi(\theta\mid x)

equals θ^MLE(x)\hat\theta_{\mathrm{MLE}}(x).

Proof

By Bayes' rule for statistical models, and posterior propriety, π(θ∣x)=f(x∣θ) c/m(x)\pi(\theta\mid x) = f(x\mid\theta)\,c / m(x). As a function of θ\theta (with xx fixed), the factor c/m(x)c/m(x) is a positive constant not depending on θ\theta, so maximizing π(θ∣x)\pi(\theta \mid x) over θ\theta is equivalent to maximizing f(x∣θ)=L(θ∣x)f(x\mid\theta) = L(\theta\mid x) over θ\theta. By uniqueness of the maximizer of L(⋅∣x)L(\cdot \mid x), the two arg-maxes coincide. ■\blacksquare

∎

Worked examples

Example 1 — Beta–Bernoulli update with asymmetric numbers. A quality-control engineer wants to estimate the defect probability θ∈(0,1)\theta \in (0,1) of a manufacturing line. She encodes mild prior belief that defects are somewhat uncommon using a Beta(2,7)\mathrm{Beta}(2,7) prior:

π(θ)=θ2−1(1−θ)7−1B(2,7),B(2,7)=Γ(2)Γ(7)Γ(9)=1! 6!8!=72040320=156.\pi(\theta) = \frac{\theta^{2-1}(1-\theta)^{7-1}}{B(2,7)}, \qquad B(2,7) = \frac{\Gamma(2)\Gamma(7)}{\Gamma(9)} = \frac{1!\,6!}{8!} = \frac{720}{40320} = \frac{1}{56}.

The prior mean is 22+7=29≈0.222\dfrac{2}{2+7} = \dfrac{2}{9} \approx 0.222. She then inspects n=25n = 25 items independently, each defective with probability θ\theta, and observes s=4s = 4 defects. The likelihood is Binomial in form (as a function of θ\theta, up to a combinatorial constant that does not depend on θ\theta):

L(θ∣s=4)∝θ4(1−θ)21.L(\theta \mid s=4) \propto \theta^4 (1-\theta)^{21}.

By Bayes' rule,

π(θ∣s=4)∝π(θ) L(θ∣s=4)∝θ1(1−θ)6⋅θ4(1−θ)21=θ5(1−θ)27.\pi(\theta \mid s=4) \propto \pi(\theta)\,L(\theta\mid s=4) \propto \theta^{1}(1-\theta)^{6}\cdot \theta^{4}(1-\theta)^{21} = \theta^{5}(1-\theta)^{27}.

This unnormalized kernel is exactly the kernel of a Beta(6,28)\mathrm{Beta}(6, 28) density (parameters: exponent of θ\theta is 6−1=56-1=5, exponent of 1−θ1-\theta is 28−1=2728-1=27), so

θ∣s=4 ∼ Beta(6,28).\theta \mid s=4 \ \sim\ \mathrm{Beta}(6, 28).

The normalizing constant is automatically 1/B(6,28)1/B(6,28), and we did not need to compute m(x)m(x) directly because we recognized the conjugate form — this is the payoff of conjugacy. The posterior mean is

E[θ∣s=4]=66+28=634=317≈0.1765.E[\theta \mid s=4] = \frac{6}{6+28} = \frac{6}{34} = \frac{3}{17} \approx 0.1765.

Compare this to the MLE, θ^MLE=s/n=4/25=0.16\hat\theta_{\mathrm{MLE}} = s/n = 4/25 = 0.16, and to the prior mean 2/9≈0.2222/9 \approx 0.222. The posterior mean sits strictly between the prior mean and the MLE — a general feature of conjugate Beta–Bernoulli updating, since

E[θ∣s]=α+sα+β+n=(α+βα+β+n)αα+β+(nα+β+n)sn,E[\theta\mid s] = \frac{\alpha+s}{\alpha+\beta+n} = \left(\frac{\alpha+\beta}{\alpha+\beta+n}\right)\frac{\alpha}{\alpha+\beta} + \left(\frac{n}{\alpha+\beta+n}\right)\frac{s}{n},

a weighted average of the prior mean α/(α+β)=2/9\alpha/(\alpha+\beta) = 2/9 and the MLE s/n=4/25s/n = 4/25, with weights α+βα+β+n=934\frac{\alpha+\beta}{\alpha+\beta+n} = \frac{9}{34} and nα+β+n=2534\frac{n}{\alpha+\beta+n} = \frac{25}{34}. Numerically,

934⋅29+2534⋅425=234+434=634=317,\frac{9}{34}\cdot\frac{2}{9} + \frac{25}{34}\cdot\frac{4}{25} = \frac{2}{34} + \frac{4}{34} = \frac{6}{34} = \frac{3}{17},

confirming the closed form. As nn grows with s/ns/n fixed, the weight on the MLE tends to 11: the data eventually dominates the prior, regardless of its exact shape, as long as the prior places positive density near the true θ\theta. A 95% equal-tailed credible interval is obtained from the quantiles of Beta(6,28)\mathrm{Beta}(6,28); numerically this distribution has approximately [0.070, 0.317]\big[0.070,\ 0.317\big] as its 2.5% and 97.5% quantiles (computed from the Beta CDF), giving the Bayesian statement: "given the data and the stated prior, there is a 95% posterior probability that θ\theta lies in [0.070,0.317][0.070, 0.317]." This is a direct probability statement about θ\theta, unlike a frequentist confidence interval.

Example 2 — Normal–Normal update with unequal, asymmetric variances. An analyst measures the true (unknown) mean return θ\theta of an asset. Prior belief, from a separate model, is

θ∼N(μ0,τ02),μ0=3%,  τ02=4 (so τ0=2%).\theta \sim N(\mu_0, \tau_0^2), \qquad \mu_0 = 3\%,\ \ \tau_0^2 = 4\ (\text{so } \tau_0 = 2\%).

A single noisy observation XX of the return is available, modeled as

X∣θ∼N(θ,σ2),σ2=1 (so σ=1%),X \mid \theta \sim N(\theta, \sigma^2), \qquad \sigma^2 = 1\ (\text{so } \sigma = 1\%),

reflecting that the measurement is more precise than the prior (σ2<τ02\sigma^2 < \tau_0^2). Suppose X=6%X = 6\% is observed. The likelihood as a function of θ\theta is

L(θ∣x)∝exp⁡(−(x−θ)22σ2)=exp⁡(−(6−θ)22).L(\theta \mid x) \propto \exp\left(-\frac{(x-\theta)^2}{2\sigma^2}\right) = \exp\left(-\frac{(6-\theta)^2}{2}\right).

Multiplying by the prior density and completing the square in the exponent (standard conjugate Normal–Normal algebra) gives a Normal posterior

θ∣x ∼ N(μ1,τ12),\theta \mid x \ \sim\ N(\mu_1, \tau_1^2),

with

1τ12=1τ02+1σ2=14+11=54,τ12=45=0.8,\frac{1}{\tau_1^2} = \frac{1}{\tau_0^2} + \frac{1}{\sigma^2} = \frac{1}{4} + \frac{1}{1} = \frac{5}{4}, \qquad \tau_1^2 = \frac{4}{5} = 0.8,
μ1=τ12(μ0τ02+xσ2)=0.8(34+61)=0.8×6.75=5.4.\mu_1 = \tau_1^2\left(\frac{\mu_0}{\tau_0^2} + \frac{x}{\sigma^2}\right) = 0.8\left(\frac{3}{4} + \frac{6}{1}\right) = 0.8 \times 6.75 = 5.4.

So θ∣X=6∼N(5.4, 0.8)\theta \mid X = 6 \sim N(5.4,\ 0.8). Sanity checks: the posterior variance 0.80.8 is smaller than both τ02=4\tau_0^2 = 4 and σ2=1\sigma^2 = 1 — combining two independent pieces of information can only sharpen belief, never dilute it, since precisions (1/variance1/\text{variance}) add. The posterior mean 5.45.4 lies between the prior mean 33 and the data value 66, and — because the data is three times more precise than the prior (1/σ2=11/\sigma^2 = 1 vs. 1/τ02=0.251/\tau_0^2 = 0.25, a 4:14{:}1 ratio of the total precision 5/45/4) — it lies closer to 66 than to 33; indeed 5.45.4 divides [3,6][3,6] in the ratio 2.4:0.6=4:12.4:0.6 = 4:1, exactly matching the precision ratio, since

μ1=1/τ021/τ02+1/σ2μ0+1/σ21/τ02+1/σ2x=0.251.25(3)+11.25(6)=0.2(3)+0.8(6)=0.6+4.8=5.4.\mu_1 = \frac{1/\tau_0^2}{1/\tau_0^2+1/\sigma^2}\mu_0 + \frac{1/\sigma^2}{1/\tau_0^2+1/\sigma^2}x = \frac{0.25}{1.25}(3) + \frac{1}{1.25}(6) = 0.2(3) + 0.8(6) = 0.6+4.8=5.4.

A 95% credible interval is μ1±1.96τ12=5.4±1.960.8=5.4±1.753≈[3.647, 7.153]\mu_1 \pm 1.96\sqrt{\tau_1^2} = 5.4 \pm 1.96\sqrt{0.8} = 5.4 \pm 1.753 \approx [3.647,\ 7.153].

CounterexampleDropping properness of the posterior: an improper prior with m(x)=∞m(x) = \infty

The theorem requires 0<m(x)<∞0 < m(x) < \infty for the observed xx; when this hypothesis fails the "posterior" formula is meaningless as a probability. Consider estimating a Normal mean θ∈R\theta \in \mathbb{R} with a single observation X∣θ∼N(θ,1)X \mid \theta \sim N(\theta, 1), but now use the improper uniform prior on the whole real line,

π(θ)=1 for all θ∈R(so ∫−∞∞π(θ) dθ=∞).\pi(\theta) = 1 \text{ for all } \theta \in \mathbb{R} \qquad \Big(\text{so } \int_{-\infty}^{\infty}\pi(\theta)\,d\theta = \infty\Big).

This particular case is actually benign — the reader should check that m(x)=∫−∞∞ϕ(x−θ) dθ=1<∞m(x)=\int_{-\infty}^\infty \phi(x-\theta)\,d\theta=1<\infty where ϕ\phi is the standard normal density, since the Gaussian likelihood integrates over θ\theta just as it does over xx, so a proper N(x,1)N(x,1) posterior still exists even from this improper flat prior. The genuine failure appears when the tails of the likelihood in θ\theta do not decay fast enough. Take instead the (deliberately pathological, but formally admissible as a "likelihood") model

f(x∣θ)=1π(1+(x−θ)2)⋅1{∣θ∣ unbounded}f(x \mid \theta) = \frac{1}{\pi(1+(x-\theta)^2)} \cdot \mathbf{1}\{|\theta|\ \text{unbounded}\}

replaced, more cleanly, by the following standard example: let θ∈(0,∞)\theta \in (0,\infty) be a rate parameter with the improper prior

π(θ)=1θ,θ>0(∫0∞θ−1 dθ=∞),\pi(\theta) = \frac{1}{\theta}, \qquad \theta > 0 \qquad \Big(\int_0^\infty \theta^{-1}\,d\theta = \infty\Big),

often used to express "scale invariance," and suppose we observe X=0X = 0 successes from a single Exponential(θ)\mathrm{Exponential}(\theta) waiting time truncated so that only X=0X=0 is informative in a degenerate limiting sense — more directly: suppose the data model is X∣θ∼Exponential(θ)X \mid \theta \sim \mathrm{Exponential}(\theta) with density f(x∣θ)=θe−θxf(x\mid\theta) = \theta e^{-\theta x}, and we use prior π(θ)=1/θ\pi(\theta) = 1/\theta. Then

m(x)=∫0∞θe−θx⋅1θ dθ=∫0∞e−θx dθ=1x,x>0,m(x) = \int_0^\infty \theta e^{-\theta x}\cdot \frac{1}{\theta}\,d\theta = \int_0^\infty e^{-\theta x}\,d\theta = \frac{1}{x}, \quad x>0,

which is finite for every x>0x > 0 — so this particular combination is in fact fine (a proper Exponential(x)\mathrm{Exponential}(x)-type posterior in θ\theta results). To see a genuine breakdown, push the prior further: take π(θ)=1/θ2\pi(\theta) = 1/\theta^2 on θ>0\theta > 0 together with the same exponential likelihood. Now

m(x)=∫0∞θe−θx⋅1θ2 dθ=∫0∞e−θxθ dθ,m(x) = \int_0^\infty \theta e^{-\theta x} \cdot \frac{1}{\theta^2}\, d\theta = \int_0^\infty \frac{e^{-\theta x}}{\theta}\,d\theta,

and this integral diverges at θ→0+\theta \to 0^+ (the integrand behaves like 1/θ1/\theta near 00, which is not integrable) for every x>0x > 0. Hence m(x)=∞m(x) = \infty for all observed data, hypothesis (ii) of the theorem fails everywhere, and

f(x∣θ)π(θ)m(x)\frac{f(x\mid\theta)\pi(\theta)}{m(x)}

is identically 00 (or undefined, 0/∞0/\infty) rather than a probability density: no posterior exists, no matter what data is observed. This shows that properness of the marginal m(x)m(x) — not just of the model components individually — is an essential, non-removable hypothesis: improper priors are sometimes safe (as in the first sub-case above) and sometimes fatal (as here), and the difference can only be settled by explicitly checking convergence of m(x)m(x) case by case.

Scope, edge cases & common traps

SituationWhat actually holdsCommon error
Proper prior, regular likelihoodPosterior always exists and is proper (theorem above)—
Improper priorPosterior may or may not be proper; must verify m(x)<∞m(x) < \infty directlyAssuming "flat prior" is always harmless
Flat proper prior on bounded Θ\ThetaPosterior mode = MLE (corollary above)Believing MAP always equals MLE for any prior
Credible interval vs. confidence intervalCredible interval: P(θ∈I∣x)=0.95P(\theta \in I \mid x) = 0.95, a statement about θ\thetaConfidence interval reinterpreted as "95% probability θ\theta is in this fixed interval"
Large-nn behaviorUnder regularity, posterior concentrates near the true θ0\theta_0 and often resembles N(θ^MLE,Fisher-information−1)N(\hat\theta_{\mathrm{MLE}}, \text{Fisher-information}^{-1}) (Bernstein–von Mises phenomenon)Believing Bayesian and frequentist answers always differ substantially in large samples
Choice of priorGenuinely changes finite-sample answers; different analysts, same data, different posteriorsTreating the posterior as an objective, prior-free fact

The mathematician behind it

Origins · 1 minThomas Bayes and how this was discovered
TB
Thomas Bayes
c. 1701–1761 · Presbyterian minister and mathematician, Tunbridge Wells, England

Thomas Bayes was a Nonconformist minister who pursued mathematics as a private interest, becoming a Fellow of the Royal Society in 1742 largely on the strength of earlier work defending Newton's calculus. He is remembered almost entirely for one posthumously published essay addressing a single narrow question: given the number of successes in a sequence of independent trials with unknown success probability, what can be inferred about that probability? The manuscript containing what is now called Bayes' theorem was found among his papers after his death and communicated to the Royal Society in 1763 by his friend Richard Price, who edited and published it. Bayes' own treatment was geometric and tentative, restricted to a uniform prior on the unknown probability, and it was Pierre-Simon Laplace, working independently and more generally a few years later, who developed the theorem into the systematic tool of "inverse probability" that dominated statistical practice through the nineteenth century before the frequentist reaction of Fisher, Neyman, and Pearson in the twentieth.

History
- 1701 (approx.) — Thomas Bayes born in London.
- 1742 — Elected Fellow of the Royal Society.
- 1761 — Bayes dies; his unpublished essay on inverse probability is found among his papers.
- 1763 — Richard Price communicates "An Essay towards solving a Problem in the Doctrine of Chances" to the Royal Society, containing the special case of Bayes' rule for a uniform prior on a binomial probability.
- 1774 — Laplace independently derives and generalizes the rule of inverse probability, extending it well beyond Bayes' original binomial setting and applying it broadly to astronomical and demographic problems.
Is a Bayesian posterior just a subjective opinion with no mathematical content?

No. Once a prior distribution and a likelihood are specified, the posterior is derived by an exact, non-optional application of the definition of conditional probability density, as proved in the central theorem above. The subjectivity, such as it is, lies entirely in the choice of prior (and, equally, in the choice of likelihood/model, which frequentist methods must also make). Given those choices, the computation of π(θ∣x)\pi(\theta\mid x) is as rigorous as any other conditional probability calculation, with no room for interpretation in the arithmetic itself. What is genuinely subjective is which prior honestly represents a given analyst's or field's background knowledge, and reasonable analysts can disagree about that input while agreeing completely on the mathematics that follows from it.

Do different priors always lead to very different conclusions?

Not in general. As sample size grows, under mild regularity conditions (a phenomenon called the Bernstein–von Mises theorem), the posterior distribution becomes increasingly concentrated around the true parameter value and increasingly insensitive to the prior, converging in shape to a Normal distribution centered at the MLE with variance given by the inverse Fisher information — matching frequentist asymptotics. With small samples, however, the prior can dominate, and different analysts with different (defensible) priors can reach substantially different posteriors from the same data, which is precisely why prior specification is treated as a modeling decision requiring justification, not a free parameter to be tuned for convenience.

What does it mean for a prior to be "conjugate," and why does it matter?

A prior family is conjugate to a likelihood family if multiplying a prior from that family by the likelihood yields a posterior in the same family, only with updated parameters — as seen with Beta priors under Binomial/Bernoulli likelihoods, and Normal priors under Normal likelihoods with known variance. Conjugacy matters practically because it allows closed-form posterior computation without integration or simulation, since the normalizing constant is automatically known from the conjugate family's own definition. Historically, conjugate families were essential before computational methods like Markov chain Monte Carlo made it feasible to work with arbitrary, non-conjugate posteriors numerically; conjugate models remain valuable today as building blocks, sanity checks, and fast approximations.

Can Bayesian and frequentist point estimates ever be numerically identical?

Yes, in several important cases. Under a flat proper prior, the posterior mode equals the MLE exactly, as proved in the corollary above. Under a Normal prior and Normal likelihood, the posterior mean is a precision-weighted average of the prior mean and the MLE (the sample mean), which converges to the MLE as either the sample size grows or the prior variance grows without bound (an increasingly flat, uninformative prior). More generally, whenever the prior is "uninformative" relative to the amount of data, Bayesian and frequentist point estimates tend to agree closely, even though the interpretation of the resulting number — a posterior mode versus a likelihood maximizer — remains conceptually distinct.

Why does the article insist an improper prior sometimes gives a valid posterior and sometimes does not?

Because propriety of the posterior is a statement about the marginal likelihood m(x)=∫f(x∣θ)π(θ) dθm(x) = \int f(x\mid\theta)\pi(\theta)\,d\theta, not about the prior in isolation. An improper prior (one that does not integrate to a finite number over the whole parameter space) can still combine with a sufficiently fast-decaying likelihood to produce a finite m(x)m(x), as happened with the flat prior on a Normal mean in the counterexample discussion. But if the likelihood does not decay quickly enough relative to how the improper prior blows up, m(x)m(x) can diverge for every observed value of the data, and then the "posterior" formula is not a probability density at all. This is exactly why the central theorem lists 0<m(x)<∞0 < m(x) < \infty as an explicit, separately checkable hypothesis rather than assuming it follows automatically from properness of the prior.

How does Bayesian inference handle prediction of future data, as opposed to just estimating θ?

By integrating the likelihood of new data over the posterior distribution of θ\theta, rather than plugging in a single point estimate. The posterior predictive distribution for a new observation X~\tilde{X} given data xx is p(x~∣x)=∫Θf(x~∣θ) π(θ∣x) dθp(\tilde x \mid x) = \int_\Theta f(\tilde x \mid \theta)\,\pi(\theta \mid x)\,d\theta, which automatically accounts for remaining uncertainty about θ\theta rather than treating θ^\hat\theta as if it were known exactly. This tends to produce wider, more honest predictive intervals than "plug-in" frequentist prediction using θ^MLE\hat\theta_{\mathrm{MLE}} alone, particularly with small sample sizes where the posterior for θ\theta is still spread out.

Is one framework simply "correct" and the other "wrong"?

No; they are different mathematical formalizations of the word "inference," each internally consistent given its own definitions, and each with different strengths. Frequentist methods give procedures with guaranteed long-run error-rate control across repeated experiments, independent of any prior belief, which is valuable when reproducibility across many studies is the priority. Bayesian methods give direct, interpretable probability statements about the unknown parameter given the one dataset actually in hand, and integrate prior information coherently, which is valuable when genuine prior knowledge exists or when decisions must be made from a single dataset. Competent modern statistical practice uses both, chosen according to the scientific question rather than treated as rival ideologies.

Before this