Discrete Distributions

A discrete distribution assigns probabilities to a countable set of outcomes via a probability mass function (PMF). Key result: the probabilities are nonnegative and sum to 1, and all moments are computed by summation rather than integration. Where it applies: counting problems, success/failure trials, queueing, genetics, reliability, and any phenomenon with separate, enumerable outcomes. Who/when: developed by Bernoulli, Poisson, Pascal, and Fermat (17th–19th centuries), formalized by Kolmogorov (1933).

A discrete probability distribution describes a random variable $X$ that can take only countably many values (finite or countably infinite), with each value $x$ assigned a probability $p (x) = P (X = x)$ . It matters because an enormous class of real phenomena — coin tosses, defect counts, photon arrivals, allele frequencies — are intrinsically discrete, and the entire machinery of expectation, variance, and inference rests on the simple summation $\sum_{x} p (x) = 1$ . The PMF fully determines the distribution: everything about $X$ — its mean, spread, generating functions, and tail behavior — is recoverable from $p (x)$ . A surprising fact is that a discrete random variable can have infinite expected value even when every individual outcome is finite (the St. Petersburg paradox), showing that "discrete" does not mean "well-behaved."

The Core Insight

A discrete distribution is completely specified by a list of (value, probability) pairs that are nonnegative and total 1. Every probabilistic question reduces to summing the right subset of these probabilities. Continuous distributions integrate; discrete distributions sum — but the conceptual structure (sample space, measure, expectation) is identical.

Formal Statement

Discrete Random Variable and Probability Mass Function

Let $(Ω, F, P)$ be a probability space. A random variable $X : Ω \to R$ is discrete if its image $S = {x_{1}, x_{2}, \dots}$ (the support) is a countable set. Its probability mass function (PMF) is

p_{X} (x) = P (X = x) = P ({ω \in Ω : X (ω) = x}) .

⟳the PMF is the pushforward measure

P_{X} = P \circ X^{- 1}

evaluated on the singleton

{x}

The PMF satisfies two axioms:

p_{X} (x) \geq 0 for all x, x \in S \sum p_{X} (x) = 1.

⟳nonnegativity follows from

P \geq 0

; the sum equals

P (Ω) = 1

because the events

{X = x_{i}}

are disjoint and exhaust

Ω

up to a null set, then apply countable additivity.

The cumulative distribution function (CDF) is a right-continuous step function:

F_{X} (x) = P (X \leq x) = x_{i} \leq x \sum p_{X} (x_{i}) .

⟳write

{X \leq x} = ⋃_{x_{i} \leq x} {X = x_{i}}

as a disjoint union and apply countable additivity.

Expectation, Variance, and Moments

For a discrete random variable $X$ and a function $g$ , the expected value is

E [g (X)] = x \in S \sum g (x) p_{X} (x),

provided $\sum_{x} ∣ g (x) ∣ p_{X} (x) < \infty$ (absolute convergence). In particular the mean is $μ = E [X]$ and the variance is

Var (X) = E [(X - μ)^{2}] = E [X^{2}] - (E [X])^{2} .

⟳expand

(X - μ)^{2} = X^{2} - 2 μ X + μ^{2}

, take expectation termwise using linearity, and use

E [X] = μ

The probability generating function (PGF), defined for integer-valued $X \geq 0$ , encodes the whole distribution:

G_{X} (s) = E [s^{X}] = k = 0 \sum \infty p_{X} (k) s^{k}, ∣ s ∣ \leq 1.

⟳this is the definition of expectation [4] applied to

g (X) = s^{X}

;

G_{X} (1) = 1

by normalization [2].

Moments from the PGF

If $X$ is a nonnegative integer-valued random variable with PGF $G_{X} (s)$ , then

E [X] = G_{X}^{'} (1), E [X (X - 1)] = G_{X}^{''} (1),

and consequently $Var (X) = G_{X}^{''} (1) + G_{X}^{'} (1) - [G_{X}^{'} (1)]^{2}$ .

Moments from the PGF

Step 1. Differentiate [6] term by term (valid inside the radius of convergence, which is

\geq 1

G_{X}^{'} (s) = k = 1 \sum \infty k p_{X} (k) s^{k - 1} .

Step 2. Set

s = 1

G_{X}^{'} (1) = k = 1 \sum \infty k p_{X} (k) = E [X] .

Step 3. Differentiate again:

G_{X}^{''} (s) = k = 2 \sum \infty k (k - 1) p_{X} (k) s^{k - 2}, G_{X}^{''} (1) = E [X (X - 1)] .

Step 4. Combine with the variance identity [5]:

Var (X) = E [X^{2}] - μ^{2} = (E [X (X - 1)] + E [X]) - μ^{2} = G_{X}^{''} (1) + G_{X}^{'} (1) - [G_{X}^{'} (1)]^{2} . ■

Siméon Denis Poisson | 1781–1840

Poisson introduced the distribution bearing his name in his 1837 treatise Recherches sur la probabilité des jugements en matière criminelle et en matière civile as a limiting case of the binomial when the number of trials grows large and the success probability shrinks. He was studying the rate of wrongful convictions in French courts. Ironically, the distribution gained fame much later through Ladislaus Bortkiewicz's 1898 analysis of Prussian soldiers killed by horse kicks — a textbook example of rare-event counting that vindicated Poisson's mathematics decades after his death.

Building Intuition

Picture a histogram made of vertical spikes at the support points, each spike's height equal to its probability. The constraint that "all spike heights sum to 1" is just conservation of probability: the random outcome must land somewhere. The mean is the balance point (center of mass) of these spikes treated as point masses; the variance measures how far they spread from that balance point.

The key mental shift from continuous distributions: there is no probability density and no smooth curve. $P (X = x)$ is a genuine, often nonzero, probability — unlike the continuous case where $P (X = x) = 0$ always. This is why discrete distributions have CDFs that jump by exactly $p_{X} (x)$ at each support point, like climbing a staircase.

A useful slogan: the binomial is "n fixed trials," the geometric is "trials until first success," and the Poisson is "events in continuous time." All three are discrete because you are counting integer-valued things.

PMF spikes and the corresponding step CDF for a Poisson(λ=2) distribution

Worked Examples

Binomial — Defective Components

Given: A factory produces components, each independently defective with probability

p = 0.05

. A box contains

n = 20

components. Let

X

= number of defectives.

X \sim Binomial (20, 0.05)

with PMF

p_{X} (k) = (k 20) (0.05)^{k} (0.95)^{20 - k} .

Find: (a)

P (X = 0)

, (b)

P (X \geq 2)

, (c) the mean and standard deviation. Step 1 — No defectives:

P (X = 0) = (0 20) (0.05)^{0} (0.95)^{20} = (0.95)^{20} = 0.3585.

Step 2 — Exactly one defective:

P (X = 1) = (1 20) (0.05) (0.95)^{19} = 20 \cdot 0.05 \cdot 0.3774 = 0.3774.

Step 3 — At least two (use complement):

P (X \geq 2) = 1 - P (X = 0) - P (X = 1) = 1 - 0.3585 - 0.3774 = 0.2641.

Step 4 — Moments: for Binomial,

μ = n p = 20 (0.05) = 1

defective;

Var (X) = n p (1 - p) = 1 \cdot 0.95 = 0.95

, so

σ = 0.95 = 0.975

P (X = 0) = 0.3585, P (X \geq 2) = 0.2641, μ = 1.0 defect, σ = 0.975 defects

Poisson — Call Center Arrivals

Given: A call center receives calls at an average rate of

λ = 3

calls per minute, modeled as

X \sim Poisson (3)

with PMF

p_{X} (k) = \frac{e ^{- λ} λ ^{k}}{k !} = \frac{e ^{- 3} 3 ^{k}}{k !} .

Find: (a)

P (exactly 2 calls)

, (b)

P (at least 1 call)

, (c) verify mean and variance. Step 1 — Exactly two calls:

P (X = 2) = \frac{e ^{- 3} 3 ^{2}}{2 !} = \frac{9 e ^{- 3}}{2} = \frac{9 ( 0.049787 )}{2} = 0.2240.

Step 2 — At least one (complement):

P (X \geq 1) = 1 - P (X = 0) = 1 - e^{- 3} = 1 - 0.049787 = 0.9502.

Step 3 — Mean and variance. For Poisson, the PGF is

G_{X} (s) = e^{λ (s - 1)}

. Then

G_{X}^{'} (s) = λ e^{λ (s - 1)}

, so

E [X] = G_{X}^{'} (1) = λ = 3

. Also

G_{X}^{''} (1) = λ^{2}

, giving

Var (X) = λ^{2} + λ - λ^{2} = λ = 3

. (Mean = variance is the Poisson signature.)

P (X = 2) = 0.2240, P (X \geq 1) = 0.9502, μ = σ^{2} = 3 calls/min

Common Misconceptions

PMF values can exceed... no — but densities can

A common error is confusing the PMF with a probability density. For a discrete distribution, $p_{X} (x)$ is itself a probability, so $0 \leq p_{X} (x) \leq 1$ always. This differs from continuous densities $f (x)$ , which can be larger than 1. Conversely, never write $P (X = x) = 0$ for a discrete variable at a support point — that is a continuous-distribution fact and is false here.

Convergence of the mean

The expectation $E [X] = \sum x p_{X} (x)$ requires absolute convergence. If the series only converges conditionally or diverges, the mean is undefined or infinite. Plugging values into a formula without checking can give nonsense. Always verify $\sum ∣ x ∣ p_{X} (x) < \infty$ for infinite supports.

A discrete distribution with infinite mean — St. Petersburg

Define $X$ on the support ${2, 4, 8, 16, \dots} = {2^{k} : k \geq 1}$ with $p_{X} (2^{k}) = 2^{- k}$ . This is a valid PMF: $\sum_{k = 1}^{\infty} 2^{- k} = 1$ . But the mean is

E [X] = k = 1 \sum \infty 2^{k} \cdot 2^{- k} = k = 1 \sum \infty 1 = \infty.

Every outcome is finite, the probabilities are perfectly well-defined, yet the expected value diverges. This refutes the naïve belief that "discrete = tame" and underlies the famous St. Petersburg paradox where a "fair" game has infinite expected payoff.

Connections

Discrete distributions are the countable-support special case of the general theory in Probability Theory, built on Measure Theory via pushforward measures. They contrast with Continuous Distributions, where integration over densities replaces summation over PMFs — and the two unify under Lebesgue–Stieltjes Integration applied to the CDF. The Central Limit Theorem connects them dynamically: a sum of many independent discrete variables (e.g., binomial) converges to the continuous Normal Distribution. The Poisson Distribution arises both as the rare-event limit of the Binomial Distribution (law of small numbers) and from the Poisson Process in continuous time. Generating-function techniques tie this topic to Combinatorics and the algebra of formal power series.

Real-World Applications

Quality control (Binomial): Acceptance sampling in manufacturing computes the probability that a batch passes inspection given a defect rate, as in the component example above.
Insurance and actuarial science (Poisson / Negative Binomial): Modeling the number of claims per policyholder per year; the negative binomial handles overdispersion when variance exceeds the mean.
Telecommunications and queueing (Poisson): Erlang's call-arrival models for telephone exchanges and modern packet-arrival models in network routers use Poisson arrivals.
Genetics (Binomial / Hypergeometric): Mendelian inheritance ratios and the Hardy–Weinberg framework count alleles; the hypergeometric distribution models sampling without replacement in finite gene pools.
Epidemiology (Poisson / Negative Binomial): Counting disease cases per region per week; the dispersion parameter $k$ in branching-process models quantifies superspreading.
Reliability engineering (Geometric / Negative Binomial): The number of operations until a component fails follows a geometric distribution under constant failure probability.
Particle physics (Poisson): Radioactive decay counts and photon detection in low-light imaging (shot noise) are quintessentially Poisson, with signal-to-noise scaling as $N$ .

Historical Development

1654 — Blaise Pascal and Pierre de Fermat exchange letters on the "problem of points," founding the systematic calculation of discrete probabilities for games of chance.
1657 — Christiaan Huygens publishes De ratiociniis in ludo aleae, the first printed probability text, introducing expectation.
1713 — Jacob Bernoulli's posthumous Ars Conjectandi presents the binomial distribution and proves the (weak) law of large numbers.
1718 — Abraham de Moivre's The Doctrine of Chances develops the normal approximation to the binomial — an early central limit result.
1837 — Siméon Denis Poisson derives the Poisson distribution as a binomial limit in his study of judicial decisions.
1898 — Ladislaus Bortkiewicz publishes Das Gesetz der kleinen Zahlen ("Law of Small Numbers"), popularizing the Poisson distribution with the Prussian horse-kick data.
1909 — Agner Krarup Erlang applies Poisson and related distributions to telephone-traffic engineering, founding queueing theory.
1920 — George Udny Yule and others develop the negative binomial and overdispersion modeling in biology.
1933 — Andrey Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung axiomatizes probability measure-theoretically, placing discrete distributions on rigorous countable-additivity foundations.

Subtleties & Limits

Several deep questions surround discrete distributions at and beyond the undergraduate boundary. Mixed and singular distributions: a random variable can be partly discrete and partly continuous (e.g., insurance payouts with a point mass at zero plus a continuous tail), requiring the general Lebesgue decomposition rather than a pure PMF. Heavy tails: distributions like the zeta (discrete power-law) distribution can have infinite variance or even infinite mean, complicating estimation and breaking the standard CLT; characterizing convergence for such cases (to stable laws) is subtle. Maximum-entropy characterization: which discrete distribution maximizes entropy under given moment constraints is well understood (geometric under fixed mean, etc.), but the interplay with model selection in machine learning remains an active applied area. Infinite divisibility: classifying which discrete distributions can arise as the distribution of a sum of $n$ i.i.d. parts for every $n$ (the Poisson and negative binomial qualify; the binomial does not) connects to Lévy processes and is a rich research thread.

Discrete Distributions

The Core Insight

Formal Statement

Discrete Random Variable and Probability Mass Function

Expectation, Variance, and Moments

Moments from the PGF

Moments from the PGF

Siméon Denis Poisson | 1781–1840

Building Intuition

PMF spikes and the corresponding step CDF for a Poisson(λ=2) distribution

Worked Examples

Binomial — Defective Components

Poisson — Call Center Arrivals

Common Misconceptions

PMF values can exceed... no — but densities can

Convergence of the mean

A discrete distribution with infinite mean — St. Petersburg

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also

Before this

Where this leads