Error Analysis

Error analysis is the systematic study of uncertainty in measured and computed quantities. Every measurement and every numerical calculation carries error; the discipline quantifies how these errors arise, combine, and propagate so results can be reported with honest, defensible precision. It applies across experimental science, statistics, and numerical computation. Foundational ideas trace to Gauss and Laplace (early 1800s), with the modern numerical strand consolidated by Wilkinson in the 1960s. Error analysis is the body of methods used to characterize the discrepancy between a measured or computed value and the true (often unknowable) value of a quantity. It encompasses both the classification of errors (systematic vs. random, absolute vs. relative, rounding vs. truncation) and the quantitative machinery for propagating them through functions and algorithms. Its importance is fundamental: a number without an uncertainty is scientifically meaningless, and an algorithm without an error bound cannot be trusted. A surprising fact is that two algorithms computing the same mathematical quantity in exact arithmetic can differ by many orders of magnitude in accuracy when run in floating-point — error is a property of the method, not just the problem.

The Core Insight

Error is not a nuisance to be ignored but a measurable, propagable quantity obeying its own algebra. If you know how much uncertainty enters at each step, you can predict — often to leading order — how much exits. The central tool is the linearization of a function about the measured point: small input errors map to output errors through derivatives. The central distinction is between the conditioning of a problem (how sensitive the true answer is to input changes) and the stability of an algorithm (how much extra error the computation introduces).

Formal Statement

Absolute and Relative Error

Let $x$ be a true value and $\overset{x}{^}$ an approximation. The absolute error is

E_{abs} = \overset{x}{^} - x, ∣ E_{abs} ∣ = ∣ \overset{x}{^} - x ∣

⟳This is definitional — the signed or unsigned difference between approximation and truth.

The relative error (for $x \neq = 0$ ) is

E_{rel} = \frac{x ^ - x}{x}, ∣ E_{rel} ∣ = \frac{∣ x ^ - x ∣}{∣ x ∣}

⟳Normalize the absolute error by the magnitude of the true value to obtain a scale-independent, often percentage-expressed, quantity.

Here $x$ is the exact value, $\overset{x}{^}$ the estimate. Relative error is dimensionless; absolute error carries the units of $x$ .

Random and Systematic Error

A measurement model decomposes the observed value $\overset{x}{^}$ as

\overset{x}{^} = x + β + ε

⟳Additively separate the deterministic bias

β

(systematic) from the zero-mean fluctuation

ε

(random);

x

is the true value.

where $β$ is the systematic error (bias, constant under repetition) with $E [\overset{x}{^}] = x + β$ , and $ε$ is the random error with $E [ε] = 0$ and variance $σ^{2}$ . Random error is reduced by averaging; systematic error is not.

General Error Propagation Formula

Let $q = f (x_{1}, x_{2}, \dots, x_{n})$ be a function of measured quantities with independent random errors of standard deviations $σ_{x_{1}}, \dots, σ_{x_{n}}$ . Then to first order the standard deviation of $q$ is

σ_{q} = i = 1 \sum n (\frac{\partial f}{\partial x _{i}})^{2} σ_{x_{i}}^{2}

⟳Take the first-order Taylor expansion of

f

, compute the variance of the resulting linear combination, and use independence to kill cross terms.

If the variables are correlated, add the covariance terms:

σ_{q}^{2} = i = 1 \sum n (\frac{\partial f}{\partial x _{i}})^{2} σ_{x_{i}}^{2} + 2 i < j \sum \frac{\partial f}{\partial x _{i}} \frac{\partial f}{\partial x _{j}} σ_{x_{i} x_{j}}

⟳The variance of a sum includes covariance terms

σ_{x_{i} x_{j}} = Cov (x_{i}, x_{j})

; independence sets these to zero, recovering [4].

Why It Holds / Derivation

Derivation of the Propagation Formula

Step 1 — Linearize. Expand

f

about the true point

(x_{1}, \dots, x_{n})

, writing

δ x_{i} = \overset{x}{^}_{i} - x_{i}

for small deviations:

q \approx f (x_{1}, \dots, x_{n}) + i = 1 \sum n \frac{\partial f}{\partial x _{i}} δ x_{i}

⟳First-order multivariate Taylor expansion; second-order terms are dropped assuming errors are small relative to curvature scales.

Step 2 — Subtract the mean. Since

E [δ x_{i}] = 0

, the deviation of

q

from its mean is

δ q = \sum_{i} \frac{\partial f}{\partial x _{i}} δ x_{i}

. Step 3 — Take the variance.

σ_{q}^{2} = E (i \sum \frac{\partial f}{\partial x _{i}} δ x_{i})^{2} = i \sum j \sum \frac{\partial f}{\partial x _{i}} \frac{\partial f}{\partial x _{j}} E [δ x_{i} δ x_{j}]

⟳Expand the square of the sum into a double sum; linearity of expectation moves the constant partial derivatives outside.

Step 4 — Apply independence. For independent variables

E [δ x_{i} δ x_{j}] = σ_{x_{i}}^{2} δ_{ij}

, collapsing the double sum to the diagonal and giving [4]. Retaining off-diagonal terms gives [5].

■

Carl Friedrich Gauss | 1777–1855

Gauss developed the method of least squares and, crucially, the theory of the normal ("Gaussian") distribution as the law governing random observational errors, motivated by astronomical and geodetic measurements. His 1809 Theoria Motus and later work established that the arithmetic mean is the optimal estimator under normally distributed errors and laid the groundwork for the entire propagation-of-uncertainty apparatus. Independently, Laplace contributed the central limit theorem, explaining why errors so often follow the normal law.

Building Intuition

Think of error propagation geometrically. A function $f$ near a point is a tilted plane (its tangent); an input uncertainty is a little cloud of points. The output cloud is the input cloud passed through the tangent plane — stretched by the slope $\partial f / \partial x_{i}$ in each direction. A steep slope amplifies error; a flat slope suppresses it.

Two rules of thumb fall out immediately:

For sums and differences, absolute errors add (in quadrature). Subtracting two nearly-equal numbers is dangerous: the absolute error stays fixed while the result shrinks, so the relative error explodes — this is catastrophic cancellation.
For products and quotients, relative errors add (in quadrature). Multiplying by an exact constant scales both value and error equally, leaving relative error unchanged.

Error propagation through a function: input uncertainty maps to output uncertainty via the local slope

Worked Examples

Propagating Error Through a Product

Given. The area of a rectangle from

L = 12.0 \pm 0.1

cm and

W = 5.0 \pm 0.1

cm. Find

A = L W

and its uncertainty. Step 1 — Central value.

A = 12.0 \times 5.0 = 60.0 cm^{2}

. Step 2 — Partial derivatives.

\partial A / \partial L = W = 5.0

\partial A / \partial W = L = 12.0

. Step 3 — Apply [4].

σ_{A} = (5.0)^{2} (0.1)^{2} + (12.0)^{2} (0.1)^{2} = 0.25 + 1.44 = 1.69 = 1.3 cm^{2} .

Step 4 — Check via relative errors.

\frac{σ _{L}}{L} = 0.00833

\frac{σ _{W}}{W} = 0.02

; in quadrature

0.0083 3^{2} + 0.0 2^{2} = 0.0217

, times

A = 60

gives

1.3 cm^{2}

. ✓

A = 60.0 \pm 1.3 cm^{2}

Catastrophic Cancellation in Floating Point

Given. Compute

f (x) = \frac{1 - cos x}{x ^{2}}

x = 1 0^{- 4}

in double precision (

\approx 16

digits). True value

\to 1/2

. Step 1 — Naive evaluation.

cos (1 0^{- 4}) \approx 0.999999995

. Stored to 16 digits,

1 - cos x \approx 4.999 \dots \times 1 0^{- 9}

, but the leading 9 significant digits of

cos x

are lost in the subtraction — only

\sim 7

digits survive. Step 2 — Divide.

f \approx 4.999 \times 1 0^{- 9} /1 0^{- 8} = 0.4999 \dots

, but with corrupted trailing digits; relative error

\sim 1 0^{- 8}

instead of

\sim 1 0^{- 16}

. Step 3 — Stable reformulation. Use the identity

1 - cos x = 2 sin^{2} (x /2)

f (x) = \frac{2 sin ^{2} ( x /2 )}{x ^{2}} = \frac{1}{2} (\frac{sin ( x /2 )}{x /2})^{2} .

No subtraction of near-equal numbers; full precision retained.

f (1 0^{- 4}) = 0.5000000000 (stable form, full accuracy)

Common Misconceptions

Errors Add Linearly — They Usually Don't

A frequent mistake is to add uncertainties directly: $σ_{q} = \sum_{i} ∣ \partial f / \partial x_{i} ∣ σ_{x_{i}}$ . This is the worst-case bound, correct only for fully correlated errors. For independent random errors, uncertainties combine in quadrature (sum of squares), per [4], which is smaller. Using linear addition overestimates uncertainty; using quadrature when errors are correlated underestimates it.

More Digits ≠ More Accuracy

Reporting $A = 60.0000 cm^{2}$ when the uncertainty is $\pm 1.3$ is dishonest: the trailing digits are noise. Significant figures must match the uncertainty. A computed quantity is only as accurate as its least accurate input (for products, the largest relative error dominates).

A Stable-Looking Algorithm That Is Unstable

Consider computing the variance of data via the "one-pass" formula $s^{2} = \frac{1}{n} \sum x_{i}^{2} - \overset{x}{ˉ}^{2}$ . Mathematically exact, but if the data are large and clustered (e.g., $x_{i} \approx 1 0^{8}$ with spread $\sim 1$ ), then $\frac{1}{n} \sum x_{i}^{2}$ and $\overset{x}{ˉ}^{2}$ are huge and nearly equal — their difference suffers catastrophic cancellation and can even come out negative. The mathematically identical two-pass formula $s^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2}$ is numerically stable. Same problem, same answer in exact arithmetic, vastly different floating-point behavior.

Connections

Error analysis is the meeting point of several fields. The random-error strand is pure Probability Theory and Statistics: error propagation is just the variance of a function of random variables, and the Central Limit Theorem explains the ubiquity of the normal error law. The numerical strand connects to Floating-Point Arithmetic, Numerical Linear Algebra, and the theory of Condition Number and Numerical Stability. The optimization of estimators under error links to Least Squares and Maximum Likelihood Estimation. The Taylor-expansion core ties directly to Taylor's Theorem and Differential Calculus.

Real-World Applications

Particle physics (CERN/LHC). Discovery thresholds are stated as "5σ" — error analysis quantifies the probability that a signal is a statistical fluctuation; the Higgs boson announcement (2012) hinged on rigorous combination of systematic and statistical uncertainties.
Metrology (NIST, BIPM). The Guide to the Expression of Uncertainty in Measurement (GUM) standardizes propagation of uncertainty for all calibrated instruments worldwide.
Numerical weather prediction. Ensemble forecasting perturbs initial conditions within their uncertainty to estimate forecast error growth in chaotic systems.
Engineering tolerance stacking. Mechanical assemblies sum component tolerances in quadrature (per [4]) to predict whether parts will fit.
GPS positioning. Position error is propagated from clock, ephemeris, and atmospheric-delay uncertainties; the "dilution of precision" is a condition-number-like amplification factor.
Finite-element simulation. A posteriori error estimators bound the discretization error of PDE solutions and drive adaptive mesh refinement.
Analytical chemistry. Concentration results from spectrophotometry carry propagated uncertainties from absorbance, path length, and calibration-curve fit.

Historical Development

1632 — Galileo discusses observational errors in Dialogo, noting they cluster symmetrically about the true value and that small errors are more probable than large ones.
1755 — Thomas Simpson formally analyzes the arithmetic mean, showing averaging reduces error — an early statistical argument.
1805 — Adrien-Marie Legendre publishes the method of least squares in an appendix on comet orbits.
1809 — Gauss publishes Theoria Motus, deriving the normal distribution as the error law and justifying least squares; sparks a priority dispute with Legendre.
1810–1812 — Laplace proves the central limit theorem, explaining why summed errors tend to normality.
1900s–1920s — Development of statistical error theory by Pearson, Student (Gosset), and Fisher; the t-distribution handles small-sample uncertainty.
1947 — von Neumann and Goldstine give the first rigorous rounding-error analysis of matrix inversion.
1961–1965 — James H. Wilkinson founds modern backward error analysis; his Rounding Errors in Algebraic Processes (1963) and The Algebraic Eigenvalue Problem (1965) become canonical.
1993 — Publication of the international GUM (Guide to Uncertainty in Measurement), standardizing experimental error reporting globally.

Subtleties & Limits

The linear propagation formula [4] is only first-order: it fails when $f$ is strongly nonlinear over the scale of the uncertainties, or when partial derivatives vanish (e.g., near an extremum, where second-order terms dominate). Quantifying error for such cases requires higher Taylor terms, full distributional propagation, or Monte Carlo methods. Deeper open questions persist:

Tight average-case stability bounds. Worst-case backward-error bounds (e.g., for Gaussian elimination with partial pivoting) are often pessimistic by orders of magnitude; explaining the typical observed accuracy remains an active research area in random-matrix-based analysis.
Systematic error. There is no general theory for unknown systematic errors — they cannot be reduced by repetition and are fundamentally limited by understanding of the apparatus; their treatment is partly an art.
Interval and probabilistic arithmetic seek rigorous guaranteed enclosures, but interval methods often overestimate error severely (the "dependency problem"), and reconciling rigor with tightness is unsolved in general.