Simulation and Resampling Methods

Classical inference — the t-test, the normal-theory confidence interval, the chi-square test — works by deriving, mathematically, what the sampling distribution of a statistic would look like under stated assumptions (normality, independence, large n, known variance). Once that theoretical distribution is known, a p-value or a confidence interval follows from a formula or a lookup table. This works beautifully for means of reasonably-sized, roughly-normal samples. It fails, or becomes an approximation of unknown quality, for medians, ratios, correlation coefficients in small samples, maximum-likelihood estimates from complex models, and statistics computed on inherently non-independent data like phylogenies.

Resampling (also called computational or simulation-based inference) sidesteps the derivation entirely. Rather than asking "what distribution does theory predict?", it asks "what distribution do I actually get if I repeatedly resample from data structured like mine?" Two flavors dominate biological practice: The bootstrap, introduced by Bradley Efron in 1979, treats the observed sample as a stand-in for the population. It draws thousands of new samples of the same size, with replacement, from the original data, computes the statistic of interest (mean, median, regression slope, phylogenetic tree, allele frequency) on each resample, and uses the resulting spread of values to estimate standard error and construct a confidence interval — all without assuming a parametric form for the population. The permutation test, whose logic traces to Ronald Fisher's exact test (1935) and R.A. Fisher's discussion of the Lady Tasting Tea experiment, addresses hypothesis testing rather than estimation. Under the null hypothesis of no association or no group difference, the labels attached to observations (treatment vs. control, species A vs. species B) carry no real information — they are exchangeable. The test recomputes the statistic (a difference in means, a correlation, a chi-square statistic) after randomly reshuffling those labels thousands of times, building an empirical null distribution against which the actually observed statistic is judged. Both methods are forms of Monte Carlo inference: repeated random simulation used to approximate a quantity too hard to derive analytically.

Step-by-step: the nonparametric bootstrap for a confidence interval 1. Collect the original sample of n independent observations, e.g., body masses of 14 salamanders: $x_1, x_2, \dots, x_{14}$. 2. Compute the statistic of interest on the real data — say the sample median, $\hat{\theta}$. 3. Generate a bootstrap resample: draw n observations from the original n, with replacement. Because sampling is with replacement, some original observations appear multiple times in a given resample and others are omitted entirely. On average, about 63.2% of the original observations appear at least once in each resample (this follows from $1-(1-1/n)^n \to 1-e^{-1}$ as n grows). 4. Compute the same statistic (the median) on this resample: $\hat{\theta}^{*}_1$. 5. Repeat steps 3–4 a large number of times, B — typically $B = 1{,}000$ to $10{,}000$ — producing a distribution of bootstrap replicate estimates $\hat{\theta}^{*}_1, \hat{\theta}^{*}_2, \dots, \hat{\theta}^{*}_B$. 6. This distribution of replicate estimates approximates the true sampling distribution of $\hat{\theta}$. Its standard deviation estimates the bootstrap standard error of the statistic. 7. For a confidence interval, take the percentile method: the 2.5th and 97.5th percentiles of the $B$ bootstrap replicates form a 95% confidence interval. More refined variants — the bias-corrected and accelerated (BCa) interval — adjust for skew and bias in the bootstrap distribution and are preferred in practice for small or skewed samples.

Step-by-step: the permutation test for a two-group comparison 1. Collect data from two groups — e.g., gene expression levels in tumor tissue (n = 6) versus normal tissue (n = 5). 2. Compute the observed test statistic, typically the difference in group means (or medians): $T_{\text{obs}} = \bar{x}_{\text{tumor}} - \bar{x}_{\text{normal}}$. 3. Pool all 11 observations together, erasing the group labels. 4. Randomly reassign 6 of the 11 values to a "pseudo-tumor" group and the remaining 5 to "pseudo-normal," respecting the original group sizes. 5. Recompute the difference in means on this relabeled data: one permutation replicate, $T^{*}_1$. 6. Repeat for all possible relabelings (for small n this can be exhaustive — $\binom{11}{6} = 462$ ways) or for a large random subset of relabelings (typically 10,000+) when the full set is too large to enumerate. 7. The resulting collection of $T^{*}$ values is the empirical null distribution — what differences in means would look like if group membership were irrelevant. 8. The p-value is the proportion of permuted statistics as extreme as, or more extreme than, the observed one:

(the "+1" corrections account for the observed data itself being one valid permutation, ensuring the p-value is never exactly zero). A crucial distinction: the bootstrap resamples with replacement from a single sample to characterize estimation uncertainty. The permutation test resamples without replacement (reshuffling labels on fixed data) to characterize what "no effect" looks like. Confusing the two — permuting to get a confidence interval, or bootstrapping to get a null p-value — produces incorrect inference, because bootstrapping does not, by itself, enforce a null hypothesis, and permutation does not preserve the marginal structure needed for interval estimation.

Bradley Efron and the invention of the bootstrap (1979). Before 1979, statisticians estimating standard errors for complicated statistics — medians, ratios, correlation coefficients — relied on the delta method or jackknife approximations, both of which could be inaccurate for small or skewed samples. Efron's 1979 paper, "Bootstrap Methods: Another Look at the Jackknife," proved that resampling the observed data with replacement, and recomputing the statistic, produces an estimator of the sampling distribution that converges to the truth as sample size grows, under quite general conditions. He demonstrated this first on simulated data where the true answer was known by construction, showing the bootstrap standard error tracked the true theoretical standard error closely — including in cases (like the median) where classical formulas were unreliable or unavailable. Subsequent theoretical work (Efron & Tibshirani, 1993; Peter Hall's asymptotic theory in the 1980s) formalized when the bootstrap works: it requires the original sample to be reasonably representative and, technically, that the statistic be "smooth" enough (some statistics, like the maximum of a sample, are known to behave poorly under the naive bootstrap). The bootstrap was rapidly validated across fields by comparing bootstrap confidence intervals against known-truth simulations and against exact intervals in cases where exact solutions existed, and the two matched closely — the core evidence that a computational shortcut can substitute rigorously for calculus.

Felsenstein and the phylogenetic bootstrap (1985). Joseph Felsenstein's 1985 paper, "Confidence Limits on Phylogenies: An Approach Using the Bootstrap," is one of the most cited applications in all of biology. The problem: given an alignment of DNA or protein sequence characters, a phylogenetic tree-building algorithm produces one estimated tree — but how confident should biologists be in any particular branch (clade)? No classical formula existed, because tree-building is a complex, non-linear, discrete optimization over a huge space of possible trees. Felsenstein's solution: resample the alignment columns (sites) with replacement to create pseudo-replicate alignments of the same length, rebuild the tree for each pseudo-replicate, and record what fraction of the resulting bootstrap trees contain each clade found in the original tree. A clade appearing in, say, 950 of 1,000 bootstrap replicate trees is reported with 95% bootstrap support — the now-ubiquitous number seen on published phylogenies. This did not derive from an assumption of normality; it built the tree-uncertainty distribution empirically from the same sequence data that produced the original tree, and was validated by simulation studies (Hillis & Bull, 1993) showing that bootstrap support values correlate with the actual probability that a clade is correctly reconstructed, though the correspondence is conservative (a 70% bootstrap value often corresponds to genuine support in more than 70% of true replications, an issue explored extensively afterward).

Fisher's exact test and the logic of permutation (1935). Ronald Fisher's book The Design of Experiments (1935) introduced the Lady Tasting Tea experiment as a paradigm case for exact permutation-based reasoning. A woman claimed she could tell, by taste, whether milk or tea was poured into a cup first. Fisher designed a test: present her with 8 cups, 4 prepared each way, in random order, and ask her to identify the 4 milk-first cups. Under the null hypothesis that she has no discriminating ability, the number of correct identifications follows a distribution derivable by exhaustively enumerating all $\binom{8}{4} = 70$ ways the cups could be labeled. If she correctly identifies all 4 milk-first cups, the exact probability of doing so by chance alone is $1/70 \approx 0.014$ (using the hypergeometric distribution, itself a permutation argument) — a testable, falsifiable prediction under pure chance. This is the direct ancestor of the modern permutation test: instead of assuming a parametric distribution, enumerate (or simulate) all the ways labels could have been assigned under the null, and see how extreme the observed outcome is relative to that reference set. Later, R.A. Fisher and, more explicitly, E.J.G. Pitman (1937) generalized this logic into permutation tests for continuous data, providing exact, assumption-light p-values — validated because, unlike the t-test, the permutation p-value is exact for the specific sample at hand under the null of exchangeability, not merely asymptotically correct.

Example 1 — Bootstrap confidence interval for a skewed ecological trait. An ecologist samples clutch sizes (number of eggs) from 10 nests of an endangered bird species, obtaining the following, deliberately asymmetric data (one very large clutch skews the distribution):

The sample mean is $\bar{x} = 5.0$ eggs, but the distribution is strongly right-skewed by the outlier clutch of 14 — a t-based confidence interval assuming normality would be distorted by this skew and could even include biologically impossible negative clutch sizes at the lower bound. Instead, bootstrap the mean: 1. Draw a resample of 10 values, with replacement, from the 10 observations above. One such resample might be: $3, 3, 4, 4, 4, 5, 5, 6, 6, 14$ → mean* = 5.4. 2. Repeat this 10,000 times, recording the resample mean each time. 3. Sort the 10,000 bootstrap means and read off the 2.5th and 97.5th percentiles. A representative outcome (values are illustrative of the typical pattern for this dataset):

Notice the bootstrap interval is asymmetric around 5.0 (closer to 3.4 below, further to 7.5 above) — correctly reflecting that the outlier of 14 pulls the upper tail of the sampling distribution out further than the lower tail, something the symmetric normal-theory interval cannot capture and which even permits a biologically implausible near-zero lower bound. This asymmetry, automatically produced by resampling the actual skewed data, is the chief practical advantage of the bootstrap over classical formulas for small, skewed biological samples.

Example 2 — Permutation test for a small-sample genetic association. A researcher studies whether a candidate SNP genotype is associated with resistance to a fungal pathogen in a small greenhouse trial. Because of limited seed stock, sample sizes are small and unequal:

• Resistant genotype (AA), n = 5 plants: disease severity scores (0–10 scale) = 1, 2, 2, 3, 1
• Susceptible genotype (aa), n = 8 plants: disease severity scores = 4, 5, 6, 3, 7, 5, 4, 6

Observed group means: $\bar{x}_{AA} = 1.8$, $\bar{x}_{aa} = 5.0$, so $T_{\text{obs}} = 5.0 - 1.8 = 3.2$. With n = 5 and n = 8 (total 13), a t-test would require assuming approximate normality within each tiny group — risky with only 5 observations in one arm, and disease severity scores are bounded, discrete, and plausibly non-normal. Permutation approach: 1. Pool all 13 severity scores. 2. Randomly assign 5 of the 13 to a "pseudo-AA" group and the remaining 8 to "pseudo-aa," matching the original group sizes. 3. Recompute the difference in means for this relabeling. 4. Repeat for all $\binom{13}{5} = 1{,}287$ possible relabelings (small enough to enumerate exactly) or a large random sample of them. 5. Count how many of the 1,287 relabelings produce a difference in means $\geq 3.2$ (one-tailed, since resistance is hypothesized to reduce severity). Suppose only 6 of the 1,287 possible relabelings produce a difference this large or larger. Then:

This is a valid, exact p-value for this specific sample — it required no assumption about the shape of the severity-score distribution, only that under the null hypothesis of no genotype effect, any of the 1,287 ways of splitting the 13 scores into groups of 5 and 8 were equally likely. This is exactly the situation permutation testing was built for: small, non-normal, unbalanced groups where classical test assumptions are shaky but the logic of random assignment is airtight.

Bradley Efron, born in 1938, trained as a statistician at Stanford University, where he has spent most of his career. Working alongside biostatisticians and genomicists, Efron had a long-standing interest in problems where classical asymptotic theory gave unsatisfying answers for real, finite samples — exactly the regime biologists face constantly with modest sample sizes of organisms, patients, or specimens. His 1979 bootstrap paper generalized an existing idea (the jackknife, developed by Maurice Quenouille and John Tukey in the 1950s, which drops one observation at a time rather than resampling with replacement) into something far more flexible and broadly applicable. Efron was explicit that the bootstrap was made practical only by the arrival of accessible computing power — the method is fundamentally computational, not a pen-and-paper technique, and its rise tracks the rise of cheap, fast computers through the 1980s and 1990s. Efron later collaborated extensively on applying resampling and empirical Bayes methods to genomics and large-scale hypothesis testing, cementing the bootstrap's place at the center of quantitative biology.

Phylogenetics and molecular evolution. Every published gene tree or species tree today reports bootstrap support values (or the related Bayesian posterior probabilities) on its branches, directly following Felsenstein's 1985 method, guiding decisions about which evolutionary relationships are well-supported by the sequence data versus which remain uncertain.

Genome-wide association studies (GWAS) and multiple testing. Testing millions of genetic markers for association with a trait creates a massive multiple-testing burden. Permutation testing is used to empirically determine genome-wide significance thresholds by repeatedly permuting phenotype labels across the actual genotype data, capturing the correlation structure (linkage disequilibrium) among markers that theoretical Bonferroni corrections handle poorly.

RNA-seq and differential gene expression. With often only 3–6 biological replicates per condition, permutation-based and bootstrap-based methods (and their refinements in tools like SAM — Significance Analysis of Microarrays) are used to assess whether observed expression differences for thousands of genes simultaneously exceed what reshuffled labels would produce by chance, controlling the false discovery rate.

Conservation biology and small-population ecology. Endangered species studies routinely have tiny sample sizes (a handful of individuals, nests, or populations); bootstrap confidence intervals for parameters like genetic diversity indices, population growth rate, or trait means are standard because normal-theory approximations are unreliable at n < 20.

Ecological community analysis. Permutation-based multivariate methods (PERMANOVA, Mantel tests) test whether community composition differs between habitats or whether genetic distance correlates with geographic distance, explicitly built around resampling because ecological community data (counts, presence-absence matrices) violate the assumptions of classical multivariate normal theory.

Clinical and epidemiological studies. Small clinical trials, rare-disease cohorts, and case-control studies with modest sample sizes use bootstrap confidence intervals for odds ratios, hazard ratios, and other non-normally-distributed estimators, and permutation tests when treatment assignment is the only random element that can be reshuffled. Both techniques form an essential bridge to more modern computational inference: the same simulation logic — generate many plausible datasets, observe how a quantity behaves across them — underlies Markov chain Monte Carlo (MCMC) sampling in Bayesian Inference in Biology: An Introduction, where instead of resampling data, one samples from a posterior distribution over parameters. Understanding resampling well is the natural stepping stone to understanding Bayesian computation.

Ask the tutor: How do I choose between the percentile bootstrap and the BCa (bias-corrected and accelerated) confidence interval for a skewed biological dataset?