The Maximum Entropy Principle

Statistical mechanics asks you to assign probabilities to microstates when you know almost nothing about them — only a few macroscopic averages like the energy. Which distribution should you pick out of the infinitely many consistent with that data? The maximum entropy principle, articulated by Edwin Thompson Jaynes in 1957 building on Claude Shannon's 1948 information theory, gives a sharp answer: choose the distribution that maximizes the entropy $S=-k\sum_i p_i\ln p_i$ subject to whatever you actually know. This is the least biased choice — it fits your constraints and assumes nothing else, spreading probability as evenly as the data permit. The payoff is that the standard ensembles fall out as theorems rather than postulates. Maximize entropy subject to fixed average energy and you get the canonical distribution $p_i\propto e^{-\beta E_i}$ ; add a constraint on average particle number and you get the grand canonical distribution $p_i\propto e^{-\beta(E_i-\mu N_i)}$ ; impose only normalization and you get the uniform microcanonical distribution $p_i=1/W$ . The Lagrange multiplier conjugate to the energy constraint is exactly the quantity that later turns out to be $1/kT$ . This article derives that machinery, works a concrete three-state example numerically, and states honestly what the principle does and does not prove — it is a rule for reasoning under incomplete information, not a mechanical law that forces nature to maximize anything. The uniqueness of the Shannon form and the dynamical generalization (maximum caliber) are named where they belong and not proved in full here.

Key takeaways

MaxEnt rule: given only some averages, pick the probability distribution that maximizes $S=-k\sum_i p_i\ln p_i$ subject to those constraints — the least-biased distribution consistent with what you know.
Fixing $\langle E\rangle$ (plus normalization) and maximizing gives the canonical distribution $p_i=e^{-\beta E_i}/Z$ ; the multiplier $\beta$ conjugate to energy is later identified with $1/kT$ .
Adding $\langle N\rangle$ gives the grand canonical distribution; using only normalization gives the microcanonical uniform distribution $p_i=1/W$ with $S=k\ln W$ .
Shannon's axioms (continuity, monotonicity for equiprobable outcomes, and a grouping/composition rule) single out $-k\sum p\ln p$ as the unique measure of uncertainty, up to the constant $k$ .
Any distribution matching the same constraints but not maximizing $S$ carries hidden assumptions and has strictly lower entropy.
Key idea: the equilibrium ensembles are not extra postulates — they are what honest inference from a handful of averages inevitably produces.

The core insight

Out of all probability distributions consistent with the averages you know, the one that maximizes the Shannon–Gibbs entropy $-k\sum_i p_i\ln p_i$ is the unique least-biased choice; carrying out that maximization with Lagrange multipliers reproduces the canonical, grand canonical, and microcanonical ensembles, with the energy multiplier $\beta$ equal to $1/kT$ .

Where this lives today

The maximum entropy principle is now a standard inference tool wherever one must build a distribution from limited moments. In physics it underlies the modern axiomatic route to the ensembles and generalizes to the maximum caliber principle, which selects distributions over whole trajectories (not just microstates) and is used in nonequilibrium and single-molecule statistics. Outside physics it powers maximum-entropy image and spectrum reconstruction, maximum-entropy (logistic/exponential-family) models in machine learning and natural-language processing, and species-distribution models in ecology. The deeper questions — the precise sense in which the Shannon form is unique, the role of the constraints, and whether MaxEnt can be founded on pure logic (the Cox–Jaynes program) — are active topics in the foundations of probability, and the dynamical extension to entropy production is treated in the nonequilibrium articles rather than here.

The physical picture

Imagine you must bet on which face of a many-sided die comes up, but all you are told is the long-run average of the numbers painted on the faces. Infinitely many probability assignments reproduce that one average. Assigning all the probability to a single face reproduces it too if you choose the face cleverly — but that is an outrageous claim of certainty you have no right to make. The honest assignment hugs the constraint while otherwise staying as flat and noncommittal as possible. "As flat as possible" needs a number to maximize, and that number is the entropy: it is largest for the uniform distribution and shrinks as probability piles up. Maximizing it under the constraint is the mathematical expression of "assume nothing you were not told." The same logic runs statistical mechanics. A macroscopic gas has an astronomical number of microstates; experiment pins down only a few averages — most importantly the mean energy. The microcanonical, canonical, and grand canonical ensembles are three different states of knowledge (energy fixed exactly; mean energy fixed; mean energy and mean number fixed), and each is the maximum-entropy distribution for its constraints. What looks in older treatments like a separate physical postulate for each ensemble becomes one principle applied three times.

Figure

The entropy of a two-outcome distribution peaks at p equal one half

The concavity in the figure is the whole engine: because $S$ is strictly concave on the space of distributions and the constraints are linear, the maximization has a unique solution, and it is found by the standard Lagrange-multiplier condition. The second figure shows what that solution looks like once a single average-energy constraint is switched on: the flat "know-nothing" distribution tilts into a decaying exponential in energy — the Boltzmann distribution — with the steepness set by the multiplier $\beta$ .

Figure

A mean-energy constraint tilts the flat distribution into a Boltzmann exponential

The core equations

The information entropy of a discrete probability distribution is the central object; everything else follows from maximizing it.

S = -k\sum_i p_i \ln p_i \qquad (1)

where: $p_i$ is the probability of microstate $i$ , the sum runs over all accessible microstates, and $k$ is a positive constant (Boltzmann's constant $k_B$ in physics, or $1/\ln 2$ to measure information in bits). $S\ge 0$ , vanishes when one $p_i=1$ (certainty), and is maximal for the uniform distribution. This is the Gibbs entropy; its ownership and general properties live in the sibling article on the Gibbs entropy. To find the least-biased distribution we maximize (1) subject to constraints, which we enforce with Lagrange multipliers. For the canonical case the constraints are normalization and a fixed mean energy.

\sum_i p_i = 1, \qquad \sum_i p_i E_i = \langle E\rangle \qquad (2)

where: $E_i$ is the energy of microstate $i$ and $\langle E\rangle$ is the (known) average energy. These are the only pieces of information assumed; the principle refuses to inject anything further. Maximizing (1) under (2) yields the canonical (Boltzmann) distribution, the central result.

p_i = \frac{e^{-\beta E_i}}{Z}, \qquad Z = \sum_i e^{-\beta E_i} \qquad (3)

where: $\beta$ is the Lagrange multiplier enforcing the energy constraint and $Z$ is the partition function that enforces normalization. The value of $\beta$ is fixed implicitly by requiring $\sum_i p_i E_i=\langle E\rangle$ ; thermodynamics later identifies it as $\beta=1/kT$ . The canonical ensemble and its partition function are owned by their own articles; here they emerge from inference. Feeding (3) back into (1) gives the maximized entropy in closed form, the quantity all lower-entropy distributions fall short of.

S_{\max} = k\left(\ln Z + \beta\langle E\rangle\right) \qquad (4)

where: the two terms are the log-partition function and the multiplier times the constrained average. Equation (4) is the bridge to thermodynamics: with $\beta=1/kT$ it reads $S_{\max}=(U-F)/T$ where $F=-kT\ln Z$ is the Helmholtz free energy, so free energy is $-kT\ln Z$ precisely because the ensemble is a maximum-entropy distribution. Adding a second constraint on the mean particle number introduces a second multiplier and produces the grand canonical distribution.

p_i = \frac{e^{-\beta(E_i-\mu N_i)}}{\Xi}, \qquad \Xi = \sum_i e^{-\beta(E_i-\mu N_i)} \qquad (5)

where: $N_i$ is the particle number of microstate $i$ , $\mu$ is (proportional to) the multiplier conjugate to $\langle N\rangle$ — the chemical potential — and $\Xi$ is the grand partition function. With no constraint but normalization, both multipliers vanish and (3) collapses to the uniform microcanonical distribution $p_i=1/W$ with $S=k\ln W$ .

Derivation AMaximizing entropy under a mean-energy constraint gives the Boltzmann distribution

The goal is to prove equation (3): among all normalized distributions with a prescribed mean energy, the one maximizing the entropy (1) is $p_i=e^{-\beta E_i}/Z$ . The strategy is the method of Lagrange multipliers — we build a functional that adds the constraints to the entropy with undetermined coefficients, set every partial derivative to zero, and solve. The one subtlety worth flagging is that we must check the stationary point is a maximum, not a saddle; this is guaranteed because $S$ is strictly concave (its second derivative $-k/p_i$ is negative) while the constraints are linear, so any interior stationary point is the unique global maximum.

Step 1. Build the Lagrangian. Working in units of $k=1$ for clarity, maximize $-\sum_i p_i\ln p_i$ subject to $\sum_i p_i=1$ and $\sum_i p_i E_i=\langle E\rangle$ . Introduce multipliers $\alpha$ and $\beta$ : $\mathcal{L}=-\sum_i p_i\ln p_i-\alpha\left(\sum_i p_i-1\right)-\beta\left(\sum_i p_i E_i-\langle E\rangle\right)$ .

Step 2. Differentiate with respect to each $p_i$ and set to zero. Since $\partial(p_i\ln p_i)/\partial p_i=\ln p_i+1$ , the condition is $-\ln p_i-1-\alpha-\beta E_i=0$ , i.e. $\ln p_i=-(1+\alpha)-\beta E_i$ .

Step 3. Exponentiate: $p_i=e^{-(1+\alpha)}\,e^{-\beta E_i}$ . The factor $e^{-(1+\alpha)}$ is independent of $i$ ; call its reciprocal $Z$ . Then $p_i=e^{-\beta E_i}/Z$ .

Step 4. Fix the multipliers by the constraints. Normalization $\sum_i p_i=1$ forces $Z=\sum_i e^{-\beta E_i}$ , the partition function. The remaining multiplier $\beta$ is set implicitly by $\sum_i (E_i e^{-\beta E_i})/Z=\langle E\rangle$ ; equivalently $\langle E\rangle=-\partial\ln Z/\partial\beta$ . This is one equation for one unknown $\beta$ , and because $\langle E\rangle(\beta)$ is monotonically decreasing it has a unique solution for any physical $\langle E\rangle$ .

Step 5. Confirm it is a maximum and evaluate $S$ . The Hessian of $-\sum p_i\ln p_i$ is diagonal with entries $-1/p_i<0$ , so on the linear constraint surface the stationary point is the strict global maximum. Substituting $p_i=e^{-\beta E_i}/Z$ into $S=-\sum p_i\ln p_i$ and using $\ln p_i=-\beta E_i-\ln Z$ gives $S=\sum p_i(\beta E_i+\ln Z)=\beta\langle E\rangle+\ln Z$ , which is equation (4) (restoring $k$ ). Verified: the companion script maximizes $-\sum p_i\ln p_i$ numerically for a three-state system with a fixed mean and confirms the optimizer equals $e^{-\beta E_i}/Z$ to machine precision, that $\beta=-\partial\ln Z/\partial\beta$ reproduces the imposed $\langle E\rangle$ , and that $S_{\max}=\beta\langle E\rangle+\ln Z$ .

Go deeper · Derivation B · 1 minShannon's axioms single out the entropy functional

Derivation BShannon's axioms single out the entropy functional

The aim here is to explain why the specific combination $-\sum p_i\ln p_i$ — and not some other measure of "spread" — is the right thing to maximize. Shannon asked what functional $H(p_1,\dots,p_n)$ could serve as a unique measure of the uncertainty in a distribution, and showed that three reasonable requirements determine it up to an overall positive constant. We state the axioms and sketch why they force the logarithmic form; a fully rigorous proof is a short exercise in functional equations that we cite rather than reproduce, so the scope of the claim stays honest.

Step 1. State the axioms. (i) Continuity: $H$ is a continuous function of the $p_i$ . (ii) Monotonicity: for $n$ equally likely outcomes, $H(1/n,\dots,1/n)=A(n)$ increases monotonically with $n$ — more equally likely alternatives means more uncertainty. (iii) Grouping (composition): if a choice is broken into successive sub-choices, $H$ is the weighted sum of the entropies of the stages. Concretely $H(p_1,p_2,p_3)=H(p_1+p_2,\,p_3)+(p_1+p_2)\,H\!\left(\tfrac{p_1}{p_1+p_2},\tfrac{p_2}{p_1+p_2}\right)$ .

Step 2. Reduce to the equiprobable case. Applying the grouping axiom repeatedly to a distribution with rational weights lets you express any $H$ in terms of the function $A(n)=H(1/n,\dots,1/n)$ . The grouping axiom applied to $n=mr$ equally likely outcomes, split into $m$ groups of $r$ , gives the Cauchy-type functional equation $A(mr)=A(m)+A(r)$ .

Step 3. Solve the functional equation. The only monotonic solution of $A(mr)=A(m)+A(r)$ is $A(n)=k\ln n$ for some constant $k>0$ . Continuity extends the result from rational to all distributions, yielding $H=-k\sum_i p_i\ln p_i$ .

Step 4. Interpret the constant. $k$ merely sets the units: $k=k_B$ gives thermodynamic entropy in joules per kelvin, $k=1/\ln 2$ gives Shannon information in bits. No axiom fixes $k$ , which is exactly why the same functional serves both physics and communication theory. Verified: the script checks the grouping axiom numerically, confirming $H(\tfrac12,\tfrac13,\tfrac16)=H(\tfrac12,\tfrac12)+\tfrac12 H(\tfrac23,\tfrac13)$ , and that the equiprobable entropy $A(n)=\ln n$ satisfies $A(mr)=A(m)+A(r)$ .

Limiting cases

Each limit recovers a familiar ensemble or a sanity check on the construction.

Case	Result	Meaning
No constraint but normalization	$p_i=1/W$ , $S=k\ln W$	Total ignorance is the uniform microcanonical distribution — Boltzmann's $S=k\ln W$ .
Fixed $\langle E\rangle$	$p_i=e^{-\beta E_i}/Z$	The canonical (Boltzmann) distribution; $\beta$ conjugate to energy becomes $1/kT$ .
Fixed $\langle E\rangle$ and $\langle N\rangle$	$p_i=e^{-\beta(E_i-\mu N_i)}/\Xi$	The grand canonical distribution; $\mu$ conjugate to number is the chemical potential.
$\beta\to 0$ (very large $\langle E\rangle$ )	$p_i\to 1/W$	Weak energy constraint returns the flat distribution — high "temperature" is near-total ignorance.
One outcome certain	$S\to 0$	Perfect knowledge carries zero entropy; the principle then has nothing left to maximize.

Common trapMaxEnt is a rule of inference, not a law forcing nature to maximize entropy

The maximum entropy principle tells you how to assign probabilities honestly given limited data; it does not assert that a physical system dynamically climbs to maximum entropy, and it is not a substitute for the microscopic dynamics that actually establish equilibrium. Two cautions follow. First, the answer depends entirely on which constraints you impose: choose the wrong or incomplete set of averages and MaxEnt cheerfully returns the wrong distribution — the principle is only as good as your list of known quantities. Second, the derivation gives you the form of the equilibrium ensemble and the meaning of $\beta$ , but the identification $\beta=1/kT$ and the claim that real gases actually realize this distribution rest on separate physical input (the dynamics, ergodicity, and contact with a heat bath). MaxEnt unifies and clarifies the ensembles; it does not by itself prove the second law or replace kinetic theory.

Worked examples

Example 1 — A three-state system with a fixed mean energy. A system has three microstates with energies $0,\ \varepsilon,\ 2\varepsilon$ . You are told only that the average energy is $\langle E\rangle=0.8\,\varepsilon$ . Find the maximum-entropy distribution and its entropy, and compare with the unconstrained maximum.

Step 1. Write the MaxEnt form. By equation (3), $p_i=e^{-\beta E_i}/Z$ with $Z=1+e^{-\beta\varepsilon}+e^{-2\beta\varepsilon}$ (measuring $\beta$ in units of $1/\varepsilon$ ).

Step 2. Solve for $\beta$ from the constraint $\langle E\rangle=\varepsilon\,(e^{-\beta\varepsilon}+2e^{-2\beta\varepsilon})/Z=0.8\,\varepsilon$ . Numerically this gives $\beta\varepsilon=0.3046$ , so the probabilities are $p=(0.4384,\ 0.3233,\ 0.2384)$ . Check: $0.3233+2(0.2384)=0.8001$ ✓.

Step 3. Compute the entropy: $S/k=-\sum p_i\ln p_i=0.4384\ln\!\tfrac{1}{0.4384}+0.3233\ln\!\tfrac{1}{0.3233}+0.2384\ln\!\tfrac{1}{0.2384}=\mathbf{1.0684}$ nats. The unconstrained maximum (uniform $p=1/3$ ) would be $\ln 3=\mathbf{1.0986}$ nats. The constraint has lowered the achievable entropy, exactly as it must — knowing the mean energy is information, and information reduces entropy. The takeaway: the MaxEnt distribution is the flattest one consistent with $\langle E\rangle=0.8\varepsilon$ , and any other distribution with the same mean — for instance $(0.5,0.2,0.3)$ , which also averages $0.8\varepsilon$ — has strictly smaller entropy ( $0.5\ln2+\dots=1.030$ nats $<1.068$ ).

Example 2 — Recovering microcanonical and grand canonical limits. Take the same three states but impose no energy constraint, only normalization.

Step 1. With only $\sum p_i=1$ , the Lagrangian has a single multiplier and Step 2 of the derivation gives $\ln p_i=\text{const}$ , so $p_i=1/3$ for all $i$ : the uniform distribution.

Step 2. Its entropy is $S/k=\ln 3=1.0986$ nats $=k\ln W$ with $W=3$ — the microcanonical result. For $W=6$ equally likely states one would get $\ln 6=1.7918$ nats.

Step 3. Now suppose the states also carry particle numbers $N_i$ and you additionally know $\langle N\rangle$ . Adding the term $-\gamma(\sum p_i N_i-\langle N\rangle)$ to the Lagrangian and repeating Steps 2–3 gives $p_i\propto e^{-\beta E_i-\gamma N_i}=e^{-\beta(E_i-\mu N_i)}$ with $\mu\equiv-\gamma/\beta$ : the grand canonical distribution (5). The takeaway: one principle, applied with zero, one, or two constraints, generates the microcanonical, canonical, and grand canonical ensembles respectively — they are three states of knowledge, not three separate laws.

Origins · 1 minmaximum entropy and how this was discovered

maximum entropy

principle stated by E. T. Jaynes, 1957 · The rule that, given partial information, the least-biased probability distribution is the one of greatest Shannon–Gibbs entropy consistent with the data

The maximum entropy principle was crystallized by Edwin Thompson Jaynes (1922–1998) in two 1957 Physical Review papers that reinterpreted statistical mechanics as a branch of statistical inference rather than mechanics. Jaynes built directly on Claude Shannon's 1948 measure of information, $H=-\sum p\ln p$ , arguing that the Gibbs ensembles are simply the distributions of maximum entropy subject to the macroscopic constraints one happens to know. The viewpoint reframed temperature, free energy, and the partition function as consequences of honest reasoning under uncertainty, and it seeded modern applications far beyond physics — image reconstruction, spectral analysis, natural-language modeling, and ecology. (This card is titled by the concept because the individual founders, Jaynes and Shannon, already head other articles in this collection.)

History

1877

— Boltzmann relates entropy to the number of microstates,

S=k\log W

1902

— Gibbs formalizes the microcanonical, canonical, and grand canonical ensembles

1948

— Shannon defines information entropy

H=-\sum p\ln p

in "A Mathematical Theory of Communication"

1957

— Jaynes publishes "Information Theory and Statistical Mechanics," Phys. Rev. 106, 620

1957

— Jaynes publishes the sequel, Phys. Rev. 108, 171, extending the method to irreversibility

1963

— Jaynes' Brandeis lectures spread the maximum-entropy formalism through physics

1984

— Skilling and Gull develop the maximum-entropy method for image reconstruction

Going further

References. Jaynes, "Information Theory and Statistical Mechanics," Phys. Rev. 106, 620 (1957) and 108, 171 (1957) · Shannon, "A Mathematical Theory of Communication," Bell Syst. Tech. J. 27, 379 (1948) · Jaynes, Probability Theory: The Logic of Science (2003) · Sethna, Statistical Mechanics: Entropy, Order Parameters, and Complexity, ch. 5–6 · Pathria & Beale, Statistical Mechanics, 3rd ed., §3.

Provenance. A verifier should reproduce: (1) the three-state optimizer $\beta\varepsilon=0.3046$ , $p=(0.4384,0.3233,0.2384)$ from numerically maximizing $-\sum p\ln p$ at fixed $\langle E\rangle=0.8\varepsilon$ , matching $e^{-\beta E_i}/Z$ ; (2) $S_{\max}=1.0684$ nats and its identity with $\beta\langle E\rangle+\ln Z$ ; (3) $S_{\max}<\ln 3=1.0986$ and a lower entropy $1.030$ for the alternative $(0.5,0.2,0.3)$ ; (4) the grouping axiom $H(\tfrac12,\tfrac13,\tfrac16)=H(\tfrac12,\tfrac12)+\tfrac12 H(\tfrac23,\tfrac13)=1.0114$ ; (5) $\ln 6=1.7918$ for six equally likely states; (6) the equiprobable additivity $A(mr)=A(m)+A(r)$ with $A(n)=\ln n$ .

Builds on: Gibbs Entropy · Fundamental Postulate of Equal A Priori Probabilities · Canonical Ensemble. Leads to: Canonical Ensemble · Grand Canonical Ensemble · Maximum caliber and dynamical ensembles.

Isn't picking "maximum entropy" just as arbitrary as any other choice?

No — it is the unique choice that adds no information beyond your constraints. Shannon's axioms show that $-\sum p\ln p$ is the only functional (up to units) that consistently measures uncertainty, so maximizing it is the only way to be maximally noncommittal about everything you were not told. Any other rule would smuggle in extra assumptions: it would make some outcomes more or less likely than the data warrant. That is why MaxEnt is called the least-biased inference, not merely one option among many.

Why does the canonical distribution come out exponential rather than some other shape?

The exponential is forced by two facts: the entropy's derivative is logarithmic ( $\partial(p\ln p)/\partial p=\ln p+1$ ), and the energy constraint enters linearly. Setting the derivative of the Lagrangian to zero therefore gives $\ln p_i=\text{const}-\beta E_i$ , and exponentiating gives $p_i\propto e^{-\beta E_i}$ . Any constraint that is linear in $p_i$ times some quantity $Q_i$ adds a term $-\lambda Q_i$ to the exponent, so the maximum-entropy distribution is always of the exponential-family form $p_i\propto\exp(-\sum_a\lambda_a Q_i^{(a)})$ .

Does the constant

k

matter?

Only for units. The functional $-k\sum p\ln p$ is maximized at the same distribution for any positive $k$ , so the shape of the answer — the Boltzmann distribution, the value of the probabilities — is independent of $k$ . Choosing $k=k_B$ measures entropy in joules per kelvin and makes $\beta=1/k_BT$ ; choosing $k=1/\ln 2$ measures information in bits. The physics is in the constraints and the multipliers, not in $k$ .

How is this different from Boltzmann's

S=k\ln W

Boltzmann's formula is the special case of MaxEnt with no constraint but normalization: when all $W$ accessible microstates are equally likely, $-k\sum p\ln p=-kW\cdot\tfrac1W\ln\tfrac1W=k\ln W$ . So $S=k\ln W$ is the maximum-entropy (microcanonical) answer, and the Gibbs form $-k\sum p\ln p$ generalizes it to distributions that are not uniform, which is exactly what you need once a heat bath fixes the mean energy instead of the energy itself.

If I add more and more constraints, what happens?

Each independent constraint adds a Lagrange multiplier and an exponential factor, and lowers (or leaves unchanged) the maximum achievable entropy — more knowledge means less uncertainty. In the limit where you constrain enough moments to pin the distribution completely, the entropy drops to that of a single fixed distribution. In practice one keeps only the constraints that are physically controlled (energy, particle number, sometimes volume or magnetization); adding spurious constraints is the classic way to get a wrong answer from a correct principle.

Does MaxEnt prove the second law of thermodynamics?

Not by itself. MaxEnt tells you the equilibrium distribution is the maximum-entropy one for the known constraints, but the claim that an isolated system evolves toward that distribution requires the microscopic dynamics (mixing, coarse-graining, contact with a bath). Jaynes' second 1957 paper and later work connect MaxEnt to irreversibility, but the arrow of time is extra physics layered on top of the inference principle. Treat MaxEnt as the correct bookkeeping for equilibrium ensembles, and look to the kinetic and nonequilibrium articles for why systems approach equilibrium.