Gibbs Entropy

When a message arrives and resolves your uncertainty, exactly how much did you learn? Claude Shannon answered that in 1948 with a formula — $-\sum p_i\log p_i$ — and was startled to find it was already famous: Josiah Willard Gibbs had written the same expression half a century earlier as the entropy of a statistical ensemble. The Gibbs entropy $S = -k_{\mathrm B}\sum_i p_i\ln p_i$ is the master definition of entropy in statistical mechanics, and its coincidence with Shannon's information measure is no accident: both count the same thing — the number of microscopic possibilities left open by what you know. The Gibbs entropy generalizes Boltzmann's $S = k_{\mathrm B}\ln\Omega$ to any probability distribution over microstates, not just the uniform one. Where Boltzmann counts equally weighted microstates on an energy shell, Gibbs weighs each microstate $i$ by its probability $p_i$ and forms the functional $-k_{\mathrm B}\sum p_i\ln p_i$ . When the distribution happens to be uniform — the microcanonical case — the two collapse to the identical number. But Gibbs's form keeps working when microstates are not equally likely: in a heat bath the canonical weights $p_i \propto e^{-\beta E_i}$ are wildly unequal, and only the Gibbs functional gives the right entropy. This one expression is the pivot of the whole subject. Maximize it subject to a fixed mean energy and the Boltzmann distribution falls out — thermodynamics from a variational principle. Evaluate it on the equilibrium distribution and you recover the free energy $F = U - TS$ . Its non-negativity and its concavity encode the second law and thermodynamic stability. And because it is bounded above by $k_{\mathrm B}\ln W$ — achieved only when every accessible microstate is equally likely — it tells you that ignorance, quantified honestly, is maximized by the uniform distribution. Entropy, in Gibbs's hands, becomes the least-biased accounting of what a probability distribution does not pin down.

Key takeaways

The Gibbs entropy of a distribution $\{p_i\}$ over microstates is $S = -k_{\mathrm B}\sum_i p_i\ln p_i$ — the ensemble generalization of the Boltzmann entropy.
It reduces exactly to $S = k_{\mathrm B}\ln\Omega$ on the uniform (microcanonical) distribution $p_i = 1/\Omega$ , so Boltzmann entropy is its special case.
Maximizing it at fixed mean energy $\langle E\rangle$ yields the canonical (Boltzmann) distribution $p_i = e^{-\beta E_i}/Z$ — thermodynamics as a variational principle.
It is bounded: $0 \le S \le k_{\mathrm B}\ln W$ , the maximum reached only for the uniform distribution over $W$ accessible microstates.
The same functional $-\sum p\ln p$ is Shannon's information entropy; the Gibbs inequality (KL divergence $\ge 0$ ) underpins both the variational principle and the second law.
Key idea: the Gibbs entropy $-k_{\mathrm B}\sum p_i\ln p_i$ measures the uncertainty in a distribution over microstates; maximizing it under constraints generates the equilibrium ensembles and all their thermodynamics.

The core insight

The Gibbs entropy $S = -k_{\mathrm B}\sum_i p_i\ln p_i$ is the entropy of a probability distribution over microstates — the least-biased measure of how much that distribution leaves undetermined. It equals the Boltzmann entropy when the distribution is uniform, and maximizing it under the constraints that define an ensemble produces the equilibrium distributions and their thermodynamics.

Where this lives today

The Gibbs functional is the beating heart of the maximum-entropy method, which now reaches far beyond physics. Jaynes recast statistical mechanics in 1957 as inference: the equilibrium ensembles are simply the distributions that maximize $-\sum p\ln p$ subject to whatever macroscopic averages you know, and the same recipe now builds models in machine learning (maximum-entropy classifiers), ecology, natural-language processing, and neuroscience. Every time a "MaxEnt" model is fit to data, it is Gibbs's entropy being maximized under constraints. The deepest modern current is the identity of thermodynamic and information entropy. Landauer's principle — erasing a bit costs at least $k_{\mathrm B}T\ln 2$ of dissipated heat — is the Gibbs/Shannon coincidence made physical, and it sets fundamental limits on the energy cost of computation now approached by real chips. In quantum information the same functional becomes the von Neumann entropy $-k_{\mathrm B}\mathrm{Tr}(\rho\ln\rho)$ , the currency of entanglement and quantum thermodynamics. Gibbs wrote down a formula for gases; it turned out to be a universal statement about uncertainty.

The physical picture

Picture the accessible microstates as boxes and the probability $p_i$ as how much of your belief sits in each. If all your belief piles into one box ( $p_1 = 1$ , the rest zero), you are certain, and the Gibbs entropy is zero — you have no uncertainty to measure. Spread your belief evenly across $W$ boxes and uncertainty is maximal: $S = k_{\mathrm B}\ln W$ , the same as Boltzmann's count. Any lopsided distribution sits between these extremes. Entropy is literally the width of your ignorance, weighted honestly by how likely each possibility is. The figure below plots the Gibbs entropy of a two-outcome distribution as one probability slides from 0 to 1. It is zero at both ends (certainty) and peaks in the dead centre (maximum ignorance), a concave hill — the shape that makes entropy maximization have a unique answer.

Figure

Gibbs entropy of a two-state distribution: concave, zero at certainty, peaked at p = ½

Now the punchline that makes the Gibbs form indispensable. In a heat bath the microstates are not equally likely — high-energy states are exponentially rare, weighted by $e^{-\beta E_i}$ . The next figure shows those unequal Boltzmann weights, and the point is that only a distribution-aware entropy ( $-\sum p\ln p$ ), not a flat count of states, can assign the right value here. The Gibbs entropy of this exponential distribution, maximized at fixed mean energy, is precisely what selects it.

Figure

Canonical weights are wildly unequal — the Gibbs entropy handles what a flat count cannot

The two entropies — Boltzmann's flat count and Gibbs's weighted sum — agree exactly when the weights happen to be equal, and only Gibbs's survives when they are not. That is the sense in which Gibbs is the master definition and Boltzmann the important special case.

The core equations

The defining functional assigns an entropy to any normalized probability distribution over microstates. It is the same expression whether the microstates come from a spin lattice, a gas, or an abstract information source.

S = -k_{\mathrm B}\sum_i p_i\ln p_i \qquad (1)

where: $S$ — entropy of the ensemble; $k_{\mathrm B}$ — Boltzmann constant; $p_i$ — probability of microstate $i$ , with $\sum_i p_i = 1$ . Read each term $-p_i\ln p_i$ as the "surprise" of outcome $i$ weighted by how often it occurs; entropy is the average surprise. For a classical system the microstates form a continuum, so the sum becomes an integral over phase space, with the same $h^{3N}N!$ factors that made the Boltzmann count dimensionless and extensive.

S = -k_{\mathrm B}\int \rho(q,p)\,\ln\!\big[\rho(q,p)\,h^{3N}N!\big]\,\mathrm d\Gamma,\qquad \mathrm d\Gamma = \prod_a \mathrm d^3q_a\,\mathrm d^3p_a \qquad (2)

where: $\rho(q,p)$ — phase-space density, normalized $\int\rho\,\mathrm d\Gamma = 1$ ; $h^{3N}N!$ — the quantum cell and Gibbs factor inside the log, so the argument is dimensionless. The bracketed $\rho h^{3N}N!$ is exactly the local microstate occupation, tying the continuum form to the discrete count. The single most important property: on the uniform distribution the Gibbs entropy collapses to the Boltzmann entropy. This is the bridge between the two definitions, and it makes the microcanonical ensemble a special case rather than a separate postulate.

p_i = \frac{1}{\Omega}\ \ \forall i \quad\Longrightarrow\quad S = -k_{\mathrm B}\sum_{i=1}^{\Omega}\frac{1}{\Omega}\ln\frac{1}{\Omega} = k_{\mathrm B}\ln\Omega \qquad (3)

where: $\Omega$ — number of accessible microstates. The sum has $\Omega$ identical terms each equal to $\frac{1}{\Omega}\ln\Omega$ , recovering the tombstone formula owned by the Boltzmann-entropy article. Maximizing the functional at fixed mean energy is where thermodynamics enters. The constrained maximum is the canonical distribution — the Boltzmann weights — and this is the variational principle at the root of the canonical ensemble.

p_i = \frac{e^{-\beta E_i}}{Z},\qquad Z = \sum_i e^{-\beta E_i} \qquad (4)

where: $\beta = 1/k_{\mathrm B}T$ — the Lagrange multiplier enforcing fixed $\langle E\rangle$ ; $E_i$ — energy of microstate $i$ ; $Z$ — the partition function. The exponential is not assumed; it is derived by maximizing Eq. (1), as the second derivation shows. Substituting the canonical weights back into the Gibbs functional produces the thermodynamic bridge to the free energy — the equation that turns the partition function into all of equilibrium thermodynamics.

S = \frac{U - F}{T},\qquad F = -k_{\mathrm B}T\ln Z,\qquad U = \langle E\rangle \qquad (5)

where: $U$ — internal energy $\sum_i p_iE_i$ ; $F$ — Helmholtz free energy; $T$ — absolute temperature. Rearranged, this is the familiar $F = U - TS$ : the Gibbs entropy is the exact link between the microscopic $Z$ and macroscopic free energy. Finally, the two bounds that encode the second law and the maximum-ignorance principle. Entropy is non-negative and capped by the log of the number of accessible microstates.

0 \le S \le k_{\mathrm B}\ln W \qquad (6)

where: $W$ — number of microstates with nonzero probability. The lower bound (equality for a certain outcome) and the upper bound (equality for the uniform distribution) both follow from the Gibbs inequality below.

The core equations — the inequality behind it all

The Gibbs (Kullback–Leibler) inequality is the single mathematical fact from which the variational principle, the entropy bounds, and the second law all descend. It states that the relative entropy of any distribution against any reference is non-negative.

D(p\Vert q) = \sum_i p_i\ln\frac{p_i}{q_i} \ge 0,\quad\text{equality iff } p = q \qquad (7)

where: $q_i$ — any reference distribution with $\sum_i q_i = 1$ ; $D(p\Vert q)$ — the Kullback–Leibler divergence. Choosing $q_i = 1/W$ uniform turns this into the upper bound $S \le k_{\mathrm B}\ln W$ ; choosing $q$ to be the equilibrium distribution turns it into the second law for open systems.

Derivation AThe uniform distribution maximizes entropy, and there Gibbs equals Boltzmann

This is one half of the article's owned result. We prove that among all probability distributions on a fixed set of $\Omega$ accessible microstates — with no constraint beyond normalization — the Gibbs entropy is maximized by the uniform distribution $p_i = 1/\Omega$ , that the maximum value is exactly $k_{\mathrm B}\ln\Omega$ , and that the second-order condition confirms it is a genuine maximum. The uniform choice is therefore not an extra postulate: it is the least-biased (maximum-entropy) distribution for an isolated system, and evaluating Gibbs on it reproduces Boltzmann's tombstone formula. The tool is a single Lagrange multiplier.

Step 1. Build the Lagrange functional $\mathcal L = -k_{\mathrm B}\sum_i p_i\ln p_i - \lambda\!\left(\sum_i p_i - 1\right)$ , with $\lambda$ enforcing normalization. Why: normalization is the only information an isolated system supplies, so it is the sole constraint we impose while extremizing.

Step 2. Stationarity $\partial\mathcal L/\partial p_i = 0$ gives $-k_{\mathrm B}(\ln p_i + 1) - \lambda = 0$ , so $p_i = \exp(-1 - \lambda/k_{\mathrm B})$ — a constant independent of $i$ . Why: the derivative condition is identical for every microstate, forcing the maximizing distribution to be flat.

Step 3. Normalization $\sum_i p_i = 1$ over $\Omega$ states fixes $p_i = 1/\Omega$ , and substituting into Eq. (1):

S_{\max} = -k_{\mathrm B}\sum_{i=1}^{\Omega}\frac{1}{\Omega}\ln\frac{1}{\Omega} = k_{\mathrm B}\ln\Omega

Why: the uniform distribution IS the postulate of equal a priori probabilities, here derived not assumed, and evaluating Gibbs on it collapses the sum into the Boltzmann entropy.

Step 4. Confirm it is a maximum: the second variation of $-\sum p_i\ln p_i$ has diagonal entries $\partial^2 S/\partial p_i^2 = -k_{\mathrm B}/p_i < 0$ , so the functional is strictly concave and the stationary point is the unique global maximum. Why: strict concavity guarantees the extremum is a maximum, not a saddle, so the uniform distribution genuinely maximizes ignorance. *Verified: stationarity gives constant $p_i$ , normalization fixes $1/\Omega$ , the collapse to $\ln\Omega$ , and the concavity $\partial^2 S/\partial p_i^2 < 0$ are all checked symbolically and numerically in verify_aq-sm-gibbs-entropy.py. Cf. Reif Ch. 3, Sethna Ch. 5.*

Go deeper · Derivation B · 1 minThe canonical distribution from constrained maximization, and additivity over independent systems

Derivation BThe canonical distribution from constrained maximization, and additivity over independent systems

This completes the owned result. First we maximize the Gibbs entropy subject to a second constraint — fixed mean energy $\langle E\rangle = U$ — and show the maximizer is the Boltzmann distribution $p_i = e^{-\beta E_i}/Z$ , deriving the canonical ensemble from a variational principle rather than postulating it. Then we prove the Gibbs entropy is additive over independent subsystems, $S(\rho_A\otimes\rho_B) = S_A + S_B$ — the property that makes it a legitimate thermodynamic entropy and mirrors the multiplicativity of microstate counts.

Step 1. Extremize $\mathcal L = -k_{\mathrm B}\sum_i p_i\ln p_i - \lambda(\sum_i p_i - 1) - \beta k_{\mathrm B}(\sum_i p_iE_i - U)$ with two multipliers: $\lambda$ for normalization, $\beta$ for the energy constraint. Why: an isolated system fixes only the count, but a system in a heat bath additionally fixes its mean energy — the extra constraint is what produces temperature.

Step 2. Stationarity gives $-k_{\mathrm B}(\ln p_i + 1) - \lambda - \beta k_{\mathrm B}E_i = 0$ , so $p_i \propto e^{-\beta E_i}$ ; normalization then supplies $Z = \sum_i e^{-\beta E_i}$ and $p_i = e^{-\beta E_i}/Z$ . Why: solving for $p_i$ yields the exponential Boltzmann weight, with $\beta = 1/k_{\mathrm B}T$ emerging as the multiplier conjugate to energy — the canonical ensemble derived, not assumed.

Step 3. Insert these weights into Eq. (1): $\ln p_i = -\beta E_i - \ln Z$ , so $S = -k_{\mathrm B}\sum_i p_i(-\beta E_i - \ln Z) = k_{\mathrm B}\beta U + k_{\mathrm B}\ln Z = (U - F)/T$ with $F = -k_{\mathrm B}T\ln Z$ . Why: this is the thermodynamic bridge — the Gibbs entropy of the canonical distribution is exactly $(U-F)/T$ , delivering $F = U - TS$ .

Step 4. For two independent subsystems the joint distribution factorizes, $p_{ab} = p^A_a p^B_b$ . Then $S = -k_{\mathrm B}\sum_{ab}p^A_a p^B_b(\ln p^A_a + \ln p^B_b) = S_A + S_B$ , using $\sum_b p^B_b = 1$ and $\sum_a p^A_a = 1$ . Why: independence makes the log of a product a sum, so entropy adds — the additivity that qualifies $-\sum p\ln p$ as a thermodynamic entropy and echoes $\Omega = \Omega_A\Omega_B$ . *Verified: the canonical maximizer $p_i = e^{-\beta E_i}/Z$ , the free-energy identity $S = (U-F)/T$ on a two-level system, and the additivity $S(\rho_A\otimes\rho_B) = S_A + S_B$ are all checked numerically in the verify script. Cf. Pathria §3.2–3.3, Kardar Ch. 4.*

Limiting cases

Case	Result	Meaning
Uniform distribution	$S = k_{\mathrm B}\ln\Omega$	Gibbs collapses to the Boltzmann entropy — the microcanonical special case
Certain outcome ( $p_j = 1$ )	$S = 0$	no uncertainty, no entropy; the third-law limit of a unique ground state
Canonical ( $p_i = e^{-\beta E_i}/Z$ )	$S = (U - F)/T$	the equilibrium entropy in a heat bath, bridging $Z$ to free energy
Two independent systems	$S = S_A + S_B$	additivity from factorized probabilities, mirroring $\Omega = \Omega_A\Omega_B$

Common trap“Fine-grained Gibbs entropy increases as a system relaxes to equilibrium”

It does not — and this is the subtlest point in the subject. The fine-grained Gibbs entropy $-k_{\mathrm B}\int\rho\ln\rho\,\mathrm d\Gamma$ is exactly constant under the microscopic (Liouville) dynamics, because the flow is volume-preserving and merely relabels which microstate carries which probability. If you compute Gibbs entropy from the true $\rho(t)$ , it never changes. The entropy that rises during relaxation is the coarse-grained version — $\rho$ averaged over finite phase-space cells — which grows because mixing stretches the distribution into filaments finer than any resolution. Confusing the two makes the second law look like it contradicts reversible mechanics; keeping them distinct dissolves the paradox. The full reconciliation is owned by the statistical-irreversibility article.

Worked examples

1 · Gibbs entropy of an explicit three-microstate distribution (confidence builder)

A system has three accessible microstates with probabilities $p = (\tfrac12, \tfrac13, \tfrac16)$ . Compute its Gibbs entropy, then compare with the uniform distribution over the same three states to confirm the uniform case is larger.

Step 1. Term by term, $-p_i\ln p_i$ : for $p_1 = \tfrac12$ , $-\tfrac12\ln\tfrac12 = 0.3466$ ; for $p_2 = \tfrac13$ , $-\tfrac13\ln\tfrac13 = 0.3662$ ; for $p_3 = \tfrac16$ , $-\tfrac16\ln\tfrac16 = 0.2986$ .

Step 2. Sum them: $S/k_{\mathrm B} = 0.3466 + 0.3662 + 0.2986 = \boxed{1.011}$ , so $S = 1.011\,k_{\mathrm B} = 1.40\times10^{-23}\ \mathrm{J/K}$ .

Step 3. The uniform distribution over three states gives $S/k_{\mathrm B} = \ln 3 = 1.099$ , which is larger. The difference is exactly the KL divergence: $D(p\Vert \text{unif}) = \ln 3 - 1.011 = 0.088 > 0$ , confirming Eq. (7).

Step 4. The lesson: any departure from uniform lowers the entropy, and by precisely the relative-entropy penalty. Uniform is the maximum-ignorance distribution, and the Gibbs functional measures exactly how far from it you have strayed.

2 · Entropy of a two-level system in a heat bath (exam level)

A single two-level system has ground state at energy $0$ and excited state at energy $\varepsilon$ . In a bath at temperature $T$ , compute the Gibbs entropy as a function of $x = \beta\varepsilon = \varepsilon/k_{\mathrm B}T$ , and evaluate its high- and low-temperature limits.

Step 1. Canonical weights: $Z = 1 + e^{-x}$ , so $p_0 = 1/(1+e^{-x})$ and $p_1 = e^{-x}/(1+e^{-x})$ . The mean energy is $U = \varepsilon\,p_1 = \varepsilon\,e^{-x}/(1+e^{-x})$ .

Step 2. Use the bridge $S = (U-F)/T$ with $F = -k_{\mathrm B}T\ln Z$ : $S/k_{\mathrm B} = \beta U + \ln Z = \dfrac{x\,e^{-x}}{1+e^{-x}} + \ln(1+e^{-x})$ .

Step 3. High temperature $x\to 0$ : both states equally likely, $S/k_{\mathrm B}\to 0 + \ln 2 = \boxed{\ln 2 = 0.693}$ — maximum entropy for two states. Check directly: at $x=0$ , $p_0 = p_1 = \tfrac12$ , and $-2\cdot\tfrac12\ln\tfrac12 = \ln 2$ ✓.

Step 4. Low temperature $x\to\infty$ : $p_0\to 1$ , $p_1\to 0$ , and $S/k_{\mathrm B}\to 0$ — the system freezes into its ground state, entropy vanishes (third law). Between the two limits the Gibbs entropy rises from $0$ to $\ln 2$ , tracking exactly how uncertain the occupation is — a Schottky-style curve. As a concrete point, at $x = 1$ ( $k_{\mathrm B}T = \varepsilon$ ): $S/k_{\mathrm B} = \tfrac{e^{-1}}{1+e^{-1}} + \ln(1+e^{-1}) = 0.269 + 0.313 = 0.582$ .

Origins · 1 minClaude Shannon and how this was discovered

Claude Shannon

1916–2001 · Bell Labs, USA

Shannon founded information theory in his 1948 paper A Mathematical Theory of Communication, where he sought a measure of the uncertainty in a probability distribution and derived $H = -\sum p_i\log p_i$ from a short list of natural axioms (continuity, monotonicity for uniform distributions, and a grouping/composition rule). The story goes that von Neumann advised him to call it "entropy," partly because Gibbs's identical expression had governed statistical mechanics since 1902 and partly because "no one really knows what entropy is, so in a debate you will always have the advantage." The coincidence is deep, not cosmetic: Gibbs's thermodynamic entropy and Shannon's information entropy are the same functional counting the same thing — the possibilities a distribution leaves open — and the bridge between them (Landauer, Jaynes) is now a research field of its own.

History

1865

— Clausius names entropy and gives dS = δQ/T for reversible heat transfer.

1877

— Boltzmann relates entropy to the number of microscopic complexions.

1902

— Gibbs, in Elementary Principles in Statistical Mechanics, writes the ensemble entropy −k Σ p ln p.

1948

— Shannon derives −Σ p log p as the measure of information, later named entropy on von Neumann's advice.

1957

— Jaynes formulates statistical mechanics as maximum-entropy inference from the Gibbs functional.

Going further

Texts. Pathria & Beale, Statistical Mechanics, 3rd ed., §3.2–3.3, §6 · Reif, Fundamentals of Statistical and Thermal Physics, Ch. 3 · Kardar, Statistical Physics of Particles, Ch. 4 · Sethna, Statistical Mechanics: Entropy, Order Parameters, and Complexity, Ch. 5–6 · Jaynes, Information Theory and Statistical Mechanics, Phys. Rev. 106 (1957).

Provenance. The Gibbs functional $S = -k_{\mathrm B}\sum p_i\ln p_i$ and its continuum form are from Gibbs (1902) and Pathria §3.3. A verifier should recompute: the collapse to $k_{\mathrm B}\ln\Omega$ on the uniform distribution; the Lagrange derivation of the canonical maximizer $p_i = e^{-\beta E_i}/Z$ and the concavity $\partial^2 S/\partial p_i^2 < 0$ ; additivity $S(\rho_A\otimes\rho_B) = S_A + S_B$ ; the three-state worked entropy $S/k_{\mathrm B} = 1.011$ with $D(p\Vert\text{unif}) = 0.088$ ; and the two-level entropy $S/k_{\mathrm B} = \ln 2$ at $x=0$ , $0.582$ at $x=1$ , $\to 0$ at large $x$ . The reduction to the Boltzmann entropy is owned by that article and recalled here by name. No numbers are fabricated.

Prerequisites. Microstates and Macrostates · Statistical Ensembles · Phase Space Density and the Distribution Function · Fundamental Postulate of Equal A Priori Probabilities. Leads to. Boltzmann Entropy · Equivalence and Distinction of Boltzmann and Gibbs Entropies · Liouville's Theorem · Ergodic Theory and Mixing.

Is the Gibbs entropy more fundamental than the Boltzmann entropy, or the other way around?

The Gibbs entropy is more general: it is defined for any distribution over microstates, while the Boltzmann entropy $k_{\mathrm B}\ln\Omega$ is the value Gibbs takes on the uniform (microcanonical) distribution. So Boltzmann is a special case of Gibbs. That said, "fundamental" cuts both ways — Boltzmann's is the one that connects most directly to a physical count of states and to the second law's arrow, while Gibbs's is the one that works in a heat bath and underlies the variational principle. The comparison article treats exactly when and why they agree or differ.

If the fine-grained Gibbs entropy never changes, how can there be a second law at all?

The second law lives in the coarse-grained entropy. The exact fine-grained $-k_{\mathrm B}\int\rho\ln\rho\,\mathrm d\Gamma$ is conserved by the reversible Liouville dynamics — no information is truly lost. But any real observation has finite resolution: average $\rho$ over small phase-space cells and, as mixing shreds $\rho$ into ever-finer filaments, the cell-averaged distribution relaxes to uniform and its entropy climbs to the equilibrium value $k_{\mathrm B}\ln\Omega$ . Irreversibility is lost information about fine structure, not lost reversibility. This is developed fully in the statistical-irreversibility article.

Why is the canonical distribution a maximum of entropy rather than a minimum?

Because the Gibbs functional is strictly concave: its second derivative with respect to each $p_i$ is $-k_{\mathrm B}/p_i < 0$ . A concave function on the convex set of probability distributions (a simplex, possibly sliced by the energy constraint) has a unique interior stationary point, and concavity guarantees it is the global maximum. So maximizing $-\sum p\ln p$ at fixed $\langle E\rangle$ has exactly one answer, the Boltzmann distribution, and it is a genuine maximum — equilibrium is the state of maximum entropy consistent with the known mean energy.

Where does temperature come from in the maximization? I only imposed a mean-energy constraint.

Temperature enters as the Lagrange multiplier conjugate to the energy constraint. When you maximize $-\sum p\ln p$ subject to $\sum p_iE_i = U$ , the multiplier that enforces that constraint is $\beta$ , and it appears in the answer as $p_i \propto e^{-\beta E_i}$ . Identifying $\beta = 1/k_{\mathrm B}T$ is what defines the thermodynamic temperature of the resulting distribution. So temperature is not put in by hand — it is the "price" of the mean-energy constraint, exactly analogous to how a chemical potential emerges as the multiplier for a particle-number constraint in the grand canonical ensemble.

Does the continuum Gibbs entropy really need the h³ᴺ and N! inside the logarithm?

Yes, for the same reasons as in the Boltzmann count. Without $h^{3N}$ the argument of the log $\rho h^{3N}N!$ would be dimensionful (since $\rho$ has units of inverse phase-space volume), making the "entropy" depend on your choice of units — physically meaningless. Without $N!$ the entropy is non-extensive and you resurrect the Gibbs paradox for identical particles. The combination $\rho h^{3N}N!$ is precisely the dimensionless local microstate occupation, so the continuum and discrete Gibbs entropies agree, and both match the Boltzmann count on the uniform distribution.

How is the Gibbs entropy related to the von Neumann entropy in quantum mechanics?

They are the same idea, one level up. Replace the classical distribution $\{p_i\}$ by a density matrix $\hat\rho$ and the Gibbs entropy $-k_{\mathrm B}\sum p_i\ln p_i$ becomes the von Neumann entropy $-k_{\mathrm B}\mathrm{Tr}(\hat\rho\ln\hat\rho)$ . When $\hat\rho$ is diagonal (a classical mixture) the two coincide, with the $p_i$ being the eigenvalues of $\hat\rho$ . The quantum version inherits every property discussed here — non-negativity, the $k_{\mathrm B}\ln W$ bound, concavity, additivity for product states — and adds genuinely quantum features (entanglement entropy, subadditivity), making it the central quantity of quantum information and quantum thermodynamics.