Quantum Joint, Conditional Entropy and Mutual Information

These entropies are the standard measuring instruments of modern quantum information. In quantum channel coding, the Lloyd–Shor–Devetak theorem sets the quantum capacity equal to the regularized coherent information $\max_\rho I_c(A\rangle B)$; the search for its single-letter form drives the study of degradable and complementary channels, and superactivation — two zero-capacity channels with positive joint capacity — is a statement about the non-additivity of $-S(A|B)$. In quantum key distribution, the duality $S(A|B)=-S(A|C)$ is the security statement: if Bob's conditional entropy about the key is low, Eve's ($C$) is forced high, and the secret-key rate is written in these terms via the Devetak–Winter formula. In many-body physics and holography, $I(A;B)$ across a boundary obeys area laws in gapped ground states; the Ryu–Takayanagi formula computes entanglement entropy geometrically, and strong subadditivity becomes a constraint on spacetime. Negative conditional entropy has been measured — photonic and NMR realizations of state merging demonstrate entanglement gain when $S(A|B)<0$. The open frontier is the capacity problem itself: no computable formula is known for the quantum capacity of a general channel, because no one has tamed the non-additivity of coherent information.

Start with the single fact that reorganizes everything: a pure entangled state is less uncertain as a whole than in either of its parts. If $|\psi\rangle_{AB}$ is pure, $S(A,B)=0$, yet tracing out $B$ leaves a mixed $\rho_A$ with $S(A)>0$. Classically, learning nothing about the whole while being uncertain about a part is impossible; for entanglement it is generic. The quantity that records it is the conditional entropy $S(A|B)=S(A,B)-S(B)$, driven below zero exactly when $B$ is entangled with $A$. The first figure puts real numbers on the smallest case — a Bell pair — showing the conditional-entropy bar dipping under the axis while the mutual-information bar overshoots the classical ceiling.

The Bell pair is one point; the whole story is a curve. Take the one-parameter family of pure states $\sqrt{p}\,|00\rangle+\sqrt{1-p}\,|11\rangle$ and slide $p$ from a product state ($p=0$ or $1$) to maximal entanglement ($p=\tfrac12$). Each marginal has spectrum $\{p,1-p\}$, so $S(A)=S(B)=H(p)$, the binary entropy, while $S(A,B)=0$ throughout. Mutual information $I(A;B)=2H(p)$ and conditional entropy $S(A|B)=-H(p)$ are perfect mirror images across the axis: everything the joint state hides below zero in $S(A|B)$ reappears, doubled, as shared correlation in $I(A;B)$. The magnitude $-S(A|B)=H(p)$ is the coherent information — the ebits a receiver holding $B$ could distil. The second figure traces both.

One theme underlies every proof below: purification simplifies. Any mixed $\rho_{AB}$ is the marginal of a pure $|\psi\rangle_{ABC}$, and for a pure global state complementary parts share entropy, $S(AB)=S(C)$ and $S(A)=S(BC)$ — which is what turns the Araki–Lieb inequality and the duality $S(A|B)=-S(A|C)$ into one-line consequences.

The starting point is a definition: apply von Neumann's entropy to the whole and to the parts. The joint entropy is the entropy of the global density operator, computed from its eigenvalues as for any single system.

where: $\rho_{AB}$ — density operator on $\mathcal{H}_A\otimes\mathcal{H}_B$; $\log$ — base-2 logarithm, so entropy is in bits (natural log gives nats); $\mathrm{Tr}$ — trace over the whole space. Reading: $S(A,B)$ is the eigenvalue entropy of the joint state, blind to the tensor split. The conditional entropy is defined by the chain rule — there is no averaged-state definition to fall back on — using the marginal $\rho_B=\mathrm{Tr}_A\,\rho_{AB}$. Keeping only this identity is what allows the quantity to go negative.

where: $\mathrm{Tr}_A$ — partial trace over $A$, producing the reduced state $\rho_B$; $S(B)=S(\rho_B)$. Reading: $S(A|B)$ is "the entropy $A$ adds once you already hold $B$", and for an entangled pair that increment is negative. Mutual information is the symmetric combination that survives from Shannon theory, equal both to the inclusion–exclusion form and to the marginal-minus-conditional form.

where: $I(A;B)$ — total (classical plus quantum) correlation; $S(A)=S(\rho_A)$ with $\rho_A=\mathrm{Tr}_B\,\rho_{AB}$. Reading: it counts how much less uncertain the pair is than the sum of its parts. The single most useful re-expression identifies mutual information as a quantum relative entropy against the product reference; non-negativity and monotonicity then come for free.

where: $S(\rho\|\sigma)=\mathrm{Tr}\,\rho(\log\rho-\log\sigma)$ is the quantum relative entropy. Reading: as a "distance" from $\rho_{AB}$ to the product state, $I(A;B)\ge 0$, zero exactly when $\rho_{AB}=\rho_A\otimes\rho_B$. Non-negativity of $I(A;B)$ is subadditivity; the matching lower bound on the joint entropy is the Araki–Lieb triangle inequality. Together they pin $S(A,B)$ between two marginal-dependent walls.

where: the right inequality is subadditivity, the left is Araki–Lieb (proved via purification). Reading: equivalently $-S(A)\le S(A|B)\le S(A)$ — the conditional entropy is squeezed symmetrically about zero, not floored at it. The deepest relation is strong subadditivity, the Lieb–Ruskai theorem on a tripartite state $\rho_{ABC}$; everything structural in quantum Shannon theory is a corollary.

where: $S(A,B,C)$ — entropy of the tripartite state; $S(B)$ — entropy of the central marginal. Reading: equivalently $I(A;BC)\ge I(A;B)$ and $S(A|BC)\le S(A|B)$ — discarding a subsystem never increases correlations, and conditioning on more never increases conditional entropy. The negative of conditional entropy earns its own name and its own operational life as the central rate in quantum channel coding.

where: $I_c(A\rangle B)$ — coherent information; the regularized maximum of $I_c$ over channel inputs is the quantum capacity (the LSD theorem). Reading: wherever $S(A|B)<0$ there is positive coherent information, and quantum information can be sent. Finally, purification gives a duality with no classical analogue: the sign of conditional entropy is exclusive between the two complementary partners of a pure global state.

where: $|\psi\rangle_{ABC}$ — pure state with $\rho_{AB}=\mathrm{Tr}_C\,|\psi\rangle\!\langle\psi|$. Reading: $A$ can have positive conditional entropy with at most one of $B,C$ — a monogamy-flavoured constraint that underlies the security of key distribution.

Evaluating the three quantities on canonical states fixes their range and exposes exactly where the classical intuition survives and where it breaks.

1 · Conditional entropy and mutual information of an entangled pair (fully worked)

Take the pure state $|\psi\rangle=\sqrt{p}\,|00\rangle+\sqrt{1-p}\,|11\rangle$. Compute $S(A,B)$, $S(A)$, $S(A|B)$, $I(A;B)$, and the coherent information $I_c(A\rangle B)$, first for the Bell case $p=\tfrac12$ and then for $p=0.8$.

Step 1. The global state is pure, so $S(A,B)=S(\rho_{AB})=0$ for every $p$. This already guarantees $S(A|B)=-S(B)\le 0$ and $I(A;B)=S(A)+S(B)$.

Step 2. Trace out $B$: $\rho_A=p\,|0\rangle\!\langle 0|+(1-p)\,|1\rangle\!\langle 1|$, spectrum $\{p,1-p\}$, so $S(A)=S(B)=H(p)=-p\log_2 p-(1-p)\log_2(1-p)$.

Step 3. Bell case $p=\tfrac12$: $H(\tfrac12)=1$ bit, so $S(A)=S(B)=1$. Then $S(A|B)=S(A,B)-S(B)=0-1=\mathbf{-1\ bit}$, $I(A;B)=S(A)+S(B)-S(A,B)=1+1-0=\mathbf{2\ bits}$, and $I_c(A\rangle B)=-S(A|B)=\mathbf{1\ bit}$. This saturates every quantum ceiling for two qubits: maximal $I=2\log 2$ and maximal negativity $S(A|B)=-\log 2$.

Step 4. Non-maximal case $p=0.8$: $H(0.8)=-0.8\log_2 0.8-0.2\log_2 0.2=0.2575+0.4644=\mathbf{0.7219\ bit}$. Hence $S(A)=S(B)=0.7219$, $S(A|B)=-0.7219$ bit, $I(A;B)=2H(0.8)=\mathbf{1.4439\ bits}$, and $I_c(A\rangle B)=\mathbf{0.7219\ bit}$.

Step 5. Operationally: in state merging a receiver holding $B$ absorbs $A$ using $S(A|B)<0$ — sending no qubits and distilling $I_c(A\rangle B)$ ebits per copy ($1$ ebit for the Bell pair, $0.72$ ebit at $p=0.8$), while $I=2H(p)$ is the total correlation to be paid for in bringing the shares together. Bell: $S(A|B)=-1$, $I=2$, $I_c=1$; at $p=0.8$: $S(A|B)=-0.72$, $I=1.44$, $I_c=0.72$ (bits).

2 · The Holevo quantity of a two-state ensemble (fully worked)

Alice encodes a classical bit by sending either $|0\rangle$ or $|+\rangle$ with equal probability. Form the classical–quantum state $\rho_{XB}=\tfrac12|0\rangle\!\langle 0|_X\otimes|0\rangle\!\langle 0|+\tfrac12|1\rangle\!\langle 1|_X\otimes|+\rangle\!\langle +|$ and compute its mutual information $I(X;B)$ — the Holevo quantity $\chi$ that bounds how much classical information a measurement of $B$ can recover.

Step 1. For a classical–quantum state the mutual information equals the Holevo $\chi=S(\bar\rho)-\sum_x p_x S(\rho_x)$, with average state $\bar\rho=\tfrac12|0\rangle\!\langle 0|+\tfrac12|+\rangle\!\langle +|$. Why: the classical register $X$ has no coherences, so $S(X)=1$, $S(XB)=1+\sum_x p_x S(\rho_x)$, and the definition collapses to the Holevo form.

Step 2. Each signal state is pure, so $S(|0\rangle)=S(|+\rangle)=0$ and the subtracted term vanishes: $\chi=S(\bar\rho)$.

Step 3. Build $\bar\rho=\tfrac12\begin{pmatrix}1&0\\0&0\end{pmatrix}+\tfrac12\cdot\tfrac12\begin{pmatrix}1&1\\1&1\end{pmatrix}=\begin{pmatrix}3/4&1/4\\1/4&1/4\end{pmatrix}$. Its eigenvalues solve $\lambda^2-\lambda+\tfrac18=0$, giving $\lambda=\tfrac{1\pm\sqrt{1/2}}{2}=\{0.8536,\;0.1464\}$.

Step 4. Evaluate the entropy: $\chi=-0.8536\log_2 0.8536-0.1464\log_2 0.1464=0.1950+0.4060=\mathbf{0.601\ bits}$.

Step 5. Interpret: even though Alice used two perfectly distinguishable labels, the non-orthogonal signals $|0\rangle$ and $|+\rangle$ (overlap $|\langle 0|+\rangle|^2=\tfrac12$) let Bob extract at most $0.601$ bit of the $1$ bit sent — the Holevo bound in action, and a direct corollary of strong subadditivity. $\chi=I(X;B)=0.601$ bit $<1$ bit.

3 · A classically correlated state hits the classical ceiling

Compare the Bell pair with the separable state $\rho_{AB}=\tfrac12|00\rangle\!\langle 00|+\tfrac12|11\rangle\!\langle 11|$.

Step 1. The eigenvalues of $\rho_{AB}$ are $\{\tfrac12,\tfrac12\}$, so $S(A,B)=1$ bit, and both marginals are maximally mixed, $S(A)=S(B)=1$ bit. Hence $S(A|B)=1-1=\mathbf{0}$ and $I(A;B)=1+1-1=\mathbf{1\ bit}=\log d$ — conditional entropy on the classical floor, mutual information on the classical ceiling.

Step 2. The Bell state $\tfrac{1}{\sqrt2}(|00\rangle+|11\rangle)$ has the same marginals but $S(A,B)=0$, driving $S(A|B)$ to $-1$ and $I$ to $2$. Same local descriptions; entanglement alone doubles the correlation and pushes conditional entropy from $0$ to $-1$ bit.

The idea that "the entropy of a part can exceed the entropy of the whole" took decades to be seen as a resource rather than a paradox. It runs from von Neumann's 1932 definition of quantum entropy, through the hard analysis of Lieb and Ruskai, to a clean operational meaning only in this century.

Texts. Nielsen & Chuang, Quantum Computation and Quantum Information, ch. 11 (entropy and information) · Wilde, Quantum Information Theory, 2nd ed., ch. 11–13 (relative entropy, strong subadditivity, coherent information) · Preskill, Lecture Notes on Quantum Computation, ch. 10 · Watrous, The Theory of Quantum Information, ch. 5 · Kitaev, Shen & Vyalyi, Classical and Quantum Computation.

Primary sources. Lieb & Ruskai, J. Math. Phys. 14, 1938 (1973) — strong subadditivity · Cerf & Adami, Phys. Rev. Lett. 79, 5194 (1997) — negative conditional entropy · Horodecki, Oppenheim & Winter, Nature 436, 673 (2005) — quantum state merging · Devetak, IEEE Trans. Inf. Theory 51, 44 (2005) — coherent information as quantum capacity. The maximally entangled values $S(A,B)=0$, $S(A|B)=-\log d$, $I=2\log d$, the pure-family curves $I=2H(p)$ and $S(A|B)=-H(p)$, and the classically correlated $I=\log d$ are standard results. The Holevo example $\chi=0.601$ bit follows directly from the eigenvalues $\{0.8536,0.1464\}$ of $\bar\rho=\big(\begin{smallmatrix}3/4&1/4\\1/4&1/4\end{smallmatrix}\big)$.

Prerequisites. Von Neumann Entropy · Shannon Entropy and Classical Information Measures · Quantum Relative Entropy. Leads to. Strong Subadditivity of Quantum Entropy · Data-Processing Inequality · Holevo Bound · Complementary Channels and Degradability.

Ask the tutor: Ask me why a maximally entangled state pushes $S(A|B)$ all the way to $-\log d$, or to trace how strong subadditivity becomes the data-processing inequality.