Exercise 15.4: More Properties of Shannon Entropy

chapter 15

Recall the definition $$\begin{array}{}\text{(1)}& H(X|Y)=\mathbb{E}\log \frac{\text{d}{\mathbb{P}}_{X|Y}}{\text{d}\mu }.\end{array}$$

(a)

Use non-negativity of the mutual information: $$\begin{array}{}\text{(2)}& H(X|Y)-H(X)=\mathbb{E}\log \frac{\text{d}{\mathbb{P}}_{X|Y}/\text{d}\mu }{\text{d}{\mathbb{P}}_X/\text{d}\mu }={\mathbb{E}}_Y\mathrm{KL}({\mathbb{P}}_{X|Y},{\mathbb{P}}_X)\ge 0.\end{array}$$

(b)

Follows from a direct computation: $$\begin{array}{}\text{(3)}& H(X)+H(Y|X)=\mathbb{E}\log \frac{\text{d}{\mathbb{P}}_X}{\text{d}\mu }\frac{\text{d}{\mathbb{P}}_{Y|X}}{\text{d}\mu }=\mathbb{E}\log \frac{\text{d}{\mathbb{P}}_{X,Y}}{\text{d}\mu }=H(X,Y).\end{array}$$

(c)

Follows from (a) applied to the r.h.s. of (b).