HDS

Exercise 15.4: More Properties of Shannon Entropy

chapter 15

Recall the definition (1)H(X|Y)=ElogdPX|Ydμ.

(a)

Use non-negativity of the mutual information: (2)H(X|Y)H(X)=ElogdPX|Y/dμdPX/dμ=EYKL(PX|Y,PX)0.

(b)

Follows from a direct computation: (3)H(X)+H(Y|X)=ElogdPXdμdPY|Xdμ=ElogdPX,Ydμ=H(X,Y).

(c)

Follows from (a) applied to the r.h.s. of (b).

Published on 29 January 2022.