Exercise 15.2: Basics of Discrete Entropy

chapter 15

(a)

Since $- \log (z) \geq 1 - z$ \begin{align} H(\mathbb{Q}) \geq \E [1 - q(X)] \geq 0 \, . \end{align}

(b)

\begin{align} \E \bigl[\log \tfrac{1}{q(X)}\bigr] \leq \log \sum_{x \in \mathcal{X}} \tfrac{q(x)}{q(x)} = \log | \mathcal{X} | \end{align}