Exercise 15.14: Gaussian Distributions And Maximum Entropy

chapter 15

Let $\Q = \gauss (0, \sigma^2)$ and $\P \in \mathcal{Q}_{\sigma^2}$ be arbitrary. Then $0 \leq \KL (\P \, \| \, \Q) = - H(\P) - \int \! p \log q \, \sd \nu$ where the latter term equals \begin{align} - \int \! p \log q \, \sd \nu = \tfrac{1}{2} \log (2\pi\sigma^2) + \tfrac{1}{2\sigma^2} \V_{\P} (X) \, . \end{align} Hence $H(\P) \leq \tfrac{1}{2} [1 + \log (2 \pi \sigma^2)] = H(\Q)$ by $\V_{\P}(X) \leq \sigma^2$.