Exercise 15.14: Gaussian Distributions And Maximum Entropy

chapter 15

Let $\mathbb{Q}=\mathcal{N}(0,{\sigma }^2)$ and $\mathbb{P}\in {\mathcal{Q}}_{{\sigma }^2}$ be arbitrary. Then $0\le \mathrm{KL}(\mathbb{P}\Vert \mathbb{Q})=-H(\mathbb{P})-\int p\log q\text{d}\nu$ where the latter term equals $$\begin{array}{}\text{(1)}& -\int p\log q\text{d}\nu =\frac{1}{2}\log (2\pi {\sigma }^2)+\frac{1}{2{\sigma }^2}{\mathbb{V}}_{\mathbb{P}}(X).\end{array}$$ Hence $H(\mathbb{P})\le \frac{1}{2}[1+\log (2\pi {\sigma }^2)]=H(\mathbb{Q})$ by ${\mathbb{V}}_{\mathbb{P}}(X)\le {\sigma }^2$.