Exercise 14.10: Hellinger Distance and Kullback–Leibler Divergence
Since \(\KL (P \| Q) = \infty\) unless \(P \ll Q\), we only need to consider the case where \(g(x) = 0\) implies \(f(x) = 0\) \(Q\)-a.s. In this case \begin{align} \KL (f \| g) &= 2 \int f \biggl(- \log \frac{\sqrt{g}}{\sqrt{f}} \biggr) \, \sd \mu \newline &\overset{\text{(i)}}{\geq} 2 \int f \biggl(1 - \frac{\sqrt{g}}{\sqrt{f}} \biggr) \, \sd \mu \newline &= \int f + g - 2 \sqrt{fg} \, \sd \mu \newline &= 2 H^2(f \| g) \, , \end{align} where (i) follows by \(x = e^{\log x} \geq 1 + \log x\).