Exercise 15.11: Mixture Distribution and KL Divergence

chapter 15

Assuming existence of all integrals, \begin{align} \frac1M \sum_{j=1}^M \KL(\P_j, \Q) &= \frac1M \sum_{j=1}^M \int p_j \log \frac{p_j}{q} \isd \mu \newline &= \int \sbrac{\frac1M \sum_{j=1}^M p_j} \log \frac{1}{q} \isd \mu + \text{constant indep. of $q$} \newline &= \KL(\overline{P}, \Q) + \text{another constant indep. of $q$}. \end{align}