Exercise 4.3: Maximum Likelihood and Uniform Laws
(a)
(i)
We have \begin{equation} p_\th(x) = \frac{e^{\th x}}{1 + e^{\th}} \end{equation} with \begin{equation} \E_\th[X] = \frac{e^{\th}}{1 + e^{\th}} \end{equation} Therefore,
\begin{equation} \E_{\th^*}\sbrac{\log \frac{p_{\th^*}(X)}{p_{\th}(X)}} = \log \frac{1 + e^{\th}}{1 + e^{\th^*}} + (\th^* - \th) \frac{e^{\th^*}}{1 + e^{\th^*}}. \end{equation}
(ii)
We have \begin{equation} p_\th(x) = \frac{1}{x!}e^{\th x} e^{-e^{\th}} \end{equation} with \begin{equation} \E_\th[X] = e^{-e^{\th}} \sum_{x=0}^\infty \frac{1}{x!}e^{\th x} = e^{-e^{\th}} (e^{e^{\th}})’ = e^{\th}. \end{equation} Therefore, \begin{equation} \E_{\th^*}\sbrac{\log \frac{p_{\th^*}(X)}{p_{\th}(X)}} = (\th^* - \th) e^{\th^*} - (e^{\th^*} - e^{\th}). \end{equation}
(iii)
This is a standard result: \begin{equation} \KL(p_{\th^*}, p_{\th}) = \frac12\parens{ \tr(\Sigma (\Sigma^*)^{-1}) + \log \frac{\det \Sigma}{\det \Sigma^*} + (\mu - \mu^*)^\T \Sigma^{-1}(\mu - \mu^*) - n }. \end{equation}