Exercise 3.9: Another Variational Representation

chapter 3

For \(g\) such that \(\E[e^{g(X)}] \le 1\), consider the measure \(\P_g(A) = \E[\ind_A e^{g(X)}]\). Then, under \(\P_g\), Jensen’s inequality still holds, so the entropy is still non-negative. Therefore,

\begin{align} &\Hb(e^{f(X)}) - \E[g(X)e^{f(X)}] \newline &\qquad= \E[(f(X) - g(X)) e^{f(X)}] - \E[e^{f(X)}] \log \E[e^{f(X)}] \newline &\qquad= \E_g[(f(X) - g(X)) e^{f(X) - g(X)}] - \E_g[e^{f(X) - g(X)}] \log \E_g[e^{f(X) - g(X)}] \newline &\qquad= \Hb^g(e^{f(X) - g(X)}) \newline &\qquad\ge 0, \end{align}

and it is clear that \(g(x) = f(x) - \log \E[e^{f(X)}]\) achieves equality.