Exercise 3.3: Variational Representation of Entropy

chapter 3

Using that \(\psi(u) = e^{-u} - 1 + u\), expand

\begin{equation} \E[\psi(\lambda(X - t)) e^{\lambda X}] = e^{\lambda t} - \E[e^{\lambda X}] + \E[\lambda X e^{\lambda X}] - \lambda t \E[e^{\lambda X}]. \end{equation}

Set the derivative to zero:

\begin{equation} \lambda e^{\lambda t} - \lambda \E[e^{\lambda X}] = 0 \implies \lambda t = \log \E[e^{\lambda X}]. \end{equation}

Therefore,

\begin{equation} \min_{t \in \R}\, \E[\psi(\lambda(X - t)) e^{\lambda X}] = \E[\lambda X e^{\lambda X}] - \E[e^{\lambda X}] \log \E[e^{\lambda X}] = \Hb(e^{\lambda X}). \end{equation}