Exercise 3.5: Equivalent Forms of Entropy

chapter 3

The \(\phi\)-entropy is defined by

\begin{equation} \Hb_\phi(X) = \E[\phi(X)] - \phi(\E[X]). \end{equation}

Let \(L\) be a linear function. Then

\begin{align} \Hb_{\phi + L}(X) &= \E[\phi(X)] + \E[L(X)] - \phi(\E[X]) - L(\E[X]) \newline &= \E[\phi(X)] - \phi(\E[X]) \newline &= \Hb_{\phi}(X). \end{align}

For example, the entropy defined by \(\phi(u) = u \log u - u\) gives the same the entropy as defined by \(\phi(u) = u \log u\).