Exercise 2.4: Sharp Sub-Gaussian Parameter for Bounded Random Variable
Consider a random variable \(X\) with mean \(\mu = \E [X]\), and such that, for some scalars \(b > a, X \in [a, b]\) almost surely.
(a)
To begin with, \(\psi(0) = \log 1 = 0\). For \(\psi'(0)\), we know that the derivative of the MGF equals \(\mu\), so
\begin{equation} \psi’(0) = \frac{\mu}{\E [e^{0 X}]} = \mu \,. \end{equation}
(b)
The identity for \(\psi''(\lambda)\) follows from the chain rule. For the upper bound, observe that we can define a new distribution \(Q_\lambda\) by taking \(e^{\lambda X} \, / \, \E [ e^{\lambda X} ]\) to be its Radon–Nikodym derivative (density) with respect to the distribution of \(X\). Hence establishing a bound on \(\psi''(\lambda)\) is equivalent to bounding the supremum over variances of random variables \(X_\lambda \sim Q_\lambda\).
Taking \(m := \frac{1}{2}(a + b)\), using that the mean minimises the mean squared error, and using that \(X_\lambda \in [a, b]\) a.s. for all \(\lambda\),
\begin{equation} \sup_\lambda \V (X_\lambda) = \sup_\lambda \E [ (X_\lambda - \E[X_\lambda])^2 ] \leq \sup_\lambda \E [ (X_\lambda - m)^2 ] \leq (b - m)^2 = \frac{(b - a)^2}{4} \, . \end{equation}
(c)
Taking a Taylor expansion of \(\psi(\lambda)\) at \(\lambda = 0\),
\begin{equation} \psi(\lambda) = \psi(0) + \lambda \psi’(0) + \frac{\lambda^2}{2} \psi’’(\e) \, , \end{equation}
for some \(\e \in (0, \lambda)\). Substituting the results from (a) and (b), \(\psi(\lambda) \leq \lambda \mu + \frac{\lambda^2}{2} \frac{(b - a)^2}{4}\) as desired.