Exercise 2.4: Sharp Sub-Gaussian Parameter for Bounded Random Variable

chapter 2

Consider a random variable $X$ with mean $\mu =\mathbb{E}[X]$, and such that, for some scalars $b>a,X\in [a,b]$ almost surely.

(a)

To begin with, $\psi (0)=\log 1=0$. For ${\psi }^{\prime }(0)$, we know that the derivative of the MGF equals $\mu$, so

$$\begin{array}{}\text{(1)}& {\psi }^{\prime }(0)=\frac{\mu }{\mathbb{E}[e^{0X}]}=\mu .\end{array}$$

(b)

The identity for ${\psi }^{″}(\lambda )$ follows from the chain rule. For the upper bound, observe that we can define a new distribution $Q_{\lambda }$ by taking $e^{\lambda X}/\mathbb{E}[e^{\lambda X}]$ to be its Radon–Nikodym derivative (density) with respect to the distribution of $X$. Hence establishing a bound on ${\psi }^{″}(\lambda )$ is equivalent to bounding the supremum over variances of random variables $X_{\lambda }\sim Q_{\lambda }$.

Taking $m:=\frac{1}{2}(a+b)$, using that the mean minimises the mean squared error, and using that $X_{\lambda }\in [a,b]$ a.s. for all $\lambda$,

$$\begin{array}{}\text{(2)}& \underset{\lambda }{sup}\mathbb{V}(X_{\lambda })=\underset{\lambda }{sup}\mathbb{E}[(X_{\lambda }-\mathbb{E}[X_{\lambda }]{)}^2]\le \underset{\lambda }{sup}\mathbb{E}[(X_{\lambda }-m{)}^2]\le (b-m{)}^2=\frac{(b-a{)}^2}{4}.\end{array}$$

(c)

Taking a Taylor expansion of $\psi (\lambda )$ at $\lambda =0$,

$$\begin{array}{}\text{(3)}& \psi (\lambda )=\psi (0)+\lambda {\psi }^{\prime }(0)+\frac{{\lambda }^2}{2}{\psi }^{″}(\epsilon ),\end{array}$$

for some $\epsilon \in (0,\lambda )$. Substituting the results from (a) and (b), $\psi (\lambda )\le \lambda \mu +\frac{{\lambda }^2}{2}\frac{(b-a{)}^2}{4}$ as desired.