Exercise 2.13: Operations on Sub-Gaussian Variables
(a)
Use independence: \begin{equation} \E[e^{\lambda(X_1 + X_2)}] = \E[e^{\lambda X_1}]\E[e^{\lambda X_2}] \le e^{\frac12(\sigma_1^2 + \sigma_2^2)\lambda^2}. \end{equation}
(b)
Use the Cauchy–Schwarz inequality: \begin{equation} \E[e^{\lambda(X_1 + X_2)}] \le \E[e^{2\lambda X_1}]^{1/2}\E[e^{2\lambda X_2}]^{1/2} \le \parens{ e^{2 \sigma_1^2 \lambda^2} }^{1/2} \parens{ e^{2 \sigma_2^2 \lambda^2} }^{1/2} = e^{\frac12 2(\sigma_1^2 + \sigma_2^2)\lambda^2}. \end{equation}
(c)
Let \(p, q \in (1, \infty)\) be conjugate. Then Hölder’s inequality gives \begin{equation} \E[e^{\lambda(X_1 + X_2)}] \le \E[e^{p\lambda X_1}]^{1/p}\E[e^{q\lambda X_2}]^{1/q} \le \parens{ e^{\frac12 p^2 \sigma_1^2 \lambda^2} }^{1/p} \parens{ e^{\frac12 q^2 \sigma_2^2 \lambda^2} }^{1/q} = e^{\frac12 (p \sigma_1^2 + q\sigma_2^2)\lambda^2}. \end{equation} To get that \(X_1 + X_2 \in \SG((\sigma_1 + \sigma_2)^2)\), we need the following expression in the exponent: \begin{equation} (\sigma_1 + \sigma_2)^2 = (\sigma_1^2 + \sigma_1 \sigma_2) + (\sigma_2^2 + \sigma_1 \sigma_2) = \underbrace{(1 + \sigma_2/\sigma_1)}_p \sigma_1^2 + \underbrace{(1 + \sigma_1/\sigma_2)}_q \sigma_2^2, \end{equation} and one readily verifies that \(p\) and \(q\) are indeed conjugate. Alternatively, we can set \(q = p/(p - 1)\) and optimise over \(p\).
(d)
Use independence of \(X_1\) and \(X_2\) and the bound on the MGF of \(X_2\): \begin{equation} \E[e^{\lambda X_1 X_2}] \le \E[e^{\frac12 \lambda^2 X_1^2 \sigma_2^2}] \end{equation} We now convert \(X_1^2\) to \(X_1\) with a trick. Let \(Z \sim \Normal(0, 1)\). Then \(\E[e^{t Z}] = e^{\frac12 t^2}\). Hence, applying this equality in reverse, \begin{equation} \E[e^{\frac12 \lambda^2 X_1^2 \sigma_2^2}] = \E[e^{\lambda X_1 \sigma_2 Z}], \end{equation} and now we can make progress: \begin{equation} \E[e^{\lambda X_1 X_2}] \le \E[e^{\frac12 \lambda^2 \sigma_1^2 \sigma_2^2 Z^2}] = \frac{1}{\sqrt{1 - 2 \rho}} \end{equation} with \(\rho = \frac12 \lambda^2 \sigma_1^2 \sigma_2^2\), where the last equality follows from explicit calculation and requires that \(\rho < \frac12\). Hence, \begin{equation} \log \E[e^{\lambda X_1 X_2}] \le -\frac12 \log(1 - \lambda^2 \sigma_1^2 \sigma_2^2). \end{equation} To show the result, we show that \begin{equation} f(t) = -\frac12\log(1 - t) \le t \end{equation} for \(\abs{t} < \frac12\). This follows directly a Taylor expansion in combination with \begin{equation} f’(t) = \frac12 \frac{1}{1 - t} \le 1, \end{equation} using that \(\abs{t} < \frac12\).