Exercise 14.5: Empirical Approximations of Kernel Integral Operators
(a)
We take it that that $\E[\hat T_k] = T_k$ means that $\E[\hat T_k(f)] = T_k(f)$ for all $f \in \Hb$: \begin{equation} \E[\hat T_k(f)] = \frac1n \sum_{i=1}^n \E[f(x_i)k(x_i, \vardot)] = \frac1n \sum_{i=1}^n T_k(f) = T_k(f). \end{equation}
(b)
Note that $\hat T_k - T_k$ is a bounded and self-adjoint operator on $\Hb$. Therefore, we have the following characterisation: \begin{equation} \norm{\hat T_k - T_k}_\Hb = \sup_{\norm{f}_\Hb \le 1} |\langle f, (\hat T_k - T_k) f\rangle_\Hb|. \end{equation} Note that \begin{equation} |\langle f, (\hat T_k - T_k) f\rangle_\Hb| = |\langle f, \hat T_k f\rangle_\Hb - \langle f, T_k f\rangle_\Hb| = |\norm{f}^2_n - \norm{f}^2_2|, \end{equation} which is immediately amenable to techniques from the chapter.
(c)
Bound \begin{equation} \norm{\hat T_k \phi_j - \mu_j \phi_j}_\Hb = \norm{\hat T_k \phi_j - T_k \phi_j}_\Hb \le \norm{\hat T_k - T_k}_\Hb \norm{\phi_j}_\Hb \end{equation} and note that \begin{equation} \norm{\phi_j}_\Hb = \frac{1}{\mu_j} \norm{T_k \phi_j}_\Hb \end{equation} where \begin{equation} \norm{T_k \phi_j}_\Hb^2 = \E[\phi_j(X) \phi_j(Y) k(X, Y)] \le \sqrt{\E[\phi_j^2(X) \phi^2_j(Y)] \E[k^2(X, Y)]} = \norm{k}_{L^2(\P \otimes \P)}^2. \end{equation} Therefore, \begin{equation} \norm{\hat T_k \phi_j - \mu_j \phi_j}_\Hb \le \frac{\norm{k}_{L^2(\P \otimes \P)}}{\mu_j} \norm{\hat T_k - T_k}_\Hb. \end{equation}