Exercise 12.18: RKHS-induced Semi-metrics
Assume \(\E_\P [\sqrt{k(X, X)}]\) and \(\E_\Q [\sqrt{k(Z, Z)}]\) are both finite so that \(\E [f(X)] \leq \| f \|_\Hb \E [\sqrt{k(X, X)}] < \infty\) for all \(f \in \Hb\). Then there exist \(\mu_\P , \mu_\Q \in \Hb\) such that \begin{align} \E_\P[f(X)] - \E_\Q[f(Z)] = \langle f , \mu_\P - \mu_\Q \rangle \, , \end{align} by the Riesz representation theorem. The supremum is thus attained by \(f = (\mu_\P - \mu_\Q) / \| \mu_\P - \mu_\Q \|_\Hb\) which yields \begin{align} \langle \mu_\P - \mu_\Q , \mu_\P - \mu_\Q \rangle &= \E_\P [\mu_\P(X) - \mu_\Q(X)] - \E_\Q [\mu_\P(Z) - \mu_\Q(Z)] \newline &= \E [k(X, X’) + k(Z, Z’) - 2 k(X, Z)] \, , \end{align} where we used \(\mu_\P (x) = \langle \mu_\P , k(\cdot , x) \rangle\) for all \(x\), and symmetry of \(k\).