Exercise 13.2: Prediction Error in Linear Regression

chapter 13

The linear regression estimate is given by \(\hat{\theta} = (X^\top X)^{-1} X^\top y\) where \(y \overset{d}{=} X \theta_\star + w\), \(w \sim \gauss(0, \sigma^2 I)\). Hence \begin{equation} \E [\| X \hat{\theta} - X \theta_\star \|_n^2] = \E [\| X (X^\top X)^{-1} X^\top w \|_n^2] = \frac{\sigma^2}{n} \tr (X (X^\top X)^{-1} X^\top) = \frac{\sigma^2}{n} \rank(X) \, . \end{equation}