in these works, the bounds were derived for supervised learning settings, and in some cases for two-layer RELU neural networks. Similarly, the bounds depend on the Gram matrix of the data, with the feature map corresponding to the NTK.

#### Proof Sketch:

The proof is presented in App. \[subsec:proof-gen-ogd\]. One challenge is that the function class is the set of linear combinations of kernel regressors (Theorem \[thm:conv\]). We state Lemma \[lem:rademacher-kernels\] to bound the Rademacher complexity for this function class. Then we derive bounds for the training error for each case in Theorem \[thm:gen-ogd\]. The first case is straightforward. For the second case, we use Lemma \[lem:train-unchanged\], then derive a similar proof to the first case. The third case presents some additional technical challenges. In order to derive the upper bounds, we draw a strong inspiration from @hu2019simple, and leverage several of their proof techniques and mathematical tools.

Distance from Initialisation through Tasks {#subsec:distance-from-initialisation-through-tasks}
------------------------------------------

Now, we state Lemma \[lem:rademacher-tasks\], which tracks the Rademacher complexity through tasks.

\[lem:rademacher-tasks\] Keeping the same notations and setting as Theorem \[thm:gen-ogd\], the Rademacher Complexity can be bounded as follows: $$\begin{aligned}
        \hat{{\mbox{$\mathcal{R}$}}}({\mbox{$\mathcal{F}$}}_{{\mbox{${\mbox{$\mathbf{B}$}}$}}_T}) &\leq \sum_{\tau=1}^T {\mathcal{O}}\left( \sqrt{\frac{{\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau^T (k_\tau({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau , {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau) +
        \lambda^2 {\mbox{${\mbox{$\mathbf{I}$}}$}})^{-1}  {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau }{n_\tau}} \right)  