the training error on the previous task is unchanged, when training with OGD. However, there are no guarantees that the knowledge from the tasks before the previous task is preserved.

The proof of Lemma \[lem:train-unchanged\] is presented in App. \[subsubsec:proof-train-unchanged\]

Generalisation of OGD for Continual Learning {#subsec:generalisation-ogd}
--------------------------------------------

Now, we state the main generalisation theorem for OGD, which provides within-task and outside-task generalisation bounds.

\[thm:gen-ogd\] Let $\{{\mbox{$\mathcal{T}$}}_1, \ldots {\mbox{$\mathcal{T}$}}_T\}$ be a sequence of tasks. Let be $\{{\mbox{$\mathcal{D}$}}_1, \ldots, {\mbox{$\mathcal{D}$}}_T\}$ the respective distributions over ${\mbox{$\mathbb{R}$}}^d \times \{-1, 1\}$. Let $\{({\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, i}, y_{\tau, i}), i \in [n_t], \tau \in [T] \}$ be i.i.d. samples from ${\mbox{$\mathcal{D}$}}_\tau, \tau \in
    [T] $. Denote ${\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau = ({\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, 1}, \ldots, {\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, n_\tau})$, ${\mbox{${\mbox{$\mathbf{y}$}}$}}_\tau = (y_{\tau, 1}, \ldots, y_{\tau,
    n_\tau})$. Consider the kernel ridge regression solution $f_T^\star$. Suppose that the kernel matrices satisfy

(k\_(\_, \_)) &= (n\_), \_[D\_T]{}(f\_T\^) & + &\
&\_[k=1]{}\^T (( ) + \_k), &\
\_[D\_[T-1]{}]{}(f\_T\^) & + &\
&\_[k=1]{}\^T (( ) + \_k), &\
\_[D\_]{}(f\_T\^) & + &\
&\_[k=1]{}\^T (( ) + \_k), &\
\_k &= () + 3 c , &\
\_[, ]{} &= k\_(\_, \_) + \^2 , &\
A\_&= &\
\_() &= \_\_ f\_[-1]{}\^(), &\
\_&= \_- \_[- 1 ]{}, &\
