with OGD+.

\[lem:train-unchanged-ogd-plus\] Given a task ${\mbox{$\mathcal{T}$}}_\tau$, for all ${\mbox{${\mbox{$\mathbf{x}$}}$}}_{k, i} \in D_k$, a sample from the training data of a previous task, it holds that: $$f_{\tau}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}_{k, i}) = f_{k}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}_{k, i}).$$

The full proof of Lemma \[lem:train-unchanged-ogd-plus\] is presented in App . \[subsubsec:proof-train-unchanged-ogd-plus\].

Generalisation Guarantees for OGD+ {#subsec:generalisation-theorem}
----------------------------------

Now, we state the generalisation theorem for OGD+, which provides tighter generalisation bounds in comparison with Theorem \[thm:gen-ogd\], for OGD.

\[thm:gen-ogd-plus\] Under the same conditions as Theorem \[thm:gen-ogd\], for OGD+, it holds that, for all tasks ${\mbox{$\mathcal{T}$}}_\tau$, within-task and outside-task generalisation error can be bounded as follows $$\begin{aligned}
        {\mbox{$\mathcal{L}$}}_{D_\tau}(f_T^\star) &\leq \frac{\lambda}{2} \sqrt{ \frac{{\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau^T k_{T}
        ({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau,{\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau)^{-1} {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau}{n_\tau} } + &\\
        &\sum_{k=1}^T {\mathcal{O}}\left( \sqrt{\frac{{\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_k^T (k_t({\mbox{${\mbox{$\mathbf{X}$}}$}}_k , {\mbox{${\mbox{$\mathbf{X}$}}$}}_k) +
        \lambda^2 {\mbox{${\mbox{$\mathbf{I}$}}$}})^{-1}  {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_k }{n_k}} \right)  + \Delta_k, &\\
        \intertext{where}
        \Delta_k &= {\mathcal{O}}(\frac{1}{\sqrt{n_k}}) + 3 c \sqrt{\frac{\log(2/\delta) }{2n_k}},  &\\
        \phi_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}) &= {\mbox{${\mbox{$\mathbf{T}$}}$}}_\tau \nabla_{{\mbox{${\mbox{$\mathbf{w}$}}$}}} f_{\tau-1}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}), &\\
        {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau &= {\mbox{${\mbox{$\mathbf{y}$}}$}}_\tau - {\mbox{${\mbox{$\mathbf{y}$}}$}}_{\tau - 1 \rightarrow \tau}, &\\
        {\mbox{${\mbox{$\mathbf{y}$}}$}}_{\tau - 1 \rightarrow \tau} &= f_{\tau-1}^\star({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau).
    \end{aligned}$$

The generalisation bounds of Theorem \[thm:gen-ogd-plus\] are tighter than the generalisation bounds for OGD. The tightness of the bounds is a consequence of Lemma \[lem:train-unchanged-ogd-plus\]. The term that corresponds to the Rademacher complexity is unchanged, while the term that bounds the training error 