lead to a lower upper bound.

#### Proof Sketch:

The proof techniques for Lemma \[lem:rademacher-tasks\] are exactly the same as the ones for Theorem \[thm:gen-ogd\]. The full proof is presented in Sec. \[subsubsec:bound-rademacher\].

OGD+: Learning without Forgetting {#sec:ogd-plus}
=================================

In the previous section, we demonstrated the limits of OGD, in terms of robustness to catastrophic forgetting on the long run. Now, we present OGD+, an extension of OGD, which we prove robust to catastrophic forgetting, across an arbitrary number of tasks (Lemma \[lem:train-unchanged-ogd-plus\]). Then, we prove tighter generalisation bounds compared to OGD (Theorem \[thm:gen-ogd-plus\]).

The OGD+ Algorithm {#subsec:ogd+-:-the-algorithm}
------------------

Algorithm \[alg:ogd-plus-train\] presents the OGD+ algorithm, we highlight the differences with OGD in red. The main difference is that OGD+ stores the feature maps with respect to the samples from previous tasks, in addition to the feature maps with respect to the samples from the current task, as opposed to OGD. This small change unlocks the proof of Lemma \[lem:train-unchanged-ogd-plus\] given below, which implies tighter bounds for Theorem \[thm:gen-ogd-plus\].

The idea behind OGD+ comes from the convergence Theorem (Sec. \[sec:convergence\], Thm. \[thm:conv\]). After training on a task ${\mbox{$\mathcal{T}$}}_\tau$, the learned model is a linear combination of the previous models. For a 