which we quantify rigorously (Sec. \[sec:generalisation\], Theorem \[thm:gen-ogd\]).

3.  We prove that OGD is robust to forgetting with respect to the previous task only (Sec. \[sec:generalisation\], Lemma \[lem:train-unchanged\]).

4.  We build-up on this insight to propose OGD+ (Sec. \[sec:ogd-plus\], Alg. \[alg:ogd-plus-train\]), an extension of OGD, which we prove robust to catastrophic forgetting across an arbitrary number of tasks (Sec. \[sec:ogd-plus\], Lemma \[lem:train-unchanged-ogd-plus\]). We also prove tighter generalisation bounds than OGD (Sec. \[sec:ogd-plus\], Theorem \[thm:gen-ogd-plus\]).

5.  As a side result, we find that Lemma \[lem:rademacher-tasks\] also quantifies the impact of the learning curriculum on the generalisation error. We find that task dissimilarity impacts negatively generalisation, and that an ordering of tasks that minimises dissimilarity between neighbouring tasks leads to a tighter generalisation bound. (Sec. \[thm:gen-ogd\], Lemma \[lem:rademacher-tasks\]).

Even though the analysis relies on the assumption that the neural network is overparametrised, the analysis leads to practical insights to improve OGD, which led us to OGD+. We run experiments in the non-overparametrised setting, and show that OGD+ achieves state-of-the-art results in settings with large number of tasks (Sec. \[sec:experiments\]).

Preliminaries {#sec:ogd}
=============

#### Notation

We use bold-faced characters for vectors and matrices. We use $\norm{\cdot}$ to denote the Euclidian norm of a 