in App . \[subsec:app-exp-gen-extended\]. The plot shows that OGD+ is more robust to catastrophic forgetting than OGD when the tasks occurrence difference is large. It also shows that OGD+ is equivalent to OGD when the tasks occurrence difference is small. These results concur with Lemma \[lem:ogd-train-gen-bounds\] which states that OGD is robust to catastrophic forgetting up to a single task ahead. forgetting up to a single task ahead. They also concur with Lemma \[lem:train-unchanged-ogd-plus\], which states that OGD+ is robust to catastrophic forgetting across any number of tasks. Two probable reasons that OGD+ is not perfectly prone to catastrophic forgetting in the experiment are the memory limit and the non-overparameterization of the neural network, while in Lemma \[lem:train-unchanged-ogd-plus\], we assumed that the memory is unlimited and that the neural network is overparameterized. The results in the Appendix \[subsec:app-exp-gen-extended\] concur with these hypotheses. Fig. \[fig:app:overparam\] shows that the test accuracy through time increases with overparameterization, in which case our approximation is more valid. Fig. \[fig:app:memory\] shows that the test accuracy also increases uniformly with the size of the memory.

We also observe that OGD outperforms OGD+ for shorter term tasks, on the long run (App . \[subsec:app-exp-gen-extended\], Fig. \[fig:app:gen\]). One probable 