model evolves through tasks. This result leads to a generalisation theorem for OGD and provides insights on the transferability of knowledge across tasks and on the importance of task ordering and how the curriculum impacts generalisation. We also present OGD+, an extension of OGD, for which we derive stronger guarantees in terms of robustness to catastrophic forgetting and generalisation error. Finally, we observe that OGD+ achieves state-of-the-art performance for settings with a large number of tasks.

There are multiple avenues for future investigation. First, similarly to @Arora:2019:FineGrained and @liu2019understanding, a lower bound over the number of parameters to fall in the overparameterization regime is an important question, it would clarify the cases for which the theory is valid.

Another possible direction to investigate is the theoretical properties of the other Continual Learning training methods and the catastrophic forgetting heuristics, such as the ones proposed by @Nguyen2019UnderstandForgetting. A theroetical understanding of these algorithms and heuristics, even for asymptotic cases, may provide insights on their limits and directions of improvements, similarly to OGD+.

We also observed in the experiments that uniform sampling across all tasks for OGD+ would be the optimal sampling strategy for an average performance balance across all past tasks. 