size, which can then be reused during training on the current task (@chaudhry2018efficient, @vandeven2018generative). Dynamic architectural methods rely on growing architectures which keep the past knowledge fixed and store new knowledge in new components, such as new nodes, layers ... (@Lee2017Lifelong, @Schwarz2018ProgressCompress) The idea behind regularization methods is to regularize the objective in order to preserve the knowledge acquired from the previous tasks (@Kirkpatrick2016EWC, @Aljundi2017MAS, @farajtabar2019orthogonal, @Zenke2017SI).

#### Catastrophic Forgetting

Catastrophic Forgetting refers to the tendency of agents to “forget” the previous tasks over the course of training. It has proven to be a challenging problem, several heuristics were developed in order to characterise it (@ANS1997989, @Ans2000Refreshing, @Goodfellow2013Forgetting, @Robert1999Connectionist, @MCCLOSKEY1989109, @Robins1995Rehearsal, @Nguyen2019UnderstandForgetting).

#### Deep Learning Theory

Recent work have started to provide explanations about the mechanics of overparametrised Neural Networks. In their seminal work, @Du2018GD prove that Gradient Descent on multilayer overparametrised RELU neural networks achieve zero training error at the limit. These works have unlocked the analysis of several properties of Deep Neural Networks, in the context of various applications, such as Transfer Learning [@liu2019understanding], Noisy Supervision [@hu2019simple], Reinforcement Learning [@wang2020long] ... Another line of works provide closed form expressions of the training dynamics of overparameterized neural networks, leveraging 