tools from statistical physics (@Goldt2019Dynamics, @goldt2020modelling).

Also, Corrolary \[cor:weight-dist\] in Sec. \[sec:convergence\] can be seen as a generalisation of the Theorem 3 by @liu2019understanding. While we don’t make any assumptions on the neural network as opposed to @liu2019understanding, which studies two-layer RELU neural networks, our analysis is based on less fine-grained approximations. Finally, as explained in Sec. \[subsec:generalisation-ogd\], we used several proof techniques from @hu2019simple to prove the Theorems \[thm:conv\], \[thm:gen-ogd\] and \[thm:gen-ogd-plus\].

#### Transfer Learning

Several works have also recently started studying Transfer Learning from a theoretical perspective (@yu2020gradient, @liu2019understanding, @Nguyen2019UnderstandForgetting, @Achille2019task2vec).

#### Statistical Learning Theory

@pmlr-v54-alquier17a define a compound regret for lifelong learning, as the regret with respect to the oracle who would have known the best common representation $g$ for all tasks in advance. Another line of works addresses lifelong learning and meta-learning from a statistical learning theory perspective, they define and provide regret bounds for lifelong learning [@pmlr-v54-alquier17a], or study the sample complexity and convergence of meta-learning algorithms (@denevi2018incremental, @du2020fewshot, @ji2020multistep, @saunshi2020sample, @ltl_common_mean).

Discussion {#sec:discussion-limits}
==========

We discuss some assumptions and limits of our analysis.

#### Linearization

We considered a linear approximation of the neural network, which may not be valid in the non overparameterized regime. 