new tasks while retaining the knowledge acquired from previous tasks. Too much plasticity will lead to catastrophic forgetting, which means the degradation of the ability of the agent to perform the past tasks (@MCCLOSKEY1989109, @Ratcliff1990ConnectionistMO, @Goodfellow2013Forgetting). On the other hand, too much stability will hinder the agent from adapting to new tasks. Recent works on the Neural Tangent Kernel [@Jacot2018NTK] and on the convergence of Stochastic Gradient Descent for overparameterized neural networks [@Arora:2019:FineGrained] have unlocked powerful tools to analyze the training dynamics of over-parameterized neural networks. We leverage these theoretical findings in order to to prove guarantees on the convergence and the generalisation of the algorithm, Orthogonal Gradient Descent for Continual Learning [@farajtabar2019orthogonal].

Our contributions are summarized as follows:

1.  We provide closed form expressions of the functions learned across tasks. We find that they can be expressed as a linear combination of kernel regressors, over the previously seen tasks. The relationship also captures task similarity and the transferability of knowledge across tasks (Sec. \[sec:convergence\], Theorem \[thm:conv\]).

2.  We prove the first generalisation bound for continual learning with OGD, to our knowledge. We derive bounds for within-task and outside-task generalisation. We find that generalisation through time depends on task similarity, 