We find that the approximation leads to multiple insights on how to improve OGD, generalisation, knowledge transferability, curriculum learning and convergence. Also, our analysis is based solely on convex optimisation, which surprisingly leads us to results from other works, which considered other approximations for overparameterized neural networks. Finally, the experiments with non-overparametrized neural networks concur with our theoretical findings.

#### How are the NTKs different between SGD and OGD ?

We note that the key difference comes from the feature map, which is projected with $T$ to a feature space orthogonal to ${\mbox{$\mathbb{E}$}}$. In practice, the size of the data is much smaller than the number of the parameters, therefore, the dimension of ${\mbox{$\mathbb{E}$}}$ is negligible in comparison with the $d$, the dimension of the parameter space of the neural net, independently of the NTK regime. It could be interesting to understand more in depth the difference between the kernels of SGD and OGD. Their formulation looks similar to Von Neumann Kernels, which may be a possible direction to investigate.

Conclusion {#sec:conclusion-and-outlook}
==========

In this paper, we present an approach to study the properties of OGD for Continual Learning theoretically. Through this approach, we present in a closed-form how the 