@lee2019wide also showed that a network evolves as a linear model in the infinite width limit when trained on certain losses under gradient descent.

Throughout our analysis, we make the assumption that the neural network is overparameterized, and consider the linear approximation of the neural network around its initialisation: $$f^{(t)}({\mbox{${\mbox{$\mathbf{x}$}}$}}) \approx f^{(0)}({\mbox{${\mbox{$\mathbf{x}$}}$}}) + \nabla_{{\mbox{${\mbox{$\mathbf{w}$}}$}}} f^{(0)}({\mbox{${\mbox{$\mathbf{x}$}}$}})^T ({\mbox{${\mbox{$\mathbf{w}$}}$}}(t) - {\mbox{${\mbox{$\mathbf{w}$}}$}}(0)).$$

Convergence of OGD for Continual Learning {#sec:convergence}
=========================================

In this section, we derive a closed form expression for the learned models across tasks. We find a recursive kernel ridge regression relationship between the models across tasks. The result is presented in Theorem \[thm:conv\], a stepping stone towards proving the generalisation bound for OGD in Sec. \[sec:generalisation\].

Convergence Theorem {#subsec:conv}
-------------------

Now, we state the main result of this section:

\[thm:conv\] Let ${\mbox{$\mathcal{T}$}}_1,\ldots, {\mbox{$\mathcal{T}$}}_T$ be a sequence of tasks. Fix a learning rate sequence $(\eta_\tau)_{\tau \in [T]}$. If, for all $\tau$, the learning rate satisfies $$\eta_\tau < \frac{1}{\norm{k_\tau({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau, {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau)} +\lambda_\tau^2},$$ $$\begin{aligned}
        \intertext{then for all $\tau$, ${\mbox{${\mbox{$\mathbf{w}$}}$}}_\tau(t)$ converges linearly to a limit solution ${\mbox{${\mbox{$\mathbf{w}$}}$}}_\tau^\star$ such
that}
        f_\tau^\star({\mbox{${\mbox{$\mathbf{x}$}}$}})
        &= f_{\tau-1}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}) + k_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}, {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau)^T {\mbox{${\mbox{$\mathbf{H}$}}$}}_{\tau, \lambda_\tau}^{-1} {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau,
        \intertext{where}
        k_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}, {\mbox{${\mbox{$\mathbf{x}$}}$}}^\prime) &= {\widetilde{\phi}}_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}})^T {\widetilde{\phi}}_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}^\prime), &\\
        {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau &= {\mbox{${\mbox{$\mathbf{y}$}}$}}_\tau - {\mbox{${\mbox{$\mathbf{y}$}}$}}_{\tau - 1 \rightarrow \tau}, &\\
        {\mbox{${\mbox{$\mathbf{y}$}}$}}_{\tau - 