, {\mbox{${\mbox{$\mathbf{X}$}}$}}_k) + \lambda_k^2 {\mbox{${\mbox{$\mathbf{I}$}}$}})^{-1} {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_k.
    \end{aligned}$$

#### Proof Sketch:

We prove Theorem \[thm:conv\] by induction. We rewrite the loss function as a regression on the residual ${\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau$ instead of ${\mbox{${\mbox{$\mathbf{y}$}}$}}_\tau$. Then, we rewrite the optimisation objective as an unconstrained strongly convex optimisation problem. Finally, we compute the unique solution in a closed form. The full proof is presented in App. \[subsec:conv-multiple-proof\].

Distance from Initialisation {#subsec:distance-from-initialisation}
----------------------------

As described in Sec. \[subsec:conv\], ${\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau$ is a residual. It is equal to zero if the model makes perfect predictions on the next task ${\mbox{$\mathcal{T}$}}_\tau$. The more the next task ${\mbox{$\mathcal{T}$}}_\tau$ is different, the further the neural network needs to move from its previous state in order to fit it. Corollary \[cor:weight-dist\] tracks the distance from initialisation as a function of task similarity.

\[cor:weight-dist\] For SGD, and for OGD under the additional assumption that $\{{\mbox{${\mbox{$\mathbf{T}$}}$}}_{\tau}, \tau \in [T]\}$ are orthonormal, $$\begin{aligned}
        \norm{{\mbox{${\mbox{$\mathbf{w}$}}$}}_{\tau + 1}^\star - {\mbox{${\mbox{$\mathbf{w}$}}$}}_\tau^\star}_\mathrm{F} &= \sqrt{{\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau^T {\mbox{${\mbox{$\mathbf{H}$}}$}}_{T, \lambda}^{-1} {\mbox{${\mbox{$\mathbf{H}$}}$}}_{\tau, 0}
        {\mbox{${\mbox{$\mathbf{H}$}}$}}_{T,
        \lambda}^{-1}
        {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau},
        \intertext{where}
        {\mbox{${\mbox{$\mathbf{H}$}}$}}_{\tau, \lambda} &= k_\tau({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau, {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau) + \lambda^2 {\mbox{${\mbox{$\mathbf{I}$}}$}}.
    \end{aligned}$$

The proof is presented in App. \[subsec:proof-weight-dist\]. The orthonormality assumption is not restrictive, since the set $\{{\mbox{${\mbox{$\mathbf{T}$}}$}}_{\tau}, \tau \in [T]\}$ is only a proxy for the analysis; indeed 