1 \rightarrow \tau} &= f_{\tau-1}^\star({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau), &\\
        \phi_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}) &= \nabla_{{\mbox{${\mbox{$\mathbf{w}$}}$}}} f_{\tau-1}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}), &\\
        {\mbox{${\mbox{$\mathbf{H}$}}$}}_{\tau, \lambda} &= k_\tau({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau, {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau) + \lambda^2 {\mbox{${\mbox{$\mathbf{I}$}}$}}, &\\
        {\widetilde{\phi}}_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}) &=
        \begin{cases}
            \phi_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}) \text{\quad $\mathrm{for SGD}$,}\\
            {\mbox{${\mbox{$\mathbf{T}$}}$}}_\tau \phi_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}) \text{\quad $\mathrm{for OGD}$.}
        \end{cases}
        \intertext{and $\{{\mbox{${\mbox{$\mathbf{T}$}}$}}_{\tau}, \tau \in [T]\}$ are proxy matrices for the analysis.}
    \end{aligned}$$

The theorem describes how the model $f_\tau^\star$ evolves across tasks. The theorem is recursive because the learning is incremental. For a given task ${\mbox{$\mathcal{T}$}}_\tau$, $f_{\tau-1}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}})$ is the knowledge acquired by the agent up to the task ${\mbox{$\mathcal{T}$}}_{\tau-1}$. At this stage, the model only fits the residual ${\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau ={\mbox{${\mbox{$\mathbf{y}$}}$}}_\tau - {\mbox{${\mbox{$\mathbf{y}$}}$}}_{\tau - 1 \rightarrow \tau}$, which complements the knowledge acquired through previous tasks. This residual is also a proxy for task similarity. If the tasks are identical, the residual is equal to zero. The knowledge increment is captured by the term: $k_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}, {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau)^T (k_\tau({\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau , {\mbox{${\mbox{$\mathbf{X}$}}$}}_\tau) +
\lambda_\tau^2 {\mbox{${\mbox{$\mathbf{I}$}}$}})^{-1} {\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau$. Finally, task similarity is computed with respect to the most recent feature map ${\widetilde{\phi}}_\tau$, and $k_\tau$ is the NTK with respect to the feature map ${\widetilde{\phi}}_\tau$.

The recursive relation from Theorem \[thm:conv\] can also be written as a linear combination of kernel regressors as follows: $$\begin{aligned}
        f_\tau^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}) &= \sum_{k=1}^\tau {\mbox{$\tilde{f}$}}_{k}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}),
        \intertext{where}
        {\mbox{$\tilde{f}$}}_{k}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}) &= k_k({\mbox{${\mbox{$\mathbf{x}$}}$}}, {\mbox{${\mbox{$\mathbf{X}$}}$}}_k)^T (k_k({\mbox{${\mbox{$\mathbf{X}$}}$}}_k 