binary classification setting for the sake of simplicity: ${\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau,i} \in {\mbox{$\mathbb{R}$}}^d$ and $y_{\tau, i} \in \{ -1, +1 \}$. We note that it does not restrict the scope of the analysis, which can be easily extended to multiclass settings.

OGD for Continual Learning {#subsec:ogd-for-continual-learning}
--------------------------

Let ${\mbox{$\mathcal{T}$}}_T$ the current task, where $T \in {\mbox{$\mathbb{N}$}}^\star$. For all $i \in [n_T]$, let ${\mbox{${\mbox{$\mathbf{v}$}}$}}_{T, i} = \nabla_\theta f^\star_{T-1}({\mbox{${\mbox{$\mathbf{x}$}}$}}_{T-1,i})$, which is the Jacobian of task ${\mbox{$\mathcal{T}$}}_T$. We define ${\mbox{$\mathbb{E}$}}_\tau = \mathrm{vec}( \{{\mbox{${\mbox{$\mathbf{v}$}}$}}_{\tau, i}, i \in [n_\tau] \} )$, which is the subspace induced by the Jacobian. The idea behind OGD is to update the weights along the projection of the gradient on the orthogonal space induced by the Jacobians over the previous tasks ${\mbox{$\mathbb{E}$}}_1 \oplus \ldots \oplus {\mbox{$\mathbb{E}$}}_{\tau-1} $. The update rule for the task ${\mbox{$\mathcal{T}$}}_T$ is as follows [@farajtabar2019orthogonal]: $${\mbox{${\mbox{$\mathbf{w}$}}$}}_T(t+1) = {\mbox{${\mbox{$\mathbf{w}$}}$}}_T(t) - \eta \Pi_{{\mbox{$\mathbb{E}$}}_{T-1}^\bot} \nabla_{{\mbox{${\mbox{$\mathbf{w}$}}$}}} {\mbox{$\mathcal{L}$}}^T_{\lambda}({\mbox{${\mbox{$\mathbf{w}$}}$}}_T(t)).$$

The intuition behind OGD is to [@farajtabar2019orthogonal].

To prevent over-fitting and guarantee the uniqueness of the global minimum in the Neural Tangent Kernel (NTK) regime, we apply a ridge regularization with a parameter $\lambda \in {\mbox{$\mathbb{R}$}}^{+}$. For a task ${\mbox{$\mathcal{T}$}}_\tau$, we write the corresponding loss as follows: $${\mbox{$\mathcal{L}$}}_\lambda^\tau({\mbox{${\mbox{$\mathbf{w}$}}$}}) = \sum_{i=1}^{n_\tau} (f_\tau({\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, i}) - y_{\tau, i})^2 + 