overparametrised RELU neural network considered in the setting of @liu2019understanding, we can recover a similar approximation.

Generalisation of OGD for Continual Learning {#sec:generalisation}
============================================

In this section, we study the generalisation properties of OGD. First, we prove that OGD is robust to catastrophic forgetting with respect to the previous task (Lemma \[lem:train-unchanged\]). Then, we present the the main generalisation theorem for OGD (Thm. \[thm:gen-ogd\]). The theorem provides several insights on the relation between task similarity and generalisation. Finally, we present how the Rademacher complexity relates to task similarity across a large number of tasks (Lemma \[lem:rademacher-tasks\]). The lemma states that the more dissimilar tasks are, the larger the class of functions explored by the neural network, with high probability.

Memorisation property of OGD {#subsec:memorisation-property-of-ogd}
----------------------------

The key to obtaining tight generalisation bounds for OGD is Lemma \[lem:train-unchanged\].

\[lem:train-unchanged\] Given a task ${\mbox{$\mathcal{T}$}}_\tau$, for all ${\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, i} \in D_\tau$, a sample from the training data of the task ${\mbox{$\mathcal{T}$}}_\tau$, it holds that $$f_{\tau + 1}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, i}) = f_{\tau}^\star({\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, i}).$$

As motivated by @farajtabar2019orthogonal, the orthogonality of the gradient updates aims to preserve the acquired knowledge, by not altering the weights along relevant dimensions when learning new tasks. Lemma \[lem:train-unchanged\] states that 