given sample ${\mbox{${\mbox{$\mathbf{x}$}}$}}$ from a task ${\mbox{$\mathcal{T}$}}_k$ where $k < \tau$, in order to keep the training error identical, the weights need to be updated along the directions that are orthogonal to *all the subsequent feature maps* of ${\mbox{${\mbox{$\mathbf{x}$}}$}}$. OGD only considers the feature map of the source task of the sample. Storing all the feature maps implies that the learned model back from task ${\mbox{$\mathcal{T}$}}_k$, can be recovered even after training on an arbitrary number of tasks.

In order to compute the feature maps with respect to the previous samples, OGD+ saves these samples in a dedicated memory, we call this storage the *samples memory*. This memory comes in addition to the orthonormal *feature maps memory*. The only role of the *samples memory* is to compute the feature maps. While the proofs below are under the assumption that the memory size is infinite, in the experiments, we keep a limited size for both memories.

1.  Initialize $S_{J}\leftarrow \{\}$ ; [ $S_{D}\leftarrow \{\}$]{}; ${\mbox{${\mbox{$\mathbf{w}$}}$}}\leftarrow {\mbox{${\mbox{$\mathbf{w}$}}$}}_0$

2.  

Memorisation Property of OGD+ {#subsec:memorisation-ogd-plus}
-----------------------------

The key to obtaining tight generalisation bounds for OGD+ is the Lemma \[lem:train-unchanged-ogd-plus\] below. It states that the training error across all previous tasks is unchanged, when training 