reason is that OGD+ performs a uniform sampling across samples from all past tasks, considering its limited memory budget, and that the memory requirements increase quadratically through tasks, we expect an information loss with respect to the most recent tasks, since the corresponding storage is used by OGD+ to older tasks. OGD uses the equivalent storage for the most recent task.

![Test accuracy on the first 3 tasks of permuted MNIST, for SGD, OGD and OGD+. We report the dynamics of additional tasks in App. \[subsec:app-exp-gen-extended\], Fig. \[fig:app:gen\]. 15 different permutations are used, and the model is trained to classify MNIST digits for 5 epochs for each permutation. The y-axis is truncated for clarity. We report the mean and standard deviation over 5 independent runs. The test error is measured for every 50 mini-batch interval. The vertical dashed lines separate the 15 tasks. []{data-label="fig:ogd-plus-acc"}](figs/out/gen/out.png){width="40.00000%"}

Related works
=============

#### Continual Learning

Approaches to Continual Learning can be categorised into: regularization methods, memory based methods, and dynamic architectural methods. We refer the reader to the survey [@PARISI201954] for an extensive overview on the existing methods. The idea behind memory-based methods is to store data from previous tasks in a buffer of fixed 