is tighter. It is also tighter than a standard supervised learning bound, because it captures the transferability of knowledge across tasks through the residual ${\mbox{${\mbox{$\mathbf{\tilde{y}}$}}$}}_\tau$, as opposed to the Supervised Learning only bounds, which would depend on ${\mbox{${\mbox{$\mathbf{y}$}}$}}_\tau$ instead.

#### Proof Sketch

The full proof is presented in App. \[subsec:proof-gen-ogd-plus\]. The proof is based on Lemma \[lem:train-unchanged-ogd-plus\] and Lemma \[lem:rademacher-kernels\]. The proof techniques are the same as the ones for Theorem \[thm:gen-ogd\].

Experiments {#sec:experiments}
===========

We performed experiments on the continual learning benchmark Permuted Mnist. The setup is the same as the one described in (@Goodfellow2013Forgetting, @kirkpatrick2017forgetting, @chaudhry2018efficient, @farajtabar2019orthogonal). However, in order to assess the robustness to catastrophic forgetting over a long sequence of tasks, we increase the size of the task sequence to 15. We generated 15 i.i.d. permutations of the MNIST pixels, then defined each task as a MNIST supervised learning problem with respect to each permutation, respectively. We trained the neural network sequentially on all tasks and tracked the validation loss on all previous tasks. We considered the same neural network architecture and mostly similar hyperparameters as [@farajtabar2019orthogonal]. The reproducibility details are presented in Appendix \[subsec:reproductibility\].

We report the results in Figure \[fig:ogd-plus-acc\], extended results are presented 