vector or the spectral norm of a matrix, and $\|\cdot\|_\mathrm{F}$ to denote the Frobenius norm of a matrix. We use $\langle\cdot,
\cdot\rangle$ for the Euclidian dot product, and $\langle\cdot,\cdot\rangle_{{\mbox{$\mathcal{H}$}}}$ the dot product in the Hilbert space ${\mbox{$\mathcal{H}$}}$. We index the the task ID by $\tau$. The $ \leq $ operator if used with matrices, corresponds to the partial ordering over symetric matrices. We denote ${\mbox{$\mathbb{N}$}}$ the set of natural numbers, ${\mbox{$\mathbb{R}$}}$ the space of real numbers and ${\mbox{$\mathbb{N}$}}^{\star}$ for the set ${\mbox{$\mathbb{N}$}}\smallsetminus\{0\}$. We use $\oplus$ to refer to the direct sum over Euclidian spaces.

Continual Learning
------------------

Continual learning considers a series of tasks $\{{\mbox{$\mathcal{T}$}}_1, {\mbox{$\mathcal{T}$}}_2, \ldots\}$, where each task can be viewed as a separate supervised learning problem. Similarly to online learning, data from each task is revealed only once. The goal of continual learning is to model each task accurately with a single model. The challenge is to achieve a good performance on the new tasks, while retaining knowledge from the previous tasks [@v2018variational].

We assume the data from each task ${\mbox{$\mathcal{T}$}}_\tau$, $\tau \in {\mbox{$\mathbb{N}$}}^\star$, is drawn from a distribution ${\mbox{$\mathcal{D}$}}_\tau$. Individual samples are denoted $({\mbox{${\mbox{$\mathbf{x}$}}$}}_{\tau, i}, y_{\tau, i})$, where $i \in [n_\tau]$. Also, we only consider the 