Elephant Neural Networks: Born to Be a Continual Learner
Catastrophic forgetting remains a significant challenge to continual learning
for decades. While recent works have proposed effective methods to mitigate
this problem, they mainly focus on the algorithmic side. Meanwhile, we do not
fully understand what architectural properties of neural networks lead to
catastrophic forgetting. This study aims to fill this gap by studying the role
of activation functions in the training dynamics of neural networks and their
impact on catastrophic forgetting. Our study reveals that, besides sparse
representations, the gradient sparsity of activation functions also plays an
important role in reducing forgetting. Based on this insight, we propose a new
class of activation functions, elephant activation functions, that can generate
both sparse representations and sparse gradients. We show that by simply
replacing classical activation functions with elephant activation functions, we
can significantly improve the resilience of neural networks to catastrophic
forgetting. Our method has broad applicability and benefits for continual
learning in regression, class incremental learning, and reinforcement learning
tasks. Specifically, we achieves excellent performance on Split MNIST dataset
in just one single pass, without using replay buffer, task boundary
information, or pre-training.
Qingfeng Lan, A. Rupam Mahmood
# Elephant Neural Networks:

###### Abstract

Catastrophic forgetting remains a significant challenge to continual learning for decades. While recent works have proposed effective methods to mitigate this problem, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse representations and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions, we can significantly improve the resilience of neural networks to catastrophic forgetting. Our method has broad applicability and benefits for continual learning in regression, class incremental learning, and reinforcement learning tasks. Specifically, we achieves excellent performance on Split MNIST dataset in just one single pass, without using replay buffer, task boundary information, or pre-training.

## 1 Introduction

One of the biggest challenges to achieving continual learning is the decades-old issue of _catastrophic forgetting_(French, 1999). Catastrophic forgetting stands for the phenomenon that artificial neural networks tend to forget prior knowledge drastically when learned with stochastic gradient descent algorithms on non-independent and identically distributed (non-iid) data. In recent years, researchers have made significant progress in mitigating catastrophic forgetting and proposed many effective methods, such as replay methods (Mendez et al., 2022), regularization-based methods (Riemer et al., 2018), parameter-isolation methods (Mendez et al., 2020), and optimization-based methods (Farajtabar et al., 2020). However, instead of alleviating forgetting by designing neural networks with specific properties, most of these methods circumvent the problem by focusing on the algorithmic side. _There is still a lack of full understanding of what properties of neural networks lead to catastrophic forgetting._ Recently, Mirzadeh et al. (2022) found that the width of a neural network significantly affects forgetting and they provided explanations from the perspectives of gradient orthogonality, gradient sparsity, and lazy training regime. Furthermore, Mirzadeh et al. (2022) studied the forgetting problem on large-scale benchmarks with various neural network architectures. They demonstrated that architectures can play a role that is as important as algorithms in continual learning.

_The interactions between continual learning and neural network architectures remain to be under-explored._ In this work, we aim to better understand and reduce forgetting by studying the impact of various architectural choices of neural networks on continual learning. Specifically, we focus on activation functions, one of the most important elements in neural networks. Experimentally, we study the effect of various activation functions (e.g., Tanh, ReLU, and Sigmoid) on catastrophic forgetting under the setting of continual supervised learning. Theoretically, we investigate the role of activation functions in the training dynamics of neural networks. Our analysis suggests that not only
sparse representations but also sparse gradients are essential for continual learning. Based on this discovery, we design a new class of activation functions, the elephant activation functions. Unlike classical activation functions, elephant activation functions are able to generate both sparse function values and sparse gradients that make neural networks more resilient to catastrophic forgetting. Correspondingly, the neural networks incorporated with elephant activation functions are named _elephant neural networks (ENNs)_. In streaming learning for regression, while many classical activation functions fail to even approximate a simple sine function, ENNs succeed by applying elephant activation functions. In class incremental learning, we show that ENNs achieve excellent performance on Split MNIST dataset in one single pass, without using any replay buffer, task boundary information, or pre-training. In reinforcement learning, ENNs improve the learning performance of agents under extreme memory constraints by alleviating the forgetting issue.

## 2 Investigating catastrophic forgetting via training dynamics

Firstly, we look into the forgetting issue via the training dynamics of neural networks. Consider a simple regression task. Let a scalar function \(f_{\mathbf{w}}(\mathbf{x})\) be represented as a neural network, parameterized by \(\mathbf{w}\), with input \(\mathbf{x}\). \(F(\mathbf{x})\) is the true function and the loss function is \(L(f,F,\mathbf{x})\). For example, for squared error, we have \(L(f,F,\mathbf{x})=(f_{\mathbf{w}}(\mathbf{x})-F(\mathbf{x}))^{2}\). At each time step \(t\), a new sample \(\{\mathbf{x}_{t},F(\mathbf{x}_{t})\}\) arrives. Given this new sample, to minimize the loss function \(L(f,F,\mathbf{x}_{t})\), we update the weight vector by \(\mathbf{w}^{\prime}=\mathbf{w}+\Delta_{\mathbf{w}}\) where \(\Delta_{\mathbf{w}}\) is the weight difference. With the stochastic gradient descent (SGD) algorithm, we have \(\mathbf{w}^{\prime}=\mathbf{w}-\alpha\nabla_{\mathbf{w}}L(f,F,\mathbf{x}_{t})\), where \(\alpha\) is the learning rate. So \(\Delta_{\mathbf{w}}=\mathbf{w}^{\prime}-\mathbf{w}=-\alpha\nabla_{\mathbf{w}}L(f,F,\mathbf{x}_{t}) =-\alpha\nabla_{f}L(f,F,\mathbf{x}_{t})\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}_{t})\). By Taylor expansion,

\[f_{\mathbf{w}^{\prime}}(\mathbf{x})-f_{\mathbf{w}}(\mathbf{x})=-\alpha\nabla_{f}L(f,F,\mathbf{x}_{ t})\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}_ {t})\right\rangle+O(\Delta_{\mathbf{w}}^{2}), \tag{1}\]

where \(\left\langle\cdot,\cdot\right\rangle\) denotes Frobenius inner product or dot product depending on the context. In this equation, since \(-\alpha\nabla_{f}L(f,F,\mathbf{x}_{t})\) is unrelated to \(\mathbf{x}\), we only consider the quantity \(\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x }_{t})\right\rangle\), which is known as the neural tangent kernel (NTK) (Jacot et al., 2018). Without loss of generality, assume that the original prediction \(f_{\mathbf{w}}(\mathbf{x}_{t})\) is wrong, i.e. \(f_{\mathbf{w}}(\mathbf{x}_{t})\neq F(\mathbf{x}_{t})\) and \(\nabla_{f}L(f,F,\mathbf{x}_{t})\neq 0\). To correct the wrong prediction while avoiding forgetting, we expect this NTK to satisfy two properties that are essential for continual learning:

**Property 2.1** (error correction).: _For \(\mathbf{x}=\mathbf{x}_{t}\), \(\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x }_{t})\right\rangle\neq 0\)._

**Property 2.2** (zero forgetting).: _For \(\mathbf{x}\neq\mathbf{x}_{t}\), \(\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x }_{t})\right\rangle=0\)._

In particular, Property 2.1 allows for error correction by optimizing \(f_{\mathbf{w}^{\prime}}(\mathbf{x}_{t})\) towards the true value \(F(\mathbf{x}_{t})\), so that we can learn new knowledge (i.e., update the learned function). If \(\left\langle\nabla_{\mathbf{w}}f(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}_{t}) \right\rangle=0\), we then have \(f_{\mathbf{w}^{\prime}}(\mathbf{x})-f_{\mathbf{w}}(\mathbf{x})\approx 0\), failing to correct the wrong prediction at \(\mathbf{x}=\mathbf{x}_{t}\). Essentially, Property 2.1 requires the gradient norm to be non-zero. On the other hand, Property 2.2 is much harder to be satisfied, especially for non-linear approximations. Essentially, to make this property hold, except for \(\mathbf{x}=\mathbf{x}_{t}\), the neural network \(f\) is required to achieve zero knowledge forgetting after one step optimization, i.e. \(\forall\mathbf{x}\neq\mathbf{x}_{t},f_{\mathbf{w}^{\prime}}(\mathbf{x})=f_{\mathbf{w}}(\mathbf{x})\). _It is the violation of Property 2.2 that leads to the forgetting issue._ For tabular cases (e.g., \(\mathbf{x}\) is a one-hot vector and \(f_{\mathbf{w}}(\mathbf{x})\) is a linear function), this property may hold by sacrificing the generalization ability of deep neural networks. In order to benefit from generalization, we propose Property 2.3 by relaxing Property 2.2.

**Property 2.3** (local elasticity).: \(\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x }_{t})\right\rangle\approx 0\) _for \(\mathbf{x}\) that is dissimilar to \(\mathbf{x}_{t}\) in a certain sense._

Property 2.3 is known as local elasticity (He and Su, 2020). A function \(f\) is locally elastic if \(f_{\mathbf{w}}(\mathbf{x})\) is not significantly changed, after the function is updated at \(\mathbf{x}_{t}\) that is dissimilar to \(\mathbf{x}\) in a certain sense. For example, we can characterize the dissimilarity with the \(2\)-norm distance. Although He and Su (2020) show that neural networks with nonlinear activation functions are locally elastic in general, there is a lack of theoretical understanding about the connection of neural network architectures and the degrees of local elasticity. In our experiments, we find that the degrees of local elasticity of classical neural networks are not enough to address the forgetting issue, as we will show next.
Understanding the success and failure of sparse representations

The above properties can help to deepen our understanding of the success and failure of sparse representations in continual learning. To be specific, we argue that sparse representations are effective to reduce forgetting in linear function approximations, but are less useful in non-linear function approximations.

It is well-known that deep neural networks can automatically generate effective representations (a.k.a. features) to extract key properties from input data. The ability to learn useful features helps deep learning methods achieve great success in many areas (LeCun et al., 2015). In particular, we call a set of representations sparse when only a small part of representations is non-zero for a given input. Sparse representations are shown to help reduce the forgetting problem and the interference issues in both continual supervised learning and reinforcement learning (Shen et al., 2021; Liu et al., 2019). Formally, let \(\mathbf{x}\) be an input and \(\phi\) be an encoder that transforms the input \(\mathbf{x}\) into its representation \(\phi(\mathbf{x})\). The representation \(\phi(\mathbf{x})\) is sparse when most of its elements are zeros.

First, consider the case of linear approximations. A linear function is defined as \(f_{\mathbf{w}}(\mathbf{x})=\mathbf{w}^{\top}\phi(\mathbf{x}):\mathbb{R}^{n}\mapsto\mathbb{R}\), where \(\mathbf{x}\in\mathbb{R}^{n}\) is an input, \(\phi:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}\) is a fixed encoder, and \(\mathbf{w}\in\mathbb{R}^{m}\) is a weight vector. Assume that the representation \(\phi(\mathbf{x})\) is sparse and non-zero (i.e., \(\|\phi(\mathbf{x})\|_{2}>0\)) for \(\mathbf{x}\in\mathbb{R}^{n}\). Next, we show that both Property 2.1 and Property 2.3 are satisfied in this case. Easy to know that \(\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x})=\phi(\mathbf{x})\). Together with Equation (1), we have

\[f_{\mathbf{w}^{\prime}}(\mathbf{x})-f_{\mathbf{w}}(\mathbf{x})=-\alpha\nabla_{f}L(f,F,\mathbf{x}_{ t})\,\phi(\mathbf{x})^{\top}\phi(\mathbf{x}_{t}). \tag{2}\]

By assumption, \(f_{\mathbf{w}}(\mathbf{x}_{t})\neq F(\mathbf{x}_{t})\) and \(\nabla_{f}L(f,F,\mathbf{x}_{t})\neq 0\). Then Property 2.1 holds since \(f_{\mathbf{w}^{\prime}}(\mathbf{x}_{t})-f_{\mathbf{w}}(\mathbf{x}_{t})=-\alpha\nabla_{f}L(f,F, \mathbf{x}_{t})\,\|\phi(\mathbf{x}_{t})\|_{2}^{2}\neq 0\). Moreover, when \(\mathbf{x}\neq\mathbf{x}_{t}\), it is very likely that \(\left\langle\phi(\mathbf{x}),\phi(\mathbf{x}_{t})\right\rangle\approx 0\) due to the sparsity of \(\phi(\mathbf{x})\) and \(\phi(\mathbf{x}_{t})\). Thus, \(f_{\mathbf{w}^{\prime}}(\mathbf{x})-f_{\mathbf{w}}(\mathbf{x})=-\alpha\nabla_{f}L(f,F,\mathbf{x}_ {t})\left\langle\phi(\mathbf{x}),\phi(\mathbf{x}_{t})\right\rangle\approx 0\) and Property 2.3 holds. We conclude that by satisfying Property 2.1 and Property 2.3, sparse representations successfully help mitigate catastrophic forgetting in linear approximations.

However, for non-linear approximations, sparse representations can no longer guarantee Property 2.3. Consider an MLP with one hidden layer \(f_{\mathbf{w}}(\mathbf{x})=\mathbf{u}^{\top}\sigma(\mathbf{V}\mathbf{x}+\mathbf{b}):\mathbb{R}^{n}\mapsto \mathbb{R}\), where \(\sigma\) is a non-linear activation function, \(\mathbf{x}\in\mathbb{R}^{n}\), \(\mathbf{u}\in\mathbb{R}^{m}\), \(\mathbf{V}\in\mathbb{R}^{m\times n}\), \(\mathbf{b}\in\mathbb{R}^{m}\), and \(\mathbf{w}=\{\mathbf{u},\mathbf{V},\mathbf{b}\}\). We compute the NTK in this case, resulting in the following lemma.

**Lemma 3.1** (NTK in non-linear approximations).: _Given a non-linear function \(f_{\mathbf{w}}(\mathbf{x})=\mathbf{u}^{\top}\sigma(\mathbf{V}\mathbf{x}+\mathbf{b}):\mathbb{R}^{n} \mapsto\mathbb{R}\), where \(\sigma\) is a non-linear activation function, \(\mathbf{x}\in\mathbb{R}^{n}\), \(\mathbf{u}\in\mathbb{R}^{m}\), \(\mathbf{V}\in\mathbb{R}^{m\times n}\), \(\mathbf{b}\in\mathbb{R}^{m}\), and \(\mathbf{w}=\{\mathbf{u},\mathbf{V},\mathbf{b}\}\). The NTK of this non-linear function is_

\[\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}_ {t})\right\rangle=\sigma(\mathbf{V}\mathbf{x}+\mathbf{b})^{\top}\sigma(\mathbf{V}\mathbf{x}_{t}+ \mathbf{b})+\mathbf{u}^{\top}\mathbf{u}(\mathbf{x}^{\top}\mathbf{x}_{t}+1)\sigma^{\prime}(\mathbf{V} \mathbf{x}+\mathbf{b})^{\top}\sigma^{\prime}(\mathbf{V}\mathbf{x}_{t}+\mathbf{b}),\]

_where \(\left\langle\cdot,\cdot\right\rangle\) denotes Frobenius inner product or dot product depending on the context._

The proof of this lemma can be found in Appendix B. Note that the encoder \(\phi\) is no longer fixed, and we have \(\phi_{\mathbf{\theta}}(\mathbf{x})=\sigma(\mathbf{V}\mathbf{x}+\mathbf{b})\), where \(\mathbf{\theta}=\{\mathbf{V},\mathbf{b}\}\) are learnable parameters. By Lemma 3.1,

\[\left\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}_ {t})\right\rangle=\phi_{\mathbf{\theta}}(\mathbf{x})^{\top}\phi_{\mathbf{\theta}}(\mathbf{x}_ {t})+\mathbf{u}^{\top}\mathbf{u}(\mathbf{x}^{\top}\mathbf{x}_{t}+1)\phi^{\prime}_{\mathbf{\theta}}( \mathbf{x})^{\top}\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x}_{t}). \tag{3}\]

Compared with the NTK in linear approximations (Equation (2)), Equation (3) has an additional term \(\mathbf{u}^{\top}\mathbf{u}(\mathbf{x}^{\top}\mathbf{x}_{t}+1)\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x} )^{\top}\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x}_{t})\), due to a learnable encoder \(\phi_{\mathbf{\theta}}\). With sparse representations, we have \(\phi_{\mathbf{\theta}}(\mathbf{x})^{\top}\phi_{\mathbf{\theta}}(\mathbf{x}_{t})\approx 0\). However, it is not necessary true that \(\mathbf{u}^{\top}\mathbf{u}(\mathbf{x}^{\top}\mathbf{x}_{t}+1)\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x })^{\top}\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x}_{t})\approx 0\) even when \(\mathbf{x}\) and \(\mathbf{x}_{t}\) are quite dissimilar, which violates Property 2.3.

To conclude, our analysis indicates that sparse representations alone are not effective enough to reduce forgetting in non-linear approximations.

## 4 Obtaining sparsity with elephant activation functions

Although Lemma 3.1 shows that the forgetting issue can not be fully addressed with sparse representations solely in deep learning methods, it also points out a possible solution: sparse gradients.
With sparse gradients, we could have \(\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x})^{\top}\phi^{\prime}_{\mathbf{\theta}}(\mathbf{x}_{t})\approx 0\). Together with sparse representations, we may still satisfy Property 2.3 in non-linear approximations, and thus reduce more forgetting. Specifically, we aim to design new activation functions to obtain both sparse representations and sparse gradients.

To begin with, we first define the sparsity of an function, which also applies to activation functions.

**Definition 4.1** (sparse function).: For a function \(\sigma:\mathbb{R}\mapsto\mathbb{R}\), we define the sparsity of function \(\sigma\) on input domain \([-C,C]\) as

\[S_{\epsilon,C}(\sigma)=\frac{|\{x\mid|\sigma(x)|\leq\epsilon,x\in[-C,C]\}|}{| \{x\mid x\in[-C,C]\}|}=\frac{|\{x\mid|\sigma(x)|\leq\epsilon,x\in[-C,C]\}|}{2C},\]

where \(\epsilon\) is a small positive number and \(C>0\). As a special case, when \(\epsilon\to 0^{+}\) and \(C\to\infty\), define

\[S(\sigma)=\lim_{\epsilon\to 0^{+}}\lim_{C\to\infty}S_{\epsilon,C}(\sigma).\]

We call \(\sigma\) a \(S(\sigma)\)-sparse function. In particular, \(\sigma\) is called a sparse function if it is a \(1\)-sparse function, i.e. \(S(\sigma)=1\).

**Remark 4.2**.: Easy to verify that \(0\leq S(\sigma)\leq 1\). The sparsity of a function shows the fraction of nearly zero outputs given a symmetric input domain. For example, both \(\mathrm{ReLU}(x)\) and \(\mathrm{ReLU}^{\prime}(x)\) are \(\frac{1}{2}\)-sparse functions. \(\mathrm{Tanh}(x)\) is a \(0\)-sparse function while \(\mathrm{Tanh}^{\prime}(x)\) is a sparse function. In the appendix, we summarize the results for common activation functions in Table 3 and visualize the activation functions with their gradients in Figure 5.

Next, we propose to use a novel class of bell-shaped activation functions, elephant activation functions 1. Formally, an elephant function is defined as

Footnote 1: We name this bell-shaped activation function as the _elephant_ function since the bell-shape is similar to the shape of an elephant (see Figure 1), honoring the work _The Little Prince_ by Antoine de Saint Exupery. This name also hints that this activation function empowers neural networks with continual learning ability, echoing the old saying that “an elephant never forgets”.

\[\mathrm{Elephant}(x)=\frac{1}{1+\left\lfloor\frac{x}{a}\right\rfloor^{d}}, \tag{4}\]

where \(a\) controls the width of the function and \(d\) controls the slope. We call a neural network that uses elephant activation functions an _elephant neural network (ENN)_. For example, for MLPs and CNNs, we have _elephant MLPs (EMLPs)_ and _elephant CNNs (ECNNs)_, respectively.

As shown in Figure 1, elephant activation functions have both sparse function values and sparse gradient values, which can also be formally proved.

**Lemma 4.3**.: \(\mathrm{Elephant}(x)\) _and \(\mathrm{Elephant}^{\prime}(x)\) are sparse functions._

Specifically, \(d\) controls the sparsity of gradients for elephant functions. The larger the value of \(d\), the sharper the slope of an elephant function and the sparser the gradient. On the other hand, \(a\) stands for the width of the function, controlling the sparsity of the function itself.

Next, we show that Property 2.3 holds under certain conditions for elephant functions.

Figure 1: (a) “My drawing was not a picture of a hat. It was a picture of a boa constructor digesting an elephant.” _The Little Prince_, by Antoine de Saint Exupery. (b) Elephant functions with \(a=2\) and various \(d\). (c) The gradient of elephant functions with \(a=2\) and various \(d\).
**Theorem 4.4**.: _Define \(f_{\mathbf{w}}(\mathbf{x})\) as in Lemma 3.1. Let \(\sigma\) be the elephant activation function with \(d\to\infty\). When \(|\mathbf{V}(\mathbf{x}-\mathbf{x}_{t})|>2a\mathbf{1}_{\text{m}}\), we have \(\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}),\nabla_{\mathbf{w}}f_{\mathbf{w}}(\mathbf{x}_{t}) \rangle=0\), where \(\succ\) denotes an element-wise inequality symbol and \(\mathbf{1}_{\text{m}}=[1,\cdots,1]^{\top}\in\mathbb{R}^{m}\)._

**Remark 4.5**.: Theorem 4.4 mainly proves that when \(d\to\infty\), Property 2.3 holds for an EMLP with one hidden layer. However, even when \(d\) is a small integer (e.g., 8), EMLPs still exhibit local elasticity, as we will show in the experiment section. The proof is in Appendix B.

## 5 Experiments

In this section, we first perform experiments in a simple regression task in the streaming learning setting. We will show that (1) sparse representations alone are not enough to address the forgetting issue, (2) ENNs can continually learn to solve regression tasks by reducing forgetting, and (3) ENNs are locally elastic even when \(d\) is a small integer. Next, we show that by incorporating elephant functions, ENNs are able to achieve better performance than several baselines in class incremental learning, without utilizing pre-training, replay buffer, or task boundaries. Finally, we apply ENNs in reinforcement learning (RL) and demonstrate ENNs' advantage in reducing catastrophic forgetting.

### Streaming learning for regression

In the real world, regression tasks are everywhere, from house price estimations (Madhuri et al., 2019) to stock predictions (Dase and Pawar, 2010), weather predictions (Ren et al., 2021), and power consumption forecasts (Dmitri et al., 2016). However, most prior continual learning methods are designed for classification tasks rather than regression tasks, although catastrophic forgetting frequently arises in regression tasks as well (He and Sick, 2021). In this section, we conduct experiments on a simple regression task in the streaming learning setting. In this setting, a learning agent is presented with one sample only at each time step and then performs learning updates given this new sample. Moreover, the learning process happens in a single pass of the whole dataset, that is, each sample only occurs once. Furthermore, the data stream is assumed to be non-iid. Finally, the evaluation happens after each new sample arrives, which requires the agent to learn quickly while avoiding catastrophic forgetting. Streaming learning methods enable real-time adaptation; thus they are more suitable in real-world scenarios where data is received in a continuous flow.

We consider streaming learning for approximating a sine function. In this task, there is a stream of data \((x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{n},y_{n})\), where \(0\leq x_{1}<x_{2}<\cdots<x_{n}\leq 2\) and \(y_{i}=\sin(\pi x_{i})\). We set \(n=200\) in our experiment. The learning agent \(f\) is an MLP with one hidden layer of size \(1,000\). At each time step \(t\), the learning agent only receives one sample \((x_{t},y_{t})\). We minimize the square error loss \(l_{t}=(f(x_{t})-y_{t})^{2}\) with Adam optimizer (Kingma and Ba, 2015), where \(f(x_{t})\) is the agent's prediction. During an evaluation, the agent performance is measured by the mean square error (MSE) on a test dataset with \(1,000\) samples, where the inputs are evenly spaced over the interval \([0,2]\).

We compare our method (EMLP) with two kinds of baselines. One is an MLP with classical activation functions, including \(\mathrm{ReLU}\), \(\mathrm{Sigmoid}\), \(\mathrm{ELU}\), and \(\mathrm{Tanh}\). The other is the sparse representation neural network (SR-NN) (Liu et al., 2019) which is designed to generate sparse representations. Specifically, we apply various classical activation functions (\(\mathrm{ReLU}\), \(\mathrm{Sigmoid}\), \(\mathrm{ELU}\), and \(\mathrm{Tanh}\)) in SR-NNs. We set \(d=8\) for all elephant functions applied in EMLP.

We summarize test MSEs in Table 1. A lower MSE indicates better performance. Additional training details are included in the appendix. For the SR-NN, we present the best result among the combinations of SR-NNs and classical activation functions. Clearly, our method EMLP achieves the best performance, reaching a very low test MSE compared with baselines. Generally, we find that the SR-NN achieves slightly better performance than MLPs with classical activation functions, showing the benefits of sparse representations. However, compared with EMLP, the test MSE of the SR-NN is still large, indicating that the SR-NN fails to approximate \(\sin(\pi x)\).

Next, we plot the true function \(\sin(\pi x)\), the learned function \(f(x)\), and the NTK function \(\mathrm{NTK}(x)=\langle\nabla_{\mathbf{w}}f_{\mathbf{w}}(x),\nabla_{\mathbf{w}}f_{\mathbf{w}}(x _{t})\rangle\) at different training stages for EMLP and SR-NN in Figure 2. The plots of MLP (\(\mathrm{ReLU}\)), MLP (\(\mathrm{Sigmoid}\)), MLP (\(\mathrm{Tanh}\)), and MLP (\(\mathrm{ELU}\)) are not presented, since they are similar to the plots of the SR-NN. The NTK function \(\mathrm{NTK}(\mathbf{x})\) is normalized such that the function value is in \([-1,1]\). In Figure 2, the plots in the first row show that for EMLP with \(d=8\), \(\mathrm{NTK}(x)\)
quickly decreases to 0 as \(x\) moving away from \(x_{t}\), demonstrating the local elasticity (Property 2.3) of EMLP with a small \(d\). However, SR-NNs (and MLPs with classical activation functions) are not locally elastic; the learned function basically evolves as a linear function, a phenomenon often appears in over-parameterized neural networks (Jacot et al. 2018, Chizat et al. 2019).

_By injecting local elasticity to a neural network, we can break the inherent global generalization ability of the neural networks (Ghiassian et al. 2020), constraining the output changes of the neural network to small local areas. Utilizing this phenomenon, we can update a wrong prediction by "editing" outputs of a neural network nearly point-wisely._ To verify, we first train a neural network to approximate \(\sin(\pi x)\) well in the traditional supervised learning style. We call the function the old learned function at this stage. Now assume that the original \(y\) value of an input \(x\) is changed to \(y^{\prime}\), while the true values of other inputs remain the same. The goal is to update the prediction of the neural network for input \(x\) to this new value \(y^{\prime}\), while keeping the predictions of other inputs close to the original predictions, without expensive re-training on the whole dataset. Note that this requirement is common, especially in RL; and we will illustrate it with more details later. For now, we focus on supervised learning for a clearer demonstration. Specifically, we would like to update the output value at \(x=1.5\) of the learned function from \(y=-1.0\) to \(y^{\prime}=-1.5\). We perform experiments on both EMLP and MLP, showing in Figure 3. Clearly, both neural works successfully update the prediction at \(x=1.5\) to the new value. However, besides the prediction at \(x=1.5\), the learned function of MLP is changed globally while the changes of EMLP are mainly confined

\begin{table}
\begin{tabular}{l c} \hline \hline Method & Test Performance (MSE) \\ \hline MLP (ReLU) & \(0.4729\pm 0.0110\) \\ MLP (Sigmoid) & \(0.4583\pm 0.0008\) \\ MLP (Tanh) & \(0.4461\pm 0.0013\) \\ MLP (ELU) & \(0.4521\pm 0.0019\) \\ SR-NN & \(0.4061\pm 0.0036\) \\ EMLP (Elephant) & \(\mathbf{0.0081\pm 0.0009}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: The test MSE of various networks in streaming learning for a simple regression task. Lower is better. All results are averaged over \(5\) runs, reported with standard errors.

Figure 2: Plots of the true function \(\sin(\pi x)\), the learned function \(f(x)\), and the NTK function \(\mathrm{NTK}(x)\) at different training stages for EMLP and SR-NN. The NTK function \(\mathrm{NTK}(x)\) of the EMLP quickly reduces to 0 as \(x\) moves away from \(x_{t}\), demonstrating the local elasticity (Property 2.3) of EMLP.
in a small local area around \(x=1.5\). That is, we can successfully correct the wrong prediction nearly point-wisely by "editing" the output value for ENNs, but not for classical neural networks. To conclude, we summarize our findings in the following:

* Sparse representations are useful but are not effective enough to solve streaming regression tasks.
* EMLP is locally elastic even when \(d\) is small (e.g., \(8\)) and it allows "editing" the output value nearly point-wisely.
* EMLP can continually learn to solve regression tasks by reducing forgetting.

### Class incremental learning

In addition to regression tasks, our method can be applied to classification tasks as well, by simply replacing classical activation functions with elephant activation functions in a classification model. Though our model is agnostic to data distributions, we test it in class incremental learning in order to compare it with previous methods. Moreover, we adopt a stricter variation of the continual learning setting by adding the following restrictions: (1) same as streaming learning, each sample only occurs once during training, (2) task boundaries are not provided or inferred (Aljundi et al.2019), (3) neither pre-training nor a fixed feature encoder is allowed (Wolfe and Kyrillidis2022), and (4) no buffer is allowed to store old task information in any form, such as training samples and gradients.

Surprisingly, we find no methods are designed for or have been tested in the above setting. As a variant of EWC (Kirkpatrick et al.2017), Online EWC (Schwarz et al.2018) almost meets these requirements, although it still requires task boundaries. To overcome this issue, we propose _Streaming EWC_ as one of the baselines, which updates the fisher information matrix after every training sample. Streaming EWC can be viewed as a special case of Online EWC, treating each training sample as a new task. Besides Streaming EWC, we consider SDMLP (Bricken et al.2023) and FlyModel (Shen et al.2021) as two strong baselines, although they require either task boundary information or multiple data passes. Finally, two naive baselines are included, which train MLPs and CNNs without any techniques to reduce forgetting.

We test various methods on several standard datasets -- Split MNIST (Deng2012), Split CIFAR10 (Krizhevsky2009), Split CIFAR100 (Krizhevsky2009), and Split Tiny ImageNet (Le and Yang2015). _We only present results on Split MNIST here due to page limit and put all other results in Appendix C.2._ For SDMLP and FlyModel, we take results from Bricken et al. (2023) directly. For MLP and CNN, we use \(\mathrm{ReLU}\) by default. For our methods, we apply elephant functions with \(d=4\) in an MLP with one hidden layer and a simple CNN. The resulting neural networks are named EMLP and ECNN, respectively.

The test accuracy is used as the performance metric. A result summary of different methods on Split MNIST is shown in Table 2. The number of neurons refers to the size of the last hidden layer in a neural network, also known as the feature dimension. Overall, FlyModel performs the best, although it requires additional task boundary information. Our methods (EMLP and ECNN) are the

Figure 3: A demonstration of local elasticity of the EMLP. EMLP allows updating the old prediction nearly point-wisely by “editing” the output value of the neural network. However, we cannot achieve this for MLP since it is not locally elastic.
second best without utilizing task boundary information by training for a single pass. Other findings are summarized in the following, reaffirming the findings in Mirzadeh et al. (2022a;b):

* Wider neural networks forget less by using more neurons.
* Neural network architectures can significantly impact learning performance: CNN (ECNN) is better than MLP (EMLP), especially with many neurons.
* Architectural improvement is larger than algorithmic improvement: using a better architecture (e.g., EMLP and ECNN) is more beneficial than incorporating Streaming EWC.

Overall, we conclude that applying elephant activation functions significantly reduces forgetting and boosts performance in class incremental learning under strict constraints.

### Reinforcement Learning

Recently, Lan et al. (2023) showed that the forgetting issue exists even in single RL tasks and it is largely masked by a large replay buffer. Without a replay buffer, a single RL task can be viewed as a series of related but different tasks without clear task boundaries (Dabney et al., 2021). For example, in temporal difference (TD) learning, the true value function is usually approximated by \(v\) with bootstrapping: \(v(S_{t})\gets R_{t+1}+\gamma v(S_{t+1})\), where \(S_{t}\) and \(S_{t+1}\) are two successive states and \(R_{t+1}+\gamma v(S_{t+1})\) is named the TD target. During training, the TD target constantly changes due to bootstrapping, non-stationary state distribution, and changing policy. To speed up learning while reducing forgetting, it is crucial to update \(v(S_{t})\) to the new TD target without changing other state values too much (Lan et al., 2023), where local elasticity can help.

_By its very nature, solving RL tasks requires continual learning ability for both classification and regression._ Concretely, estimating value functions using TD learning is a non-stationary regression task; given a specific state, selecting the optimal action from a discrete action space is very similar to a classification task. In this section, we demonstrate that incorporating elephant activation functions helps reduce forgetting in RL under memory constraints. Specifically, we use deep Q-network (DQN) (Mnih et al., 2013, 2015) as an exemplar algorithm following Lan et al. (2023). Four classical RL tasks from Gym (Prockman et al., 2016) and PyGame Learning Environment (Tasfi, 2016) are chosen: MountainCar-v0, Acrobot-v1, Catcher, and Pixelcopter. An MLP/EMLP with one hidden layer of size \(1,000\) is used to parameterize the action-value function in DQN for all tasks. For all elephant functions used in EMLPs, \(d=4\). The standard buffer size is 1e4. For EMLP, we use a

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Method & Neurons & Dataset Passes & Task Boundary & Test Accuracy \\ \hline MLP & 1K & 1 & & 0.665\(\pm\)0.014 \\ MLP+Streaming EWC & 1K & 1 & & 0.708\(\pm\)0.008 \\ SDMLP & 1K & 500 & & 0.69 \\ FlyModel & 1K & 1 & & **0.77** \\
**EMLP (ours)** & 1K & 1 & & 0.723\(\pm\)0.006 \\ CNN & 1K & 1 & & 0.659\(\pm\)0.016 \\ CNN+Streaming EWC & 1K & 1 & & 0.716\(\pm\)0.024 \\
**ECNN (ours)** & 1K & 1 & & 0.732\(\pm\)0.007 \\ \hline MLP & 10K & 1 & & 0.621\(\pm\)0.010 \\ MLP+Streaming EWC & 10K & 1 & & 0.609\(\pm\)0.013 \\ SDMLP & 10K & 500 & & 0.53 \\ FlyModel & 10K & 1 & & **0.91** \\
**EMLP (ours)** & 10K & 1 & & 0.802\(\pm\)0.002 \\ CNN & 10K & 1 & & 0.769\(\pm\)0.011 \\ CNN+Streaming EWC & 10K & 1 & & 0.780\(\pm\)0.010 \\
**ECNN (ours)** & 10K & 1 & & 0.850\(\pm\)0.004 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The test accuracy of various methods in class incremental learning on _Split MNIST_. Higher is better. All accuracies are averaged over \(5\) runs, reported with standard errors. The number of neurons refers to the size of the last hidden layer in a neural network, i.e. the feature dimension.
small replay buffer with size 32; for MLP, we consider two buffer sizes -- 1e4 and 32. More details are included in the appendix.

The return curves of various methods are shown in Figure 4, averaged over \(10\) runs, where shaded areas represent standard errors and \(m\) denotes the size of a replay buffer. Clearly, using a tiny replay buffer, EMLP (\(m=32\)) outperforms MLP (\(m=32\)) in all tasks. Moreover, except Pixelcopter, EMLP (\(m=32\)) achieves similar performance compared with MLP (\(m=1e4\)), although it uses a much smaller buffer. In summary, the experimental results further confirm the effectiveness of elephant activation functions in reducing catastrophic forgetting.

## 6 Related works

Architecture-based continual learningContinual learning methods can be divided into several categories, such as regularization-based methods (Kirkpatrick et al., 2017; Schwarz et al., 2018; Zenke et al., 2017; Aljundi et al., 2019), replay-based methods (Kemker et al., 2018; Farquhar and Gal, 2018; Van de Ven and Tolias, 2019; Delange et al., 2021), and optimization-based methods (Lopez-Paz and Ranzato, 2017; Zeng et al., 2019; Farajtabar et al., 2020). We encourage readers to check recent surveys (Khetarpal et al., 2022; Delange et al., 2021; Wang et al., 2023) for more details. Here, we focus on architecture-based methods that are more related to our work. Among them, our work is inspired by Mirzadeh et al. (2022);b), which study and analyze the effect of different neural architectures on continual learning. Several approaches involve careful selection and allocation of a subset of weights in a large network for each task (Mallya and Lazebnik, 2018; Sokar et al., 2021; Fernando et al., 2017; Serra et al., 2018; Masana et al., 2021; Li et al., 2019; Yoon et al., 2018), or the allocation of a network that is specific to each task (Rusu et al., 2016; Aljundi et al., 2017). Some other methods expand a neural network dynamically (Yoon et al., 2018; Hung et al., 2019; Ostapenko et al., 2019). Finally, Shen et al. (2021) and Bricken et al. (2023) proposed novel neural network structures inspired by biological neural circuits, acting as two strong baseline methods in our experiments.

Sparse representationsSparse representations are known to help reduce forgetting for decades (French, 1992). In supervised learning, sparse training (Dettmers and Zettlemoyer, 2019; Liu et al., 2020; Sokar et al., 2021), dropout variants (Srivastava et al., 2013; Goodfellow et al., 2013; Mirzadeh et al., 2020; Abbasi et al., 2022; Sarfraz et al., 2023), and weight pruning methods (Guo et al., 2016; Frankle and Carbin, 2019; Blalock et al., 2020; Zhou et al., 2020) are shown to speed up training and improve generalization. In continual learning, both SDMLP (Shen et al., 2021) and FlyModel (Bricken et al., 2023) are designed to generate sparse representations. In RL, Le et al. (2017), Liu et al. (2019), and Pan et al. (2022) showed that sparse representations help stabilize training and improve overall performance.

Local elasticity and memorizationHe and Su (2020) proposed the concept of local elasticity. Chen et al. (2020) introduced label-aware neural tangent kernels, showing that models trained with these kernels are more locally elastic. Mehta et al. (2021) proved a theoretical connection between the scale of neural network initialization and local elasticity, demonstrating extreme memorization using large initialization scales. Incorporating Fourier features in the input of a neural network also induces local elasticity, which is greatly affected by the initial variance of the Fourier basis (Li and Pathak, 2021).

Figure 4: The return curves of DQN in four tasks with different neural networks and buffer sizes. All results are averaged over \(10\) runs, where the shaded areas represent standard errors. Using a much smaller buffer, EMLP (\(m=32\)) outperforms MLP (\(m=32\)) and matches MLP (\(m=1e4\)).
Conclusion

In this work, we proposed elephant activation functions that can make neural networks more resilient to catastrophic forgetting. Specifically, we showed that incorporating elephant activation functions in classical neural networks helps improve learning performance in streaming learning for regression, class incremental learning, and reinforcement learning. Theoretically, our work provided a deeper understanding of the role of activation functions in catastrophic forgetting. We hope our work will inspire more researchers to study and design better neural network architectures for continual learning.