Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking
  Neural Networks
The evolution of the human brain has led to the development of complex
synaptic plasticity, enabling dynamic adaptation to a constantly evolving
world. This progress inspires our exploration into a new paradigm for Spiking
Neural Networks (SNNs): a Plasticity-Driven Learning Framework (PDLF). This
paradigm diverges from traditional neural network models that primarily focus
on direct training of synaptic weights, leading to static connections that
limit adaptability in dynamic environments. Instead, our approach delves into
the heart of synaptic behavior, prioritizing the learning of plasticity rules
themselves. This shift in focus from weight adjustment to mastering the
intricacies of synaptic change offers a more flexible and dynamic pathway for
neural networks to evolve and adapt. Our PDLF does not merely adapt existing
concepts of functional and Presynaptic-Dependent Plasticity but redefines them,
aligning closely with the dynamic and adaptive nature of biological learning.
This reorientation enhances key cognitive abilities in artificial intelligence
systems, such as working memory and multitasking capabilities, and demonstrates
superior adaptability in complex, real-world scenarios. Moreover, our framework
sheds light on the intricate relationships between various forms of plasticity
and cognitive functions, thereby contributing to a deeper understanding of the
brain's learning mechanisms. Integrating this groundbreaking plasticity-centric
approach in SNNs marks a significant advancement in the fusion of neuroscience
and artificial intelligence. It paves the way for developing AI systems that
not only learn but also adapt in an ever-changing world, much like the human
brain.
Guobin Shen, Dongcheng Zhao, Yiting Dong, Yang Li, Feifei Zhao, Yi Zeng
# Metaplasticity: Unifying Learning and Homeostatic Plasticity in Spiking Neural Networks

###### Abstract

The natural evolution of the human brain has given rise to multiple forms of synaptic plasticity, allowing for dynamic changes to adapt to an ever-evolving world. The evolutionary development of synaptic plasticity has spurred our exploration of biologically plausible optimization and learning algorithms for Spiking Neural Networks (SNNs). Present neural networks rely on the direct training of synaptic weights, which ultimately leads to fixed connections and hampers their ability to adapt to dynamic real-world environments. To address this challenge, we introduce the application of metaplasticity - a sophisticated mechanism involving the learning of plasticity rules rather than direct modifications of synaptic weights. Metaplasticity dynamically combines different plasticity rules, effectively enhancing working memory, multitask generalization, and adaptability while uncovering potential associations between various forms of plasticity and cognitive functions. By integrating metaplasticity into SNNs, we demonstrate the enhanced adaptability and cognitive capabilities within artificial intelligence systems. This computational perspective unveils the learning mechanisms of the brain, marking a significant step in the profound intersection of neuroscience and artificial intelligence.

## 1 Introduction

The long-term stability and efficiency of the brain mainly attribute to the power of neuronal plasticity [1; 2; 3]. Neuronal plasticity serves as the fundamental attribute of the brain, enabling people to learn new information from the environment, remember various experiences, and adapt to constantly changing surroundings. Neuronal plasticity primarily manifests in two forms: learning plasticity (LP) and homeostatic plasticity (HP), which cooperatively adjust and balance how the brain processes new information and memory of experiences [4; 5; 6; 7].

LP relies on the activities of the neurons, reflecting environmental information or new learning experiences by adjusting the strength of synaptic connections [8; 9; 10]. Nevertheless, relying solely on learning plasticity does not guarantee the stability of neural networks, mainly when tasked with processing environmental information that is both complex and subject to dynamic changes. Under such circumstances, the role of homeostatic plasticity is paramount. Homeostatic plasticity operates as a stabilizing mechanism within neural networks. In instances where neuronal activity
escalates too high or drops too low, this form of plasticity steps in to recalibrate and restore the activity level of neurons to an optimal state, thereby preserving the overarching stability of the neural network [11; 12; 13; 14; 15]. Metaplasticity, often referred to as the 'plasticity of plasticity', amplifies the brain's capacity for learning and adaptation. If fine-tunes neuronal learning strategies and manages the stability of neural networks, thereby allowing the brain the flexibility to adjust to diverse environmental shifts while safeguarding its capacity for normal function across many settings. This intricate balance between flexibility and stability equips our brains with the ability to learn and memorize effectively, even within the context of highly complex environments [16; 17; 18].

Spiking neural networks (SNNs) successfully simulate the discrete spike sequence information transmission process in biological nervous systems by finely modeling the dynamics of biological neurons. The event-driven and real-time nature of information processing in SNNs endows them with a superior ability to manage tasks involving temporal dynamics compared to traditional artificial neural networks (ANNs) [19]. The training methodologies for SNNs primarily bifurcate into two distinct categories. The first approach entails the optimization of synaptic weights utilizing a backpropagation algorithm based on surrogate gradients [20; 21; 22]. The second one involves the adjustment of synaptic weights through the application of biologically inspired synaptic plasticity rules, with a significant emphasis on LP [23; 24; 25]. Although these algorithms have solved the training problem of SNNs to a certain extent, these methods all use fixed learning rules to guide network training and lack high adaptability to the environment. Some research attempts to enhance the network's information processing capabilities by increasing the heterogeneity of neurons, but its flexibility still needs to be improved [26; 27; 28]. Some studies have coordinated various learning rules to enhance the learning capabilities of SNNs. However, these efforts primarily rely on manually preset coordination [29; 30]. Some studies leverage neural modulation factors to enact adaptive Spike-Timing Dependent Plasticity (STDP). However, these works tend to concentrate solely on LP, overlooking the role of HP in synaptic learning [31; 32].

In this study, we present a pioneering approach that shifts the focus from traditional weight-centric training to learning the principles of plasticity. This shift fosters the ability of SNNs, to adapt to evolving environments even in the absence of explicit reward signals. By embracing metaplasticity, a high-order process that governs plasticity rules, we enable SNNs to adjust their synaptic modifications based on the neurons' activity history. This mechanism enhances the adaptability and cognitive prowess of artificial intelligence systems and fosters their continual learning and evolution, mimicking the inherent characteristics of their biological counterparts. This augmentation in learning strategy and adaptability presents SNNs with improved generalization in dynamic real-world environments, boosting their multi-tasking learning abilities.

Our contributions can be outlined as follows:

* We introduce a novel framework emphasizing learning plasticity rules rather than direct synaptic weight modifications. This approach imbues artificial neural networks that can adapt to their environment, like biological systems.
* By combining Learning Plasticity (LP) and Homeostatic Plasticity (HP), our method potentially increases the generalization and multitasking abilities of SNNs in dynamic and real-world scenarios. It provides a platform for continuous learning and adaptation, mirroring the extraordinary capabilities exhibited by biological nervous systems.
* By closely aligning our artificial systems with biological mechanisms, our work significantly strides towards a more robust, adaptable, and biologically plausible artificial neural system. This alignment improves the system's learning and adaptation but also aids in understanding the underlying mechanisms of the brain's learning processes.

## 2 Results

### Metaplasticity: Learning to Balance Learning and Homeostatic Plasticity

To endow artificial agents with plasticity across their entire lifespan, not just during the training phase, we need to shift our focus from optimizing fixed synaptic weights to optimizing plasticity rules. Inspired by the concept of metaplasticity in biological systems, such an approach would allow artificial networks to dynamically adapt their internal structures to evolving environmental conditions, similarly to their biological counterparts.
This shift requires the development of new models and algorithms that can capture the dynamic, adaptable nature of biological plasticity. These models need to continuously learn from the environment and modify their learning rules in response to changing inputs and tasks, essentially 'learning to learn'. The challenge here lies in incorporating this metaplasticity into the design and training of SNNs, enabling them to mimic the adaptability and cognitive capabilities of biological neural systems better.

As shown in Fig. 1, in our metaplasticity framework, we define two aspects of plasticity: LP and HP:

* **Learning Plasticity (LP)**: LP, represented by Spike-Timing Dependent Plasticity (STDP), is vital for adjusting synaptic strength based on the timing of pre- and post-synaptic activity. This facet of plasticity helps shape a neuron's response to its inputs by enforcing the causality requirement in synaptic strengthening.
* **Homeostatic Plasticity (HP)**: HP plays a crucial role in maintaining the overall level of excitation within a neuron or network within a regulated range and redistributing synaptic efficacy among different synapses of a neuron. This helps preserve the total synaptic strength of a neuron within certain bounds, thus preventing over-activation or inhibition.

Inspired by these mechanisms, we propose a form of metaplasticity with learnable parameters. Throughout the lifespan of an agent, synaptic weights are updated as follows:

\[\Delta w_{i,j}=\eta\underbrace{(A_{i,j}x_{i}(x_{j}-C_{i,j})}_{\text{Learning Plasticity (LP)}}+\underbrace{B_{i,j}x_{j}+D_{i,j}}_{\text{Homeostatic Plasticity (HP)}}) \tag{1}\]

In Eq. 1, \(\Delta w_{i,j}\) represents the change in synaptic weight between neurons \(i\) and \(j\). \(x_{i}\) and \(x_{j}\) denote the traces of pre-synaptic and post-synaptic spikes, respectively. The traces increase whenever a spike occurs and gradually decay to zero with their respective intrinsic time constants. The parameters

Figure 1: **Diagram of metaplasticity.** Top: By combining Learning Plasticity (LP) and Homeostatic Plasticity (HP), neurons can achieve diverse and heterogeneous plasticity. Bottom: Agents with metaplasticity learn plasticity rather than directly adjusting weights. Different forms of synaptic plasticity can be formed between neurons, enabling better multi-task learning. Plasticity helps the agents dynamically adjust weights and learn previously unseen scenarios during training, even without explicit reward signals.
\(A_{i,j}\), \(B_{i,j}\), \(C_{i,j}\), and \(D_{i,j}\) are independent and learnable for each synapse, enabling the formation of distinct plasticity rules. These parameters govern the contributions of each plasticity mechanism to the overall update of synaptic weights.

The term \(A_{i,j}x_{i}(x_{j}-C_{i,j})\) embodies the contribution of LP and is related to the activity of the pre- and post-synaptic neurons. The parameter \(A_{i,j}\) regulates the contribution of LP, while \(C_{i,j}\) serves as a threshold parameter determining the level of post-synaptic activity required for synaptic strengthening. The term \(B_{i,j}x_{j}\) and \(D_{i,j}\) represent the contribution of HP. Here, the parameter \(B_{i,j}\) adjusts the synaptic weight based on the post-synaptic activity \(x_{j}\), while \(D_{i,j}\) acts as a unique bias for each neuron, providing a constant contribution to the synaptic weight update regardless of pre- and post-synaptic activity. Working with other plasticity mechanisms, this term helps maintain the total synaptic strength within a specific range. The learning rate \(\eta\) scales the overall synaptic weight update. This metaplasticity rule allows the agent to learn the optimal parameters \(A_{i,j}\), \(B_{i,j}\), \(C_{i,j}\), and \(D_{i,j}\) that balance the contributions of the two plasticity mechanisms, thereby adapting its synaptic weights flexibly and stably throughout its lifespan.

Building upon the metaplasticity rule defined in Eq. 1, we employ an Evolutionary Strategy (ES) [33] to optimize the parameters. This strategy is inspired by the notion that the natural selection and evolution process shapes the intrinsic properties of biological organisms, including their neural plasticity mechanisms.

In this context, the parameters of the metaplasticity rule can be viewed as intrinsic priors optimized throughout evolution to ensure survival and adaptation to various environmental conditions. The ES will involve a population of agents, each with a unique set of metaplasticity parameters. The fitness of each agent will be evaluated based on its ability to adapt and perform in various tasks and environments. The fittest agents will be selected, and their metaplasticity parameters will be used to generate the next generation of agents, with some variations introduced to encourage exploration of the parameter space. Through this process, the metaplasticity parameters will be optimized, enabling the agents to maintain plasticity and dynamically adapt to the environment, even without explicit reward signals.

### Metaplasticity Enhances Working Memory Capacity

In this section, we aim to demonstrate the effect of metaplasticity on Working Memory (WM). WM is the ability to maintain and process information temporarily and is the cornerstone of higher intelligence [34].

We explore the significant impact of metaplasticity on enhancing the WM capabilities of SNNs. We utilize a task known as the copying task [35], as illustrated in Fig. 2**A**. In this task, SNNs are initially presented with a sequence of stimuli, each lasting for \(200\)ms. This is then followed by a delay period of variable lengths, and finally, a test stimuli of equivalent duration to the sample stimuli is presented. The challenge for the SNNs lies in accurately reproducing the initial sequence of stimuli in the correct order upon receiving the test stimuli.

To thoroughly demonstrate the advantages of employing a metaplasticity-based approach in working memory tasks, we carry out a comparison with the strategy of directly optimizing weights utilizing the ES based on widely used three-layer SNNs. As shown in Fig. 2**B**, SNNs equipped with metaplasticity exhibit faster convergence rates, an ability to retain memory over longer durations, and an enhanced memory capacity.

To further investigate the influence of metaplasticity, we visualize synaptic weights following various stimuli inputs when the stimuli length is set to \(8\). As shown in Fig. 2**C**, SNNs endow with metaplasticity can form distinct connection weights for different stimuli, demonstrating their superior adaptive capacity.

We compare the neuronal states at various stages between SNNs directly trained with weights and those incorporating metaplasticity, as illustrated in Fig. 2**D**. SNNs directly trained with weights rely on neuronal activity during the delay period to maintain memory. Resetting the membrane potential to \(0\) after the stimulus input leads to memory loss for input stimuli. This indicates that in SNNs directly optimized with weights, memory is primarily stored in neuronal activity. In contrast, SNNs incorporating metaplasticity encode the input stimulus into synaptic weights, demonstrating remarkable memory capabilities. Their ability to adjust synaptic weights facilitates the enhancement
of working memory and allows neurons to remain in a resting state when not receiving task-related stimuli. This contributes to network efficiency and enhances network capacity, which has been validated in other biological and computational neuroscience studies [36; 37].

Finally, we visualize the average spike traces of SNNs under different training strategies, as shown in Fig. 2**E**. Notably, SNNs incorporating metaplasticity can sustain lower firing rates, thereby enhancing their efficiency in managing computational resources.

In summary, these results highlight the significant role of metaplasticity in bolstering the working memory capacity of SNNs, demonstrating its potential for facilitating complex cognitive tasks in artificial intelligence systems.

### Metaplasticity Enhances Multi-task Learning

Reinforcement Learning (RL) involves an agent learning to interact with its environment to achieve a specific goal, with the quality of its actions dictated by a reward signal. Conventional RL approaches often entail training the agent on a single task with a fixed reward function. However, in complex, real-world environments, agents need to handle multiple tasks and adjust to changing reward functions. This multi-task learning scenario presents significant challenges, mainly when the tasks involved are diverse and potentially conflicting.

Figure 2: Design of the WM experiment and the impact of metaplasticity on WM. **A.** Schematic of the copying task. The SNNs first receive a sequence of motion stimuli, with each stimulus lasting \(200\)ms, followed by a delay period of varying lengths, and finally, a test stimulus of the same duration as the sample stimulus. The SNNs are required to reproduce the stimuli from the first phase to receive the test stimulus. **B.** Performance comparison between SNNs with plasticity and trained with direct weights. SNNs with plasticity show faster convergence, longer memory duration, and greater memory capacity. ’Len’ refers to the length of stimulus samples, while ’Lat’ refers to the number of steps in the delay period. **C.** Synaptic weights after different motion stimulus inputs when the number of motion samples is \(8\). SNNs with plasticity can form distinct connection weights for different stimuli. The left side of the dashed line shows the input weights associated with the stimulus, and the right side shows the output weights. **D.** Neuron states at different stages in SNNs trained directly with weights and those with metaplasticity. Directly trained SNNs require neural activity during the delay period to maintain memory. Resetting the membrane potential to \(0\) after the input stimulus leads to a chance-level memory accuracy, resulting in memory loss. In contrast, SNNs with plasticity can encode input stimuli into synaptic weights, demonstrating stronger memory functionality. **E.** Visualization of the firing rates at different stages and the average spike traces for SNNs using different strategies. SNNs with plasticity can maintain lower firing rates.
In the field of RL, some of the most challenging problems lie within the domain of continuous control, usually modeled using sophisticated physics engines. These tasks require agents to manipulate simulated physical entities with high precision and coordination, similar to how humans control their limbs to carry out complex tasks. We use the Brax [38] simulator to design six continuous control environments. In these settings, agents need to navigate at various speeds, directions, and destination points. As shown in Fig. 3**A**, different task objectives are treated as observations to guide the agent in accomplishing different tasks. These challenging tasks serve as baselines in fields such as meta-learning [39; 40; 41]. During training, they are only exposed to a limited number of task instances, such as eight specific directions or eight fixed speeds. They use a single network to learn these unrelated or conflicting tasks.

Our experiments compare two types of SNNs - one with synaptic weights that have been directly optimized and another with optimized metaplasticity. Both types of SNNs maintain the same scale and structure to ensure a fair comparison. Through this, we aim to highlight the advantages of optimizing metaplasticity over direct weight optimization in a multi-task environment. This comparison also evaluates the effectiveness and potential of our proposed model, especially in the face of the inherent challenges posed by complex continuous control tasks.

We explore the performance of metaplasticity in a three-layer, fully-connected SNN model with \(128\) hidden spiking neurons. Both the synaptic weights and the plasticity parameters are initialized to \(0\). During testing, their plasticity rules are fixed for agents with plasticity, and synaptic weights are reset to \(0\). The trained weights are applied during testing for agents trained directly on weights. We utilize reinforcement learning tasks to thoroughly test metaplasticity, requiring the agents to learn to cope with different tasks simultaneously and generalize the acquired knowledge to unseen, more complex tasks.

In our experiments, agents with metaplasticity show superior performance in multiple tasks compared to agents with directly optimized synaptic weights. As seen in Fig. 3**B** and Tab. 1, metaplastic agents exhibit more effective learning curves. The metaplastic agents quickly adapted to the changes in tasks, making them more capable of tackling the multi-task challenges inherent in the designed environments. In contrast, SNNs with directly trained weights fail to adapt to different tasks and can only acquire trivial solutions, such as maintaining approximate immobility in tasks involving multiple target directions.

Ablation studies, depicted in Fig. 3**C** and Tab. 1, provide further insights into the contributions of different plasticity mechanisms. Removing any form of plasticity results in decreased agent performance, with the removal of HP causing a divergence due to the loss of the equilibrium mechanism. This result highlights the importance of all plasticity mechanisms in maintaining the stability and adaptability of the agents.

The changing curve of a synapse's metaplasticity during training (Fig. 3**D**), along with the impact of plasticity on weights for different generations of agents (Fig. 3**E**), demonstrate the effective learning of optimal parameters for the metaplasticity rule, facilitated by the evolutionary strategy. Interestingly, the specific functions of plasticity across different generations of agents differed (Fig. 3**F**), indicating the evolution and fine-tuning of plasticity mechanisms to improve agent performance across generations.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Training & ant\_dir & swimmer\_dir & halfcheetah\_vel & hopper\_vel & fetch & urSe \\ \hline \hline \(\textbf{Opt}_{LP+HP}\) & \(6904\pm 801\) & \(10531\pm 827\) & \(-549\pm 95\) & \(869\pm 38\) & \(51\pm 3\) & \(86\pm 5\) \\ \(\textsc{Opt}_{HP}\) & \(3284\pm 570\) & \(7831\pm 1479\) & \(-870\pm 312\) & \(792\pm 50\) & \(26\pm 26\) & \(58\pm 13\) \\ \(\textsc{Opt}_{LP}^{*}\) & - & - & - & - & - & - \\ \(\textsc{Opt}_{Weight}\) & \(1069\pm 98\) & \(31\pm 8\) & \(-1598\pm 324\) & \(729\pm 116\) & \(15\pm 0.6\) & \(7\pm 5\) \\ \hline \hline Chance Level & \(995\pm 0.01\) & \(0.12\pm 0.02\) & \(-4946\pm 1.3\) & \(6.51\pm 0.3\) & \(4.74\pm 0.0\) & \(0\pm 0.0\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of performance across various reinforcement learning tasks with different configurations of synaptic plasticity. Each task is evaluated using different training methods: Plasticity\({}_{LP+HP}\), Plasticity\({}_{LP}\), Plasticity\({}_{HP}\), and direct weight training. The values presented are the mean and standard deviation over \(5\) trials. The row ’Chance Level’ shows the performance metrics when randomly chooses actions. \({}^{*}\) Upon the removal of Homeostatic Plasticity (HP), the SNNs become unstable, leading to divergence.
### Metaplasticity Enhances Generalization Ability

Metaplasticity serves as an important mechanism for enhancing an agent's generalization abilities, enabling the agent to exhibit stronger performance when dealing with unfamiliar tasks or when facing neuronal damage.

We further investigate the performance of metaplasticity when the agents faced injuries. We design two different types of injuries: temporary injuries and permanent injuries. Temporary injuries refer to a scenario where all synaptic weights of the agents are reset to \(0\) and kept for \(50\) steps. Permanent injuries refer to a situation where some synapses are set to \(0\) initially and do not update according to plasticity. In the tests for injuries, agents with plasticity display a remarkable ability to recover from temporary neuronal damage simulated by resetting all synaptic weights to \(0\) for \(50\) steps (Fig. 4**A**). Fig. 4**B** illustrates the changes in network weights at various stages before and after the temporary

Figure 3: **Metaplasticity’s performance in multi-task reinforcement learning tasks and the influence of different plasticity attributes on performance.****A.** Illustration of multi-task reinforcement learning. The agent is required to utilize a singular network to simultaneously learn multiple tasks with distinct objectives or even entirely opposing ones. The objectives of the tasks are treated as observations for the agent. They are inputted to the SNNs along with other observations such as joint positions, velocities, etc. **B.** Training curves of agents with metaplasticity versus those trained directly on weights. In these multi-task RL tasks, agents need to learn to move towards different directions (ant_dir, swimmer_dir), at varying speeds (halffcheetah_vel, hopper_vel), and to different locations (fetch,ur5e). Agents with metaplasticity maintain dynamic synaptic weights, learn characteristics of different tasks, and hence achieve superior performance in multi-task challenges. **C.** Ablation analysis of different plasticity mechanisms. Different colors represent training curves with some form of plasticity (LP or HP) removed. After removing HP, SNNs diverge due to the loss of the equilibrium mechanism. Both LP and HP play significant roles in enhancing agent performance. **D.** The change curve of a synapse’s metaplasticity during the training process. Through evolutionary strategies, agents learn to adjust their plasticity. **E.** During the training process, at different inter-spike intervals of pre-synaptic and post-synaptic neurons, the impact of the plasticity of agents from different generations on weights. **F.** The specific functions of plasticity in agents from different generations as shown in **E**.
damage. Remarkably, despite losing all synaptic weights due to the inflicted temporary damage, agents manage to recover these weights using their inherent plasticity and incoming input stimuli. Moreover, even under permanent neuronal damage, with a proportion of neurons blocked and their weights unable to update, the plasticity-enabled agents continue to exhibit better performance and robustness (Fig. 4**C**). These results suggest that metaplasticity can contribute to the resilience of artificial agents, much as it does in biological systems.

Figure 4: **Performance under temporary and permanent nerve injury.****A.** The agent’s performance in the face of temporary neural damage. At the \(500\)th step, all synaptic weights were reset to \(0\) to simulate a sudden neural system injury, and this condition lasted for \(50\) steps. Agents with plasticity were able to recover from this temporary loss, demonstrating better robustness. **B.** Network weights at different times before and after temporary damage. The synaptic weights of the input layer are shown above the dashed line, while the readout layer weights are below the dashed line. Even if the agent loses all synaptic weights due to temporary damage, it can still recover these weights based on its plasticity and input stimuli. **C.** The agent’s performance in the face of permanent neuronal damage of varying degrees. At the start of the test, neurons were blocked at different proportions, their synaptic weights set to \(0\), and could not be updated, simulating permanent neural network damage. Agents with plasticity performed better and exhibited stronger robustness when dealing with such permanent damage.

Figure 5: **A.** Performance of different agents in trained tasks and tasks not seen during training. Agents with plasticity can generalize well to unseen tasks, while agents trained directly on weights have difficulty generalizing to unseen test tasks due to their weights being fixed during testing. **B.** Low-dimensional embeddings of neuronal states during reinforcement learning tasks, differentiated by training strategy. Each point corresponds to the state of the hidden layer neurons at a specific time step. The color coding signifies distinct tasks. Agents that possess plasticity demonstrate an enhanced capability to distinguish between different tasks. Moreover, the neuronal states associated with identical tasks exhibit the intriguing property of forming a manifold within the high-dimensional space.
More strikingly, metaplastic agents show a robust ability to generalize to tasks unseen during training, as demonstrated in Fig. 5**A**. For tasks with various movement directions and speeds, the agents only encounter a small subset of cases during training, represented by the red and orange points in Fig. 5**A**. Therefore, in this experiment, the agents are required to move in directions and at speeds unseen in training, emphasizing the agents' more profound understanding and generalization capacity for the tasks. In contrast, agents trained directly on weights struggle with generalization due to their fixed weights during testing. This observation underscores the flexible adaptability offered by metaplasticity, enhancing the agent's ability to navigate unseen scenarios.

During the training phase, the agents are instructed to move in straight lines in eight specific directions. However, as illustrated in Fig. 5**A**, agents with metaplasticity demonstrate a degree of generalization capability. They can learn to move in straight lines toward directions not encountered during training, while agents without metaplasticity struggle to generalize what they have learned.

To further test the generalization capacity introduced by metaplasticity, we hope that agents could learn to turn or even form more complex paths simply by changing the target signal and without any additional feedback information related to posture. The results are shown in Fig. 6. Compared to agents that directly train their weights, those with metaplasticity demonstrate impressive generalization abilities toward this complex task. They can quickly adjust synaptic weights through metaplasticity and dynamically modify their state in previously unseen scenarios during different pieces of training. This allows them to progress toward varying target directions.

In Fig. 5**B**, we visualize the neuronal states of agents trained using different strategies during RL tasks in low-dimensional space. Each point in this representation corresponds to the state of the hidden layer neurons at a particular time step, with the varying colors indicating different tasks. The remarkable aspect of these visualizations is how agents with inherent plasticity demonstrate a pronounced ability to discern between distinct tasks. Even more intriguing is the observation that the neuronal states corresponding to the same task tend to cluster together, forming a clear manifold in the high-dimensional space. This feature of metaplasticity contributes to the agent's robust ability to generalize across various tasks while maintaining distinctive task-specific patterns in neuronal states. This further illustrates the powerful capabilities of agents with metaplasticity and the profound impact of such plasticity on the agent's learning and adaptation abilities.

## 3 Discussion

Current artificial intelligence algorithms often focus on solving specific tasks, and their performance may fall short when faced with scenarios that deviate from their training conditions. These algorithms' singular functionality, lack of robustness, and limited flexibility restrict their adaptability

Figure 6: **Testing of the agent’s generalization capabilities. During training, the agent only learned to move in a straight line. The agent’s movement trajectories are shown when the target direction is altered during the testing process. The green line represents the expected trajectory of the agent moving at a constant speed. The orange line is the actual movement trajectory of the agent with plasticity. Agents with plasticity can better understand different tasks, adjust synaptic weights according to different target directions, and thus show greater flexibility and superior generalization performance.**
and application in complex and variable environments. In contrast, biological entities demonstrate exceptional adaptability in complex and changing environments, typically attributed to the plasticity of biological neural networks. Synaptic plasticity is the cornerstone and heart of exploring more generalized intelligence [42; 43].

SNNs with their brain-inspired operating mechanisms, lay the foundation for constructing flexible and robust intelligent systems, thereby attracting considerable attention in the field of artificial intelligence [44; 45; 46; 47]. However, current SNN training algorithms primarily rely on the backpropagation of external error signals and biologically-inspired plasticity rules, such as Spike-Timing-Dependent Plasticity. Although these methods demonstrate robust performance on individual tasks, the fixed learning paradigm still limits the generalization ability of SNNs and adaptability in multi-task environments. Contrary to the static nature of synaptic weight adjustments in traditional SNNs, we delve into metaplasticity, facilitated by the synergy between LP and HP. Metaplasticity represents a higher-order learning process that dynamically adjusts plasticity rules. Through fostering adaptive synaptic modifications derived from the history of neuronal activity, metaplasticity promotes the emergence of a more dynamic and self-regulated learning system. Such a mechanism potentially narrows the gap between SNNs and their biological equivalents, thereby facilitating the progression towards continual learning and adaptation.

Our experimental results reveal that metaplasticity significantly enhances the memory capacity of SNNs by encoding memories directly into synaptic weights. Moreover, it does not rely on spike activity to sustain memory, allowing the network to remain in a resting state when not processing task-related stimuli, thus significantly improving the energy efficiency of SNNs. Metaplasticity also dramatically amplifies the multi-task learning and generalization capabilities of SNNs, facilitating a swift transfer of knowledge learned from other tasks to more complex and unfamiliar tasks. Regarding adapting the paths and turning towards new directions, our metaplasticity models can maneuver in ways not encountered during training, a critical feature in complex, dynamic, and unpredictable real-world environments. Importantly, metaplasticity plays a pivotal role in bolstering the robustness of SNNs under simulated motor impairment scenarios. The exceptional resilience demonstrated in temporary damage scenarios validates the advantages of integrating metaplasticity into neural networks. Moreover, even in permanent damage, the exhibited resilience reinforces the case for metaplasticity as an inherent attribute of artificial systems. These characteristics reflect recovery mechanisms in biological systems where the brain mitigates damage through neural reorganization and the formation of new connections.

In conclusion, our study highlights metaplasticity as a critical feature that enhances the resilience and adaptability of artificial agents. These findings provide valuable insights for designing future artificial systems, opening up new possibilities for creating adaptive, robust, and intelligent agents capable of navigating complex and dynamic environments. Further work can explore more sophisticated forms of metaplasticity and study their impacts on various facets of artificial agent performance.

## 4 Method

### Neuron and synaptic models

We employed leaky integrate-and-fire (LIF) neurons in our network models due to their biological plausibility and computational efficiency. The state of each LIF neuron was represented by its membrane potential, which integrated the incoming signals and generated a spike when the potential crossed a predefined threshold, as shown in Eq. 2.

\[\tau_{m}\frac{\partial v}{\partial t}=-(v-v_{0})+I(t) \tag{2}\]

In Eq. 2, \(\tau_{m}\) is the membrane time constant, \(v\) is the membrane potential, \(v_{0}\) is the resting potential, and \(I(t)\) is the total synaptic current at time \(t\). Once the membrane potential \(v\) exceeds a certain threshold \(v_{th}\), the neuron generates a spike, and the potential is reset. A discrete form of the LIF neuron's behavior can be described as:
\[u(t) =v(t-\Delta t)+\frac{\Delta t}{\tau_{m}}(\sum_{i}w_{i}s_{i}(t)-v(t- \Delta t)+v_{0}) \tag{3}\] \[s(t) =g(u(t)-v_{th})\] \[v(t) =u(t)(1-s(t))+v_{reset}s(t)\]

In Eq. 3, \(\Delta t\) is the time step, \(v_{reset}\) is the reset potential, \(u(t)\) and \(v(t)\) represent pre- and post-spike membrane potentials, and \(g(\cdot)\) is the Heaviside function modeling spiking behavior. After the loss of the reward signal, updating the network weights stops. This strategy of directly optimizing the weights is set as a control group in our experiments. Neuronal parameters are given in Tab. 2 unless otherwise specified.

_Traces_ are the tracks produced at the pre- and post-synaptic sites by the spikes of pre- or post-synaptic neurons. Generally, these traces represent the recent activation level of pre- and post-synaptic neurons [48]. Traces can be computed by integrating spikes using a linear operator in the model and a low-pass filter in the circuit or by using non-linear operators/circuits. In the experiments, the synaptic traces were modeled as follows:

\[x(t)=\sum_{\tau=0}^{t}\lambda^{t-\tau}s(\tau) \tag{4}\]

In Eq. 4, \(x(t)\) is the synaptic trace at time \(t\), \(\lambda\) is the decay factor reflecting how quickly a spike's influence fades with time, and \(s(\tau)\) represents the spike at time \(\tau\). In the context of our experiment, these synaptic traces maintain a short-term history of neuron activation, thereby adding an element of temporal dynamics to our network model. As shown in Eq. 1, these synaptic traces are used to maintain a short-term history of neuronal activation and, in conjunction with metaplasticity, to modulate synaptic weights.

### Experimental Settings

#### 4.2.1 Working Memory Task

To validate the impact of metaplasticity on working memory, we designed a working memory task. The agent would first receive a stimulus sequence, and after a delay of \(m\) steps, the agent is asked to reproduce the received stimulus. In each experiment, a random sequence of length \(n\) would be generated, where \(r_{t}\sim\mathcal{B}(1,\frac{1}{2}),1<t\leq n\). At each time step, the input is a three-dimensional vector \(\vec{a_{t}}\), which can be divided into three stages:

* Stimulus reception: If \(1<t\leq n\), \(\vec{a_{t}}=(r_{t},1,0)\). The first element is the type of input stimulus, and the second element is the indicator for the input stimulus.
* Delay: If \(n+1<t\leq n+m\), \(\vec{a_{t}}=(0,0,0)\). This phase represents a delay period where no new stimulus is presented.
* Stimulus reproduction: If \(n+m+1<t\leq 2n+m\), \(\vec{a_{t}}=(0,0,1)\). The last element indicates whether a pulse needs to be reproduced.

\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Parameter**} & \multicolumn{2}{c}{**Value**} & \multicolumn{2}{c}{**Description**} \\ \cline{2-4}  & WM task & RL task & \\ \hline \(\Delta t\) & \(20\) ms & \(200\) ms & Simulation time step \\ \(\tau_{m}\) & \(40\) ms & \(400\) ms & Membrane time constant \\ \(\lambda\) & \(54\) ms & \(544\) ms & Decay factor \\ \(v_{th}\) & \(0.1\) V & \(0.1\) V & Membrane threshold \\ \(v_{reset}\) & \(0\) mV & \(0\) mV & Reset potential \\ \(v_{0}\) & \(0\) mV & \(0\) mV & Resting potential \\ \hline \hline \end{tabular}
\end{table}
Table 2: Parameters of the spiking neurons.
At each step, the model has a scalar output \(s_{t}\), which is a prediction for the stimulus. The Mean Square Error over the last \(m\) steps is taken as the reward of the model:

\[R=-\frac{1}{n}\sum_{\tau=1}^{n}(r_{t}-s_{m+n+\tau})^{2} \tag{5}\]

Eq. 5 is used as a reward function in training. To intuitively compare agents with different strategies, as shown in Fig. 2**B**, we utilize the average accuracy per step as the performance measure during testing, as shown in Eq. 6.

\[\text{Acc}=\frac{1}{n}\sum_{\tau=1}^{n}(r_{t}==s_{m+n+\tau}) \tag{6}\]

During the stimulus reception stage, each stimulus follows the distribution \(\mathcal{B}(1,\frac{1}{2})\), which means that the average accuracy at the chance level is \(0.5\).

#### 4.2.2 Multi-task Reinforcement Learning

We evaluated our method on five continuous control environments based on the Brax simulator (ant_dir, swimmer_dir, halfcheetah_vel, hopper_vel, ur5e, fetch).

* ant_dir: We train an ant agent to run in a target direction in this environment. The training task set includes \(8\) directions, uniformly sampled from \([0,360]\) degrees. As shown in Fig. 3**D**, the generalization test task set includes \(72\) directions, uniformly sampled from \([0,360]\) degrees. The agent's reward comprises speed along the target direction and control cost.
* swimmer_dir: In this environment, we train a swimmer agent to move in a fixed direction. The settings for training and testing tasks are similar to ant_dir.
* halfcheetah_vel: In the halfCheetah_vel environment, we train a half-cheetah agent to move forward at a specific speed. The training tasks include \(8\) speeds, uniformly sampled from [1, 10] m/s. The generalization test tasks include \(72\) different speeds, uniformly sampled from the same range as the training tasks.
* hopper_vel: In the hopper_vel environment, we train a hopper agent to advance at a specific speed. The experimental setup is the same as halfcheetah_vel, but the sampling interval for the speed is \([0,2]\) m/s.
* ur5e: The UR5e is a common 6-DOF (degrees of freedom) robotic arm frequently used in industrial automation and robotics research. The agent receives a reward when the distance between the robotic arm's end and the target position is less than \(0.02\) m. The target position is then randomly reset. The agent's goal is to reach the target position as many times as possible within the stipulated time.
* fetch: We train a dog agent to run to a target location in this environment. The experimental setup is similar to ur5e.

The agent's final reward is the average reward across all tasks, which encourages the agent to learn multiple tasks simultaneously.

### Training Strategies

We employ Parameter-Exploring Policy Gradients (PEPG) [33] to optimize SNNs. For SNNs with plasticity, the plasticity parameters in Eq. 1 are used for optimization. Evolution across generations is facilitated by modifying synaptic plasticity rules rather than directly adjusting the weights. SNNs with directly trained weights are considered a control group, where synaptic weights are the optimization parameters. The implementation of PEPG used in the experiments is provided by Algorithm 1. Unless expressly stated otherwise, the parameter settings and their explanations are shown in Table 3. The way to compute fitness \(f(\theta)\) varies depending on the task. For the working memory task, fitness is provided by Eq. 5, while for multi-task reinforcement learning, fitness is the average episodic reward across different subtasks.