Self-building Neural Networks
During the first part of life, the brain develops while it learns through a
process called synaptogenesis. The neurons, growing and interacting with each
other, create synapses. However, eventually the brain prunes those synapses.
While previous work focused on learning and pruning independently, in this work
we propose a biologically plausible model that, thanks to a combination of
Hebbian learning and pruning, aims to simulate the synaptogenesis process. In
this way, while learning how to solve the task, the agent translates its
experience into a particular network structure. Namely, the network structure
builds itself during the execution of the task. We call this approach
Self-building Neural Network (SBNN). We compare our proposed SBNN with
traditional neural networks (NNs) over three classical control tasks from
OpenAI. The results show that our model performs generally better than
traditional NNs. Moreover, we observe that the performance decay while
increasing the pruning rate is smaller in our model than with NNs. Finally, we
perform a validation test, testing the models over tasks unseen during the
learning phase. In this case, the results show that SBNNs can adapt to new
tasks better than the traditional NNs, especially when over $80\%$ of the
weights are pruned.
Andrea Ferigo, Giovanni Iacca
# Self-building Neural Networks

###### Abstract.

During the first part of life, the brain develops while it learns through a process called synaptogenesis. The neurons, growing and interacting with each other, create synapses. However, eventually the brain prunes those synapses. While previous work focused on learning and pruning independently, in this work we propose a biologically plausible model that, thanks to a combination of Hebbian learning and pruning, aims to simulate the synaptogenesis process. In this way, while learning how to solve the task, the agent translates its experience into a particular network structure. Namely, the network structure _builds itself_ during the execution of the task. We call this approach _Self-building Neural Network_ (SBNN). We compare our proposed SBNN with traditional neural networks (NNs) over three classical control tasks from OpenAI. The results show that our model performs generally better than traditional NNs. Moreover, we observe that the performance decay while increasing the pruning rate is smaller in our model than with NNs. Finally, we perform a validation test, testing the models over tasks unseen during the learning phase. In this case, the results show that SBNNs can adapt to new tasks better than the traditional NNs, especially when over 80% of the weights are pruned.

Neural networks, plasticity, pruning, neuroevolution +
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp. 1-10.

+
Footnote †: conference: conference: Proceedings of the 2018 ACM SIGGRAPH international conference on Knowledge discovery and data mining, ACM, 2018, pp.
can effectively be seen as a form of implicit pruning, especially in cases where synaptic connections can saturate their values in such a way to obtain a quasi-binary mask on the weights (Han et al., 2015; Wang et al., 2016; Wang et al., 2017; Wang et al., 2018). However, in these works, no explicit pruning mechanism has been employed.

## 3. Methods

As introduced before, we aim to construct a network that can simulate the synaptogenesis process. In this section, we introduce the structure of this network and its behavior. Then, we briefly describe the optimization process and the classical control tasks used in our experimentation. We used these environments to measure the performance of the proposed SBNNs and to prove how they can change their structure during (and depending on) the task at hand.

### Hebbian learning

Hebbian learning is a plasticity model that allows an NN to change its weights during the execution of a task. Importantly, this change is agnostic w.r.t. the reward for the task, because it is based only on the local knowledge of each synapse, in particular the activation of the pre-synaptic and post-synaptic neurons. The ABCD model used in this work updates the weights after each forward pass of the network using the following rule:

\[w_{i,j}=w_{i,j}+\eta(Aa_{i}+Ba_{j}+Ca_{i}a_{j}+D)\]

where \(a_{i}\) is the pre-synaptic activation value, \(a_{j}\) is the post-synaptic value, and \(w_{i,j}\) is the weight on the connection between the two neurons. The \(A\), \(B\), \(C\), and \(D\) are parameters to optimize.

### Pruning mechanism

The pruning mechanism aims to find a subset of an NN that performs as well as (or better than) the original network. This process is performed by removing connections based on a given strategy. In this work, we use the global magnitude pruning algorithm (Han et al., 2015; Wang et al., 2016) that simply consists in removing all the connections whose weights are smaller, in absolute value, than a threshold that is defined as the \(pr\)-th percentile, where \(pr\) is the desired pruning rate (i.e., the percentage of connections to remove).

### Self-building Neural Network

During synaptogenesis, neurons explore the extracellular space, assembling as many connections as possible with other neurons (Wang et al., 2016; Wang et al., 2017; Wang et al., 2018). Here, we aim to simulate this mechanism by allowing the possibility that any two neurons could directly connect.

Our model works as follows. We start from an NN composed of \(I\) inputs, \(H\) hidden nodes, and \(O\) outputs. At the first episode of the task, the \(I\) inputs are connected to all the \(H\) hidden nodes and the \(O\) outputs. In turn, the \(H\) hidden nodes are fully connected with each other (excluding self-loops), and with all the \(O\) outputs. In total, the number of connections \(C\), expressed as a function of \(H\) (which is the only hyperparameter, as \(I\) and \(O\) depend on the task), is \(C(H)=H^{2}+H\times(I+O)+I\times O\). Overall, the internal structure of the network resembles the one of the Boltzmann machines (Han et al., 2015; Wang et al., 2016; Wang et al., 2017; Wang et al., 2018), where the hidden nodes are fully connected. However, we also directly connect the inputs to the outputs, as the computational power of the hidden nodes in some cases may not be necessary.

We initialize the weights of all these connections to zero. In this way, we intend to simulate the initial condition where no connections between neurons exist. Then, _within_ each episode, the Hebbian procedure will update the weights based on the ABCD rule described in Section 3.1. Note that, in our model, each connection in the network has its own ABCD rule with its corresponding parameters. In this way, the network can arrange itself based on the experience that the agent accumulates during the task. We use a Hebbian rule for each connection because, starting from a condition where all the weights are 0, using a single Hebbian rule could lead all the weights to change in the same direction, which in turn would make learning ineffective.

The second step of synaptogenesis is the process that prunes the synapses, as described in Section 3.2. As we will describe later, differently from Hebbian learning, pruning occurs _across_ episodes. Note that, as soon as pruning starts, Hebbian learning is stopped. Figure 2 summarizes the pruning procedure, showing the state of the initial network and its development. Formally, we can analyze the network before and after pruning. In particular, before pruning, the hidden nodes are fully connected with each other and all the inputs are connected with all the hidden nodes. For this reason, it is not possible to define a fixed activation order. Hence, we maintain the overall order of activation: firstly, the inputs, then the hidden nodes, and then the outputs. However, for the hidden nodes, we randomly select the activation order.

After pruning, the remaining connections define the network. Differently from before, in this phase we can define an activation order more easily because the pruning mechanism naturally resolves most cycles, especially if the ratio of connections removed is high enough. Hence, to find the activation order, we can perform a topological sort of the underlined graph \(G(V,E)\), where \(V\) is the set of nodes (i.e., the neurons) and \(E\) is the set of connections. If, during the topological sort, we find a cycle, we apply the following procedure. Indicating with \(N_{c}\) the subset of \(V\) that contains all the nodes in the cycle, first we remove all the nodes in \(N_{c}\) from \(V\) and replace them with a _fake_ node, \(f\). Then, indicating with \(E(N_{c})_{incoming}=\{(o,n)\ \forall o\in V\setminus N_{c}\ \wedge\ \forall n\in N_{c}\}\) the subset of \(E\) composed of the connections that terminate in \(N_{c}\) and that do not start from \(N_{c}\), we add to \(E\) the set of connections \(\{(o,f)\ \forall(o,o^{\prime})\in E(N_{c})_{incoming}\ \wedge\ \nicefrac{{\pi}}{{\nu}}\in N_{c}\}\). We perform the same operation for the connections outgoing from \(N_{c}\). This procedure, that we apply iteratively and independently for every cycle found in the graph, results in a new graph \(G^{\prime}\) where the cycle \(N_{c}\) is replaced by the fake node \(f\). All the nodes connected to \(N_{c}\) are now connected to \(f\), and \(f\) is connected to all the nodes reached from \(N_{c}\). We store the information that the node \(f\) replaces the \(N_{c}\) nodes in the _cycles_history_ variable and then retry find a topological order. We repeat this procedure until all the cycles have been replaced, and a topological order can be defined. Algorithm 1 illustrates this simplification procedure. Note that the nodes in \(N_{c}\) can also be fake nodes from a previous iteration of the procedure, as illustrated in Figure 1.

After calculating the topological order of the network, we can follow that for the activation of the hidden nodes. If we find a fake node during the activation, we retrieve from _cycles_history_ the set of \(N_{c}\) nodes that compose the cycle, and proceed with a random activation order. If a node in \(N_{c}\) is a fake node \(f^{\prime}\) covering the
cycle, we repeat the procedure solving the inner cycle \(N_{\mathcal{C}}\), before continuing with the nodes in \(N_{\mathcal{C}}\).

Thanks to this process, the network after pruning can have a different structure. We identify three base structures, which can be described as follows. In the first case, the inputs are connected to all the hidden nodes that in turn are connected to the outputs. This creates an NN with a single hidden layer, see Figure 3(a). In the second case, the pruning process cuts all the connections between the input and hidden nodes. Hence, the inputs are directly connected to the output nodes, creating a zero-layer NN, see Figure 3(b). In the third case, the pruning process removes all the connections between the inputs and a subset of the hidden nodes, but the hidden nodes remain connected, creating an NN with more than one layer. For example, given the hidden nodes \(A\), \(B\), and \(C\), if the inputs are connected only with \(A\) and \(B\), but not with \(C\), which in turn remains connected with \(A\) and \(B\), the resulting NN will have two hidden layers: the first one, composed of neurons \(A\) and \(B\); the second one, composed only neuron \(C\), see Figure 3(c).

It is worth noticing that, starting from these three base structures, we can derive more complex structures. For example, each node in the third case can be a _fake_ node, hence "hiding" a set of nodes.

While in principle promising, this model is not free from weak points. First of all, the number of weights in the SBNN increases quadratically with the number of hidden nodes, as each hidden node is fully connected with all the other hidden nodes. Moreover, each connection is associated with an ABCD rule with its corresponding 4 parameters. Therefore, the total number of parameters to optimize is orders of magnitude greater with respect to a Feed Forward Neural Network (FFNN) with the same number of hidden nodes. Figure 3 shows a comparison in terms of number of parameters to optimize between the proposed SBNN, and two other NN-based models, namely an FFNN with and without Hebbian learning.

Secondly, before pruning the hidden nodes compose a single, fully connected subnetwork on which the activation order can influence the network's output. For example, if we consider a fixed activation order, the first hidden node will receive as inputs the values of the inputs at the current timestep, while, for the other hidden nodes, the values received as inputs will be the ones from the previous activation. On the other hand, as discussed above, after pruning we can find the topological order for network by visiting the underlying graph. Still, in the presence of a subset of hidden nodes that are linked together, we cannot determine a unique activation order.

### OpenAI tasks

To measure the performance of the proposed SBNN, we use three classical control tasks from OpenAI (Dong et al., 2018), namely _Cart Pole_, _Mountain Car_, and _Lunar Lander_.

In Cart Pole, the agent has to move a cart to maintain a pole in equilibrium. The agent can push the cart in both directions (move left/right), and it receives a positive reward for each timestep in which the pole is in equilibrium. The episode ends after 500 timesteps, or if the angle of the pole is outside the range \(\pm 12\)deg.

In Mountain Car, the agent has to drive a car from a valley to the top of a mountain. The agent has to build momentum to increase its velocity, thanks to another hill positioned before it. The agent can perform three actions (accelerate left, accelerate right, or do not accelerate), and receives a negative reward at each timestep until it reaches the top of the mountain. The episode ends if the car reaches the top of the hill, or after 200 timesteps.

Finally, in Lunar Lander, the agent has to land a spaceship. The agent has to reduce the terminal velocity of the spaceship while compensating for the lateral wind. Hence, the agent can perform four actions: two to control the lateral (left/right) engines, one to activate the main engine, and one that does not perform any action. The agent increases the received reward if it lands in the designated

Figure 1. A minimal example of the simplification procedure described in Algorithm 1. The base graph has two cycles: one composed of nodes \(A\) and \(B\), and one composed of nodes \(B\) and \(C\). The procedure identifies first the cycle \(B,C\), circled in red, and replaces it with the red fake node \(F_{1}\), which has incoming connections from \(A\) and \(I\) and outgoing connections to \(A\) and \(O\). Then, it finds the cycle \(F_{1},A\), circled in blue, and replaces it with the blue fake node \(F_{2}\). Eventually, a directed acyclic graph is obtained, with a single fake hidden node \(F_{2}\), which hides the cycle \(A,F_{1}\), where \(F_{1}\) is another fake node hiding the nodes \(B\) and \(C\).
area at a lower speed and does not tilt. The episode ends if the spaceship lands, or if its \(x\) position is greater than 1.

The three tasks above are solved if the average reward over 100 episodes is greater than a predefined threshold, which is 475 for Cart Pole, \(-110\) for Mountain Car, and 200 for Lunar Lander.

## 4. Results

In this section, we will analyze the performance and behavior of the SBNN in comparison with a FFNN for which we apply the same pruning mechanism used in the SBNN, but before the fitness evaluation. Note that, in our implementation, we use as activation function the \(tanh\) function, both on the hidden nodes and on the input/output nodes, for both the SBNN and the FFNN. Concerning the fitness evaluation, we measure the performance of the agent as the average reward over 100 episodes seen during training. For the FFNN, as the pruning process happens before the first episode, in this case the fitness by construction is measured after pruning. On the contrary, as for the SBNN the pruning process happens during the life of the agent (i.e., across episodes), in this case the fitness contains two components, one before and one after pruning (which are then averaged). We divide our experiments into three parts, to answer three different research questions:

1. [leftmargin=*]
2. What is the performance of the SBNN? What are the main hyperparameters of this model that affect the performance?
3. Is there any structural difference between the networks produced by the SBNN and an FFNN?
4. Are SBNNs able to generalize over the different tasks?

To answer these questions, we perform a campaign of simulations varying the three main hyperparameters of the SBNN: the number of hidden nodes \(hn\), the pruning rate \(pr\), and the pruning time \(pt\), the latter indicating when pruning is applied. To calculate the pruning time, we consider the number of episodes in the task, i.e., a pruning time of 10 means that pruning happens after the agent completes the 10-th episode.

For each combination of these parameters, we perform 30 independent evolutionary processes. To optimize the parameters of the network (i.e., the weights for the FFNN, or the parameters of the ABCD rules for the SBNN), we use the well-known Covariance Matrix Adaptation Evolution Strategies (CMA-ES) (Han et al., 2016; Krizhevsky et al., 2017; Krizhevsky et al., 2017; Krizhevsky et al., 2017). We stop the evolution after the generation of a fixed number of 2000 individuals for Lunar Lander and Cart Pole, and 4000 for Mountain Car. In all cases, we set \(\lambda=4+\lfloor 3*ln(|\mathbf{p}|)\rfloor\) and \(\mu=\frac{1}{2}\), where \(\mathbf{p}\) is the vector of parameters to optimize. Table 1 summarizes the configurations tested. Note that we omit the results obtained on the Cart Pole task as there were no significant differences between the FFNN and the SBNN: in fact, all the individuals using both models solved the task, that is comparatively simpler than the other two. We make our code publicly available at [https://github.com/ndr09/SBM](https://github.com/ndr09/SBM).

### RQ1: Performance

Concerning the performance of the SBNN, we aim to evaluate how it compares with that of an FFNN. Since the SBNN has more connections than the FFNN given the same number of hidden nodes, we make two comparisons: one comparing the results of the two

Figure 4. Pruning can create different structures in the SBNN. In Figure 3(a), an NN with one layer is created. In Figure 3(b), pruning cuts all the connections to the inner nodes, reducing the NN only to the input-output connections. Finally, in Figure 3(c), pruning results in the creation of two hidden layers.

Figure 3. Number of parameters to optimize with respect to the number of hidden nodes. For the (Hebbian) FFNN, we consider an NN with two hidden layers with the same number of neurons indicated on the x-axis. In all cases, a single input and a single output are considered. The FFNN with Hebbian learning uses a different ABCD rule for each connection.

Figure 2. Scheme of the SBNN over the episodes. Initially, we set all the connections to 0 (red); then during the task Hebbian plasticity changes the weights, leading to the second NN, where different thickness indicates different weights. At a certain time, the pruning mechanism cuts the weakest connections, resulting in the final structure of the NN.
models given the same number of hidden nodes, and one given the same total number of connections after pruning. In the following, we start with the Mountain Car task and then move to describe the results for the Lunar Lander one.

Figure 5 shows the results for the Mountain Car environment. The upper and lower row relate, respectively, to an FFNN with one layer with 3 or 4 hidden nodes. The left and right column present, respectively, the results with a pruning rate of 40% and 60%. In each subfigure, we plot the average results of the best individual over 30 independent runs. The first three boxplots indicate the results of the SBNN with different pruning times, namely 10, 5, and 1, respectively from left to right. The last boxplot shows the baseline results of the FFNN. In Mountain Car, the results indicate that the SBNN reaches similar or better performance with respect to an FFNN with the same pruning rate.

Interestingly, we observe a clear trend on the pruning time, i.e., the performance increases when decreasing the value of \(pt\), regardless of the pruning rate and the number of hidden nodes. Hence, we can conclude that, after the first episode, the agent has already received enough experience (information) to build the network.

To understand the effect of pruning on the performance, we make an additional analysis, by comparing the average reward before and after pruning. For instance, considering \(pt=10\), we measure separately the average reward until the 10-th episode, i.e., before pruning, and the average reward after the 10-th episode, i.e., the post-pruning one. Based on this procedure, we observe that, thanks to pruning, the performance receives a \(7-13\%\) boost on the Mountain Car task.

Figure 6 presents the results for the Lunar Lander environment. For this environment, we consider hidden nodes ranging from 5 to 9, because of the greater complexity of the task. Here, the results are shown differently from Figure 5: in particular, we plot the median rewards for the best individuals of each evolutionary run, varying the pruning rate while keeping the pruning time and the number of hidden nodes fixed. In this way, we highlight two points. The first one is that the performances of the SBNNs are in most cases equal to or better than the ones obtained by the FFNN for the same pruning rate. The second point is that, while increasing the pruning rate, the drop in performance for the SBNN is lower than what observed with the FFNN baseline. We can also observe that this trend is maintained when comparing solutions with a comparable number of connections. For example, the SBNN with 5 hidden nodes and the FFNN with 9 hidden nodes have a similar number of connections (respectively, 117 and 108). Moreover, we can see the same trend observed in Mountain Car, i.e., that the best results are achieved with \(pt=1\). Hence, also in this task it appears that a single episode contains enough information to learn about the prunable connections. These observations suggest that, at least in the tested tasks, the SBNN is effectively capable to exploit the network structure (and the information therein) better than the FFNN.

Also in this case, we perform the same analysis of before, dividing the results before and after pruning and observing the average results. As for the case of Mountain Car, also for Lunar Lander we can see an increase in performance after pruning, in this case between 6% and 100%. This improvement indicates, once again, the importance of pruning and its complementary effect with respect to Hebbian learning.

Finally, we compare the results of the networks based on total number of connections after pruning. In Figure 7, we compare the FFNN with \(pr=20\%\) with the SBNN with \(pr=60\%\) (with these values, in fact, the FFNN and the SBNN have a similar number of connections after pruning). We can observe that the SBNN reaches almost always a comparable or slightly better performance with respect to the FFNN. However, in the case where the FFNN performs better than the SBNN (i.e., with 9 nodes), the difference is not statistically significant (Wilcoxon rank-sum test, \(\alpha=0.01\)).

### RQ2: Difference between SBNN and FFNN

In this section, we want to analyze the structural difference between the network found with the SBNN and the FFNN. For this reason, we characterize the networks based on the number of _working connections_, i.e., the connections that link inputs to outputs after pruning. To calculate these connections, we remove the synapses that lead to sink or come from source nodes. While a sink is a node

\begin{table}
\begin{tabular}{l c c c} \hline \hline Parameter & Cart Pole & Mountain Car & Lunar Lander \\ \hline Fitness evaluation & 2000 & 4000 & 2000 \\ Hidden nodes & 3, 4 & 3, 4 & 5, 6, 7, 8, 9 \\ Pruning time & 5, 10 & 1, 5, 10 & 1, 5, 10, 15, 20 \\ Pruning rate & 40, 60 & 40, 60 & 20, 40, 60, 80 \\ \hline \hline \end{tabular}
\end{table}
Table 1. Parameter configuration used for each task considered in the RQ1 experiments. For RQ2 and RQ3, we use a representative subset of these configurations.

Figure 5. Results on the Mountain Car environment. The y-axis shows the average reward obtained during training from the best agent found in each of 30 runs. The dashed line indicates the solving threshold for the environment. For the SBNN, the results indicate a clear trend where the performance increases while \(pt\) decreases.
with only incoming connections that is not an output node, a source is a node with only outgoing connections that is not an input node.

Figure 8 and Figure 9 show the distribution of working connections for a representative subset of the configurations presented in the previous section, considering the best solution (one per each evolutionary run) obtained for each considered configuration. On the x-axis, we indicate the percentage of remaining working connections (after pruning) with respect to the total number of synapses, grouped every 10%, while on the y-axis we indicate how many networks (out of 30, one per run) have that number of connections. For example, if we consider a point at \(x=40\%\), \(y=0.5\), we mean that in 50% of runs (i.e., 15 out of 30) networks after pruning have a number of working connections in the range (30%, 40%).

In both figures, we can observe a quite clear pattern: all the FFNN configurations use the majority of the connections available. On the other hand, the distributions of working connections for the SBNN have two peaks: the first one occurs between 10% and 20% for the Mountain Car task and between 20% and 40% for the Lunar Lander one; the second peak is in common between the two tasks at around 90%. We visually analyzed all the networks and discovered that the SBNNs that compose the first peak have a structure like the one shown in Figure 4b, where all the hidden nodes are disconnected. Interestingly, the percentage of this kind of structures increases when \(pt\) increases, as the first peak is higher for higher values of \(pt\). This suggests that the later pruning occurs, the more probable it is that connections to the hidden nodes are pruned, thus leaving only input-output connections. Our intuition is that this form of simplification somehow correlates with the complexity needed to solve the task.

Concerning the total number of connections, we observe that in the Lunar Lander environment SBNNs maintain between 10 and 30 working connections for pruning rates higher than 20%. This range appears independent on the pruning time and the number of hidden nodes available. On the contrary, FFNNs use the majority of the connections available, as for Mountain Car. Hence, in this case the number of working connections is strongly dependent on the number of hidden nodes and the pruning rate, resulting in a total number of working connections between 10 and 64. The fact that the number of working connections varies is especially relevant when comparing the SBNN and the FFNN: for example, on the Lunar Lander task, the FFNN with 9 hidden nodes and a 40% pruning rate uses all the 64 connections available, while the SBNN

Figure 6: Median reward on the Lunar Lander task for different numbers of hidden nodes. The x-axis indicates the pruning rate, while the different lines refer to different values of pruning time. The red dashed line indicates the solving threshold. The results show that while increasing the pruning rate, the performance drop for the SBNN is lower than for the FFNN.

Figure 7: Average reward on the Lunar Lander task for a SBNN with a pruning rate of \(60\%\) and an FFNN with a pruning rate of \(20\%\). The x-axis indicates the number of hidden nodes (note that in this setting the two models have a similar number of total connections after the pruning process). The results show that the SBNN reaches similar or slightly better performances with respect to the FFNN. For 9 hidden nodes, the FFNN seems to perform slightly better, but the difference is not statistically significant (Wilcoxon rank-sum test, \(\alpha=0.01\)).
solves the task and obtains better performance using on average only \(20\) connections, see Figure 6.

In Figure 10, we show two SBNNs (after pruning) trained on the Mountain Car task. These networks are of two different kinds: one where all the hidden nodes have been removed, and that uses only two actions (accelerates left and accelerate right); and one that uses uses \(2\) out of \(4\) hidden nodes available, and all the three available actions.

### RQ3: Generalization

In this section, we evaluate the generalization capabilities of the SBNN by testing the best agent (found after an evolution process on a given task) on another, unseen task.

In particular, we test the agents trained on Lunar Lander on the Cart Pole and Mountain Car tasks. We restrict the validation to this case, as the Lunar Lander environment is the only one with enough input and output nodes to perform the other two tasks. In fact, the different tasks have different input and action spaces.

To perform this analysis, we remap inputs and outputs from the validation task to the relative inputs in the Lunar Lander environment. For example, if in the Lunar Lander environment the first input is the x-position, we map the observation of the position from the validation task (i.e., the x-position of the cart or the car, respectively for Cart Pole and Mountain Car), to the first input. We set all the unused inputs to \(0\). Regarding the output, we consider only the outputs present in the validation task. For example, considering the validation task as Cart Pole, the only actions available are _move left_ and _move right_. Hence, we consider only the outputs that control the left and right engines in Lunar Lander.

Figure 12 shows the performances in the validation tasks for a subset of the configurations (statistical significance assessed with Wilcoxon rank-sum test, \(\alpha=0.05\)). On the Cart Pole task, the results

Figure 8. Distribution of the number of connections after pruning the network on the Mountain Car task. On the x-axis, we show the percentage of working connections, which are connections that do not lead to a sink node or are outgoing from a source node, grouped every \(10\%\). On the y-axis, we show how many networks (out of \(30\), one per run) have that percentage of working connections. The results show that, for a pruning rate higher than \(20\%\), FFNN tends to use all the available connections, while SBNN has two peaks: one around \(10-20\%\), and one at \(80-90\%\).

Figure 10. Structure of two selected SBNNs obtained on the Mountain Car task. On the left, an SBNN that uses \(2\) out of \(4\) hidden nodes and a \(2\)-layer structure. On the right, an SBNN that uses only two connections.

Figure 9. Distribution of the number of connections after pruning the network on the Lunar Lander task. On the x-axis, we show the percentage of working connections, which are connections that do not lead to a sink node or are outgoing from a source node, grouped every \(10\%\). On the y-axis, we show how many networks (out of \(30\), one per run) have that percentage of working connections. The results show that, the available connections, while SBNN has two peaks: one around \(20-40\%\), and one at \(80-90\%\).
of the SBNN are similar to or better than the ones of the FFNN. In the Mountain Car environment, the SBNN performs slightly worse than the FFNN for \(pr=20\%\), although the differences are not always statistically significant; for \(pr=80\%\), the SBNN shows better performances, with higher performance in the first quartile (indicating, once again, that the SBNN models effectively learns to use the network structure).

Finally, with these validation experiments, we can observe how the task affects the network structure. Figure 11 shows the network structure of the same agent, which can solve both the Lunar Lander and the Cart Pole task after the pruning process. We can observe that the networks differ in the number of connections used and in the neurons that those synapses connect.

## 5. Conclusions

In this work, we took inspiration from the synaptogenesis process occurring in natural brains to propose a new learning model that combines both plasticity and pruning. At the beginning of the first episode of the task, all the neurons are connected with each other and initialized to 0 (i.e., the connections exist, but they are initially deactivated). Then, within each episode, a plasticity model based on Hebbian learning grows those synapses, activating the corresponding connections. Eventually, during the life of the agent (i.e. at a predetermined episode), those connections are pruned through a global magnitude pruning algorithm and Hebbian learning is stopped. We called this model Self-building Neural Network (SBNN), as it changes its structure based on the experience of the agent during the episodes of the task.

We tested our model on three classical control tasks from the OpenAI, namely Cart Pole, Mountain Car, and Lunar Lander. In our experiments, we varied the three main parameters of the model, affecting respectively _when_ to prune, _how much_ to prune, and the number of hidden nodes. We showed that, in general, the SBNN reaches better performance than the FFNN, and that it can adapt better to unseen tasks. Furthermore, we assessed the importance of the model's parameters, in particular regarding when and how much to prune. Finally, we highlighted how the same agent reorganizes its brain differently, based on the task, and how it can remove unnecessary complexity from the brain, given enough time.

In the future, we plan to develop a system for automatically deciding how much and when to prune based on the information flow in the network. In addition, we aim to tackle more complex tasks, such as the control of soft robots, where we can test the proposed SBNN on larger input and action spaces. We also plan to address the two limitations indicated in Section 3. For the number of parameters to optimize, we will test the use of a Hebbian rule for each neuron, rather than one for each connection, also modifying the update rule to avoid that each connection receives the same update. For the activation order, we plan to use a distance-based approach to solve the cycle, using as the distance the weights.
