Training neural networks with structured noise improves classification
  and generalization
The beneficial role of noise-injection in learning is a consolidated concept
in the field of artificial neural networks, suggesting that even biological
systems might take advantage of similar mechanisms to optimize their
performance. The training-with-noise algorithm proposed by Gardner and
collaborators is an emblematic example of a noise-injection procedure in
recurrent networks, which can be used to model biological neural systems. We
show how adding structure to noisy training data can substantially improve the
algorithm performance, allowing the network to approach perfect retrieval of
the memories and wide basins of attraction, even in the scenario of maximal
injected noise. We also prove that the so-called Hebbian Unlearning rule
coincides with the training-with-noise algorithm when noise is maximal and data
are stable fixed points of the network dynamics.
Marco Benedetti, Enrico Ventura
# Training neural networks with structured noise improves classification and generalization

###### Abstract

The beneficial role of noise in learning is nowadays a consolidated concept in the field of artificial neural networks. The training-with-noise algorithm proposed by Gardner and collaborators is an emblematic example of a noise injection procedure in recurrent networks. We show how adding structure into noisy training data can substantially improve memory performance, allowing to approach perfect classification and maximal basins of attraction. We also prove that the so-called unlearning rule coincides with the training-with-noise algorithm when noise is maximal and data are fixed points of the network dynamics. Moreover, a sampling scheme for optimal noisy data is proposed and implemented to outperform both the training-with-noise and the unlearning procedures.

## I Introduction

Consider a fully connected network of \(N\) binary variables \(\{S_{i}=\pm 1\}\), \(i\in[1,..,N]\), linked by couplings \(J_{ij}\). The network is endowed with a dynamics

\[S_{i}(t+1)=\mathrm{sign}\left(\sum_{j=1}^{N}J_{ij}S_{j}(t)\right),\qquad i=1,..,N \tag{1}\]

which can be run either in parallel (i.e. _synchronously_) or in series (i.e. _asynchronously_ in a random order) over the \(i\) indices. We will mainly concentrate on asynchronous dynamics, in which case eq. (1) can only converge to fixed points, when they exist [1]. This kind of network can be used as an associative memory device, namely for reconstructing a number \(p\) of configurations \(\{\xi_{i}^{\mu}\}=\pm 1\), \(\mu\in[1,...,p]\), called _memories_, when initialized into a configuration similar enough to one of them. In this work, we will concentrate on i.i.d. memories, generated with a probability \(P(\xi_{i}^{\mu}=\pm 1)=1/2\). The number of memories is extensive \(p=\alpha N\), where \(\alpha\) is called _load_ of the network. The network is considered to perform well if such asymptotic states are similar enough to the memories. Whether this is the case, depends on the choice of the coupling matrix \(J\). To give an example, Hebb's (or Hebbian) learning prescription [2; 3]

\[J_{ij}^{H}=\frac{1}{N}\sum_{\mu=1}^{p}\xi_{i}^{\mu}\xi_{j}^{\mu} \tag{2}\]

is a rudimentary yet effective rule that allows to retrieve memories up to a critical capacity \(\alpha_{c}^{H}\sim 0.14\)[4]. Notably, when \(\alpha<\alpha_{c}^{H}\) memories are not perfectly recalled, but only reproduced with a small number of errors. Several techniques have been developed to build better performing coupling matrices, i.e. to reduce the retrieval error and increase the critical capacity as well as the size of the basins of attraction to which the memories belong. Three significant examples are the unlearning algorithm [5; 6; 7; 8; 9; 10], the linear perceptron algorithms [11; 12; 13], and the training-with-noise algorithm [14]. All these procedures rely on iteratively modifying the couplings on the basis of an initial choice of the couplings and a set of training data presented to the network. Each step of the unlearning algorithm does not explicitly use the memories, and only exploits the information encoded in Hebb's rule. On the other hand, the linear perceptron is trained with the pure memories until they become attractors of the network dynamics. In analogy with a noisy linear perceptron, Gardner and collaborators proposed the training-with-noise algorithm, managing to increase the size of the basins of attraction while conserving a rather high accuracy in the memory retrieval. In this context, _noise_ is defined as the degree of similarity between training data and memories: the higher the noise, the more the presented data differ from the memories.

The main goal of this work is to show that, the training-with-noise algorithm can be considerably improved, when training data contain internal dependencies, that we call _structure_. Specifically, in case of maximal noise, we derive a condition to be satisfied by the structure to drive learning towards the best memory performance. We also show that not all initial conditions are supplied with the best training data, and that previous Hebbian learning already provides for a good noise structure. Remarkably, we have found that the unlearning rule emerges from training-with-noise in the particular case of maximal noise and training data chosen to be fixed points of (1). We also developed an effective sampler device for optimal noisy training data that can be used to reach higher critical capacities while preserving rather large basins of attraction of the memories.

The paper is organized as follows: in Sec. II we define some measures for the network performance, in particular the concepts of classification and generalization. In Sec. III and Sec. IV we describe the unlearning and the training-with-noise algorithms and their effects on the neural network. Successively, in Sec. V the role of noise structure in training data is extensively discussed,
with a particular attention on the cases of maximal and moderate noise injection. Eventually in Sec. VI we use insights from the previous sections to devise a performing sampling algorithm for training data.

## II Network performance

In this section we describe how to benchmark the neural network performance, specifically in terms of the dynamic stability achieved by the memory vectors and the ability of the system to retrieve blurry examples of the latter.

We define _classification_ as the capability to perfectly retrieve each memory when the dynamics is initialized to the memory itself. The fraction among all \(pN\) units \(\xi_{i}^{\mu}\) that are stable according to one step of the dynamics (1) will be called \(n_{SAT}\), in analogy with the celebrated perceptron optimization problem [11; 13]. Classification is reached when \(n_{SAT}=1\).

We define _generalization_ as the capability to retrieve the memory, or a configuration that is strongly related to it, by initializing the dynamics on a noise-corrupted version of the memory. This property of the neural network is related to the size of the basins of attraction to which the memories belong, and does not imply \(n_{SAT}=1\). A good measure of the performance in this sense is the _retrieval map_

\[m_{f}(m_{0}):=\overline{\Big{\langle}\frac{1}{N}\sum_{i=1}^{N}\xi_{i}^{\nu}S _{i}^{\nu}(\infty)\Big{\rangle}}. \tag{3}\]

Here, \(\vec{S}^{\nu}(\infty)\) is the stable fixed point reached by the network, when it exists, when the dynamics is initialized to a configuration \(\vec{S}^{\nu}(0)\) having overlap \(m_{0}\) with a given memory \(\vec{\xi}^{\nu}\). The symbol \(\overline{\phantom{\Big{|}}}\) denotes the average over different realizations of the memories and \(\langle\cdot\rangle\) the average over different realizations of \(\vec{S}^{\nu}(0)\). In the classification regime, one obtains \(m_{f}=1\) when \(m_{0}=1\). The analytical computation of the retrieval map might be challenging for some networks. Hence one can introduce another indicative observable for the generalization, i.e. the _one-step retrieval map_\(m_{1}(m_{0})\)[15], defined by applying a single step of _synchronous_ dynamics (1):

\[m_{1}(m_{0}):=\overline{\frac{1}{N}\sum_{i=1}^{N}\Big{\langle}\xi_{i}^{\nu} \ \mathrm{sign}\Big{(}\sum_{j=1}^{N}J_{ij}S_{j}^{\nu}(0)\Big{)}\Big{\rangle}}\;, \tag{4}\]

This work is devoted to exploring the characteristics of the training process leading to classification and good generalization.

## III Unlearning

Inspired by the brain functioning during REM sleep [6], the unlearning algorithm [5; 6; 7; 8; 9] is a training procedure leading to classification and good generalization in a symmetric neural network. This section contains an introduction to the algorithm as well as a description of the gained performance in terms of classification and generalization.

Training starts by initializing the connectivity matrix according to Hebb's rule eq. (2) (i.e. \(J^{0}=J^{H}\)). Then, the following procedure is iterated at each step \(d\):

1. Initialize of the network dynamics (1) on a random neural state.
2. Run the asynchronous dynamics until convergence to a stable fixed point \(\vec{S}^{(d)}\).
3. Update couplings according to: \[\delta J_{ij}^{(d)}=-\frac{\lambda}{N}S_{i}^{(d)}S_{j}^{(d)}\qquad\quad J_{ii }=0\quad\;\forall i.\] (5)

This algorithm was first introduced to prune the landscape of attractors from proliferating spurious states, i.e. fixed points of (1) not coinciding with the memories [1; 16]. Such spurious states are only weakly correlated with the memories. Even though this pruning action leads to the full stabilization of the memories, the exact mechanism behind this effect is not completely understood. The analysis in [9] gives deeper insights in both the classification and generalization capabilities of a network trained with unlearning.

### Classification

The classification property is achieved running the algorithm when \(\alpha\leq\alpha_{c}^{U}\) with \(\alpha_{c}^{U}\simeq 0.6\). To quantify the classification performance of the network, one can track numerically the observable

\[\Delta_{\min}=\overline{\min_{i,\mu}\left(\Delta_{i}^{\mu}\right)}, \tag{6}\]

where the _stability_\(\Delta_{i}^{\mu}\) is defined by

\[\Delta_{i}^{\mu}=\frac{\xi_{i}^{\mu}}{\sqrt{N}\sigma_{i}}\sum_{j=1}J_{ij}\xi_{ j}^{\mu},\qquad\sigma_{i}=\sqrt{\sum_{j=1}^{N}J_{ij}^{2}/N}. \tag{7}\]

As soon as \(\Delta_{\min}>0\), memories become fixed points of the dynamics [11]. Fig. 1 reports the evolution of \(\Delta_{\min}\) as a function of the number of performed updates of \(J\), for \(\alpha=0.3\). The amount of iterations \(d=d_{in}\) (indicated with a red circle) marks when \(\Delta_{\min}\) crosses \(0\). At this point, all the memories are fixed points of the dynamics. Other two points, \(d=(d_{top},d_{fin})\) are reported in the plot, even though [9] has proved \(d_{in}\) to show the best generalization performance in the large \(N\) limit. The scaling of \((d_{in},d_{top},d_{fin})\) is also described in [9]. The progressive decay of \(\Delta_{\min}\), that becomes negative again after \(d=d_{fin}\), is due to the vanishing of the first two moments of the couplings.
### Generalization

The unlearning algorithm creates large basins of attraction around the memories, when \(\alpha<\alpha_{c}^{U}\). Fig. 2 reports the retrieval map for \(N=100\) and \(\alpha=0.3\). The curves relative to the SVM and unlearning at \(d=d_{in}\) coincide. This observation is coherent with estimates for the large \(N\) limit performed in [9]. The SVM is trained with no symmetry constraints, though it superimposes with the symmetric version discussed in [9; 12; 17]: this is probably due to the high degree of symmetry displayed by SVMs. Fig. 2 also shows that curves for \(d=d_{in}\) and \(d=d_{top}\) do overlap significantly. This is certainly due to finite size effects, since [9] clearly shows the basins to progressively deteriorate, in the large \(N\) limit, along the interval \(d\in[d_{in},d_{fin}]\).

One open issue is why the optimum is found when \(\Delta_{\min}=0\) (i.e. \(d=d_{in}\)) and not when \(\Delta_{\min}\) is maximum (i.e. \(d=d_{top}\)). A second question regards why unlearning, and the choice of using fixed points as training data, should lead to maximally large basins of attraction.

## IV Training with noise

The concept of learning from noisy examples, introduced for the first time in [18], is at the basis of Gardner and co-workers work [14], a pioneering attempt to increase and control generalization through the introduction of noise during the training phase of recurrent neural networks. Here, we report the algorithm and characterize, for the first time, its performance over fully connected neural networks.

The training-with-noise algorithm [14] consists in starting from any initial coupling matrix \(J_{ij}^{0}\) with null entries on the diagonal, and updating recursively the couplings according to

\[\delta J_{ij}^{(d)}=\frac{\lambda}{N}\epsilon_{i}^{\mu_{d}}\xi_{i}^{\mu_{d}}S _{j}^{\mu_{d}},\hskip 28.452756pt\delta J_{ii}^{(d)}=0\hskip 8.535827pt\forall i, \tag{8}\]

where \(\lambda\) is a small learning rate, \(\mu_{d}\in[1,...,p]\) is a randomly chosen memory index and the mask \(\epsilon_{i}^{\mu_{d}}\) is defined as

\[\epsilon_{i}^{\mu_{d}}=\frac{1}{2}\Big{(}1-\text{sign}\Big{(}\xi_{i}^{\mu_{d} }\sum_{k=1}^{N}J_{ik}S_{k}^{\mu_{d}}\Big{)}\Big{)}. \tag{9}\]

In this setting, \(\vec{S}^{\mu_{d}}\) is a noisy memory, generated according to

\[P(S_{i}^{\mu_{d}}=x)=\frac{(1+m_{t})}{2}\delta(x-\xi_{i}^{\mu_{d}})+\frac{(1-m _{t})}{2}\delta(x+\xi_{i}^{\mu_{d}}). \tag{10}\]

The _training overlap_\(m_{t}\) is a control parameter for the level of _noise_ injected during training, corresponding to the expected overlap between \(\vec{S}^{\mu_{d}}\) and \(\vec{\xi}^{\mu_{d}}\), i.e.

\[m_{t}=\frac{1}{N}\sum_{j=1}^{N}\xi_{j}^{\mu_{d}}S_{j}^{\mu_{d}}+O\Big{(}\frac {1}{\sqrt{N}}\Big{)}. \tag{11}\]

Figure 1: The minimum stability \(\Delta_{\min}\) as a function of the normalized algorithm time. The threshold \(\Delta=0\) is indicated with the _gray_ dotted line. Three relevant amount of iterations are indicated by the colored circles: \(d=d_{in}\) in _red_, \(d=d_{top}\) in _orange_, \(d=d_{fin}\) in _yellow_. All measures are averaged over 50 realizations of the network. Choice of the parameters: \(N=100\), \(\alpha=0.3\), \(\lambda=10^{-2}\).

Figure 2: Retrieval map \(m_{f}(m_{0})\) for a SVM and the unlearning algorithm at the three relevant steps indicated in fig. 1. All measures are averaged over 10 realizations of the network. Choice of the parameters: \(N=100\), \(\alpha=0.3\), \(\lambda=10^{-2}\) for unlearning and \(\lambda=1\) for the SVM.
Each noisy configuration can be expressed in terms of a vector of _noise units_\(\vec{\chi}\), such that

\[S_{i}^{\mu_{d}}=\chi_{i}^{\mu_{d}}\xi_{i}^{\mu_{d}}. \tag{12}\]

In this setting, noise units are i.i.d variables, distributed according to

\[P(\chi_{i}^{\mu_{d}}=x)=\frac{(1+m_{t})}{2}\delta(x{-}1){+}\frac{(1-m_{t})}{2} \delta(x{+}1). \tag{13}\]

The algorithm would converge when every configuration with overlap \(m_{t}\) with a memory generates on each site a local field aligned with the memory itself. Let us define the function

\[\mathcal{L}(m,J)=-\frac{1}{\alpha N^{2}}\sum_{i,\mu}^{N,p}\text{erf}\left( \frac{m\Delta_{i}^{\mu}}{\sqrt{2(1-m^{2})}}\right). \tag{14}\]

Assuming that stabilities are self-averaging quantities, it can be proven that \(-\mathcal{L}(m=m_{0},J)\) tends to \(m_{1}(m_{0})\) when \(N\to\infty\)[19; 15]. Each step of the training procedure (8) leads to a reduction of \(\mathcal{L}(m,J)\), for any value of \(m\) and \(m_{t}\). In fact, considering a small variation of the stabilities induced by the algorithm update

\[\Delta_{i}^{\mu}\to\Delta_{i}^{\mu}+\delta\Delta_{i}^{\mu},\]

and performing a Taylor expansion of (14) at first order in \(O(N^{-1/2})\), one obtains (see Appendix A.1)

\[\mathcal{L}^{{}^{\prime}}=\mathcal{L}+\sum_{i=1}^{N}\delta\mathcal{L}_{i} \tag{15}\]

where

\[\delta\mathcal{L}_{i}=-\frac{\epsilon_{i}^{\mu_{d}}\lambda}{\alpha\sigma_{i}N ^{5/2}}\frac{\sqrt{2}m\cdot m_{t}}{\sqrt{\pi(1-m^{2})}}\exp{\left(-\frac{m^{2 }\Delta_{i}^{\mu_{d}^{2}}}{2(1-m^{2})}\right)}. \tag{16}\]

Hence, \(\delta\mathcal{L}_{i}\) is strictly non-positive when \(\frac{\lambda}{N}\) is small, so that the Taylor expansion is justified. Now, the couplings' variance \(\sigma_{i}\) (see eq. (7)) is a variable quantity over time, and numerics suggest it is slowly decreasing. As a result, the expansion performed to determine the variation of \(\mathcal{L}\) might not be appropriate after a certain number of steps, leading to a non-monotonic trend of the function. This inconvenience can be overcome by transforming the learning rate \(\lambda\) into \(\lambda_{i}=\lambda\cdot\sigma_{i}\) at each iteration.

Wong and Sherrington [20; 21] propose an elegant analysis of a network designed to optimize \(\mathcal{L}(m,J)\), i.e. whose couplings \(J\) correspond to the global minimum of \(\mathcal{L}(m,J)\). Some of their findings, relevant to this work, are:

1. For any \(m_{0}\), the maximum value of \(m_{1}(m_{0})\) is obtained if \(m=m_{0}\).
2. When \(m\to 1^{-}\), the minimization of the \(\mathcal{L}(m,J)\) trains a linear perceptron with \(N\) dimensional input/output layers (as in [12]) and maximal stability. This network will be referred to as a Support Vector Machine (SVM) [22].
3. When \(m\to 0^{+}\), the minimization of \(\mathcal{L}(m,J)\) leads to a Hebbian coupling matrix \(J_{ij}\propto J_{ij}^{H}\).

Numerically, we find that iterating (8) with a given value of \(m_{t}\) drives \(\mathcal{L}(m_{t},J)\) to its theoretical absolute minimum computed in [21], as reported in fig. 3 for one choice of \(N,\alpha\) and \(m\). This means that the performance of the training-with-noise algorithm can be completely described in the analytical framework of [21] and, in particular, all the above points hold. We are going to call \(\mathcal{L}(m_{t},J)\) loss function of the training-with-noise algorithm.

### Classification

The computations contained in [21], and resumed in Appendix B, are now used to calculate \(n_{SAT}\) as a function

Figure 3: The _blueish_ lines in the main plot report the function \(\mathcal{L}(m=0.5,J)\) for different training overlaps as functions of the number of algorithm steps \(d\). The _dotted_ line represents the theoretical minimum value from [21]. The learning strength \(\lambda\) has been rescaled by the standard deviation of the couplings as described in the text. The subplot reports the case \(m_{t}=0.5\) when the learning strength is not rescaled: \(\mathcal{L}\) is in _blue_, while a measure of the standard deviation of the couplings, defined as \(\sigma=\frac{1}{N}\sum\sigma_{i}\), is reported in _red_. The value \(\lambda\cdot N^{-1/2}\) is also depicted in _light gray_ to properly signal the moment when equation (16) loses its validity. All measures are averaged over 5 realizations of the couplings \(J\). Choice of the parameters: \(N=100\), \(\alpha=0.3\), \(\lambda=1\), the initial couplings are Gaussian with unitary mean, zero variance and \(J_{ii}^{(0)}=0\ \forall i\).
of \(m_{t}\) and \(\alpha\) in the training-with-noise problem.

The distribution of the stabilities in the trained network (see equation (4)) has always a tail in the negative values, except in the trivial case of \(m_{t}=1\), where \(n_{SAT}=1\) for \(\alpha\leq 2\). This implies that classification is never reached by the training-with-noise algorithm. Nevertheless, the values of \(n_{SAT}\) remain close to unity for relatively high values of \(m_{t}\) and relatively low values of \(\alpha\) (see fig. 4).

### Generalization

The generalization properties of a network trained through training-with-noise are now discussed. The color map in fig. 5 reports the estimate of the retrieval map \(m_{f}\) at \(m_{0}=1\) in the limit \(N\to\infty\). Notice the emergence of two phases from the map: the _non retrieval_ phase where \(m_{f}(1)\) is mostly smaller than \(0.5\), and memories are far from being at the center of the basin; the _retrieval_ phase where the \(m_{f}(1)\) is higher, mostly close to \(1\), i.e. the memory is very close to the center of the basin. Such separation is typical of fully connected neural networks [4], differently from sparse neural networks [21; 23] where the topology of the basins results more various yet harder to get measured by experiments. In Appendix C we propose an empirical criterion to predict the transition between these two regimes. The resulting critical line is reported in fig. 5.

Fixing ourselves to the retrieval regime we employ a procedure described in Appendix C to compute the typical size of the basins of attraction of networks trained as SVMs and with the training-with-noise algorithm. White dots in fig. 5 signal the combinations of \((m_{t},\alpha)\) where the basins of attraction found by training-with-noise algorithm resulted smaller than than the ones obtained by a SVM at the same value of \(\alpha\). We want to stress the importance of a comparison between the training-with-noise and the relative SVM, since numerical investigations have shown the latter to achieve very large basins of attraction, presumably due to the maximization of the stabilities [9; 17]. One can conclude that for most of the retrieval region the generalization performance is worse than the SVM, which maintains larger basins of attraction; on the other hand, at higher values of \(\alpha\) the trained-with-noise network sacrifices its classification property to achieve a basin that appears wider than the SVM one. In conclusion, the training-with-noise algorithm never outperforms the relative SVM without reducing its classification capabilities.

## V Training with structured noise

In this section, we study how adding additional constraints on the configurations used in the training-with-noise algorithm affects its performance. This amounts to imposing specific internal dependencies among the noise units \(\vec{\chi}\), which are no more i.i.d. random variables, as it was in [14]. When this choice is appropriate, the algorithm can lead to a classification regime and a high degree of generalization, resembling a SVM. We firstly evaluate the maximal noise case, i.e. \(m_{t}=0^{+}\) and derive a condition to be satisfied by noisy data-points to approach the best memory performance. In light of such

Figure 4: \(n_{SAT}\) as a function of \(m_{t}\) and \(\alpha\). Warmer shades of colour are associated to higher classification performances.

Figure 5: \(m_{f}(1)\) as a function of \(m_{t}\) and \(\alpha\). Warmer shades of colour are associated to higher retrieval performances. The _black dashed_ line represents the boundary of the retrieval regime according to the criterion in Appendix C, _white_ dots signal the points where basins of attraction to which memories belong are larger than ones obtained from a SVM at \(N=200\).
condition, we analyze the performance of fixed points of the dynamics as training data. Finally, we run training-with-noise with \(m_{t}>0\) using fixed points of the dynamics as training-data, after initializing the coupling according to Hebb's rule. We find that the structure of noise is relevant also in this case. It will be helpful for our purposes to implement a symmetric version of rule (8), i.e.

\[\delta J_{ij}^{(d)}=\frac{\lambda}{N}\left(\epsilon_{i}^{\mu_{d}}\xi_{i}^{\mu_{ d}}S_{j}^{\mu_{d}}+\epsilon_{j}^{\mu_{d}}\xi_{j}^{\mu_{d}}S_{i}^{\mu_{d}}\right) \tag{17}\]

Equation (17) can be rewritten explicitly making use of (9), leading to

\[\begin{split}\delta J_{ij}^{(d)}=&\frac{\lambda}{2 N}\big{(}\xi_{i}^{\mu_{d}}S_{j}^{\mu_{d}}+S_{i}^{\mu_{d}}\xi_{j}^{\mu_{d}}\big{)}+ \\ -&\frac{\lambda}{2N}\big{(}S_{i}^{1,\mu_{d}}S_{j}^{ \mu_{d}}+S_{i}^{\mu_{d}}S_{j}^{1,\mu_{d}}\big{)}\end{split} \tag{18}\]

where \(S_{i}^{1,\mu_{d}}=\text{sign}\left(\sum_{k=1}^{N}J_{ik}S_{k}^{\mu_{d}}\right)\). The total update to the coupling at time \(D\) can be decomposed as a sum of two contributions

\[\Delta J_{ij}(D)=\Delta J_{ij}^{N}(D)+\Delta J_{ij}^{U}(D). \tag{19}\]

The first term on right-hand side, which will be referred to as _noise_ contribution, is expressed in terms of noise units as

\[\Delta J_{ij}^{N}(D)=\frac{\lambda}{2N}\sum_{d=1}^{D}\xi_{i}^{\mu_{d}}\xi_{j} ^{\mu_{d}}\chi_{j}^{\mu_{d}}+\frac{\lambda}{2N}\sum_{d=1}^{D}\xi_{j}^{\mu_{d}} \xi_{i}^{\mu_{d}}\chi_{i}^{\mu_{d}}, \tag{20}\]

while the second term, which will be referred to as _unlearning_ contribution, is given by

\[\Delta J_{ij}^{U}(D)=-\frac{\lambda}{2N}\sum_{d=1}^{D}\left(S_{i}^{1,\mu_{d}}S _{j}^{\mu_{d}}+S_{i}^{\mu_{d}}S_{j}^{1,\mu_{d}}\right). \tag{21}\]

### \(m_{t}=0^{+}\)

Training configurations with maximal noise are now studied. Firstly, we apply eq. (19) to compute the variation of \(\mathcal{L}(m,J)\) during training, showing what are the properties of the training configurations that lead to a good network performance. Then we numerically probe two types of attractor landscapes to study the distribution of the good training data. Eventually, we consider fixed points of the dynamics (1) as training configurations, showing a connection between the training-with-noise and unlearning algorithms.

#### The optimal structure of noise

We now want to study what kind of noise structure is favorable to achieving classification and a good degree of generalization. Section (IV) has shown that the training-with-noise algorithm on fully connected networks never outperforms a SVM without reducing their classification properties. In addition to this, we know from numerics [9; 17] that SVMs, as maximally stable perceptrons, maximize the size of the basins of attraction of the memories as they are also attractors of the dynamics. We hence consider the SVMs as optimal recurrent networks in both terms of classification and generalization. As a consequence, we just need to approach the global minimum of \(\mathcal{L}(m=1^{-},J)\) to train a well performing neural network. During the rest of this work we will apply an important differentiation between the two variables \(m\) and \(m_{t}\): while the former controls the final result of the optimization procedure, the latter is a fixed level of noise, \(m_{t}=0^{+}\) in the current case. According to the training-with-noise algorithm, at \(m_{t}=0^{+}\) the variation of \(\mathcal{L}(m,J)\) is \(\delta\mathcal{L}=\delta\mathcal{L}_{N}+\delta\mathcal{L}_{U}\), where \(\delta\mathcal{L}_{N}\) cancels in time, and the only relevant contribution is

\[\delta\mathcal{L}_{U}\propto\frac{m}{\sqrt{2(1-m^{2})}}\sum_{i,\mu}^{N,p} \omega_{i}^{\mu}\exp{\left(-\frac{m^{2}\Delta_{i}^{\mu^{2}}}{2(1-m^{2})} \right)}, \tag{22}\]

where

\[\omega_{i}^{\mu}=\frac{1}{2\sigma_{i}}\left(m_{\mu}\chi_{i}^{1,\mu}+m_{1,\mu} \chi_{i}^{\mu}\right), \tag{23}\]

with

\[\chi_{i}^{\mu}=\xi_{i}^{\mu}S_{i}^{\mu_{d}}\hskip 28.452756pt\chi_{i}^{1,\mu}=\xi _{i}^{\mu}S_{i}^{1,\mu_{d}}, \tag{24}\]

and

\[m_{\mu}=\frac{1}{N}\sum_{j=1}^{N}S_{j}^{\mu_{d}}\xi_{j}^{\mu}\hskip 28.452756ptm_{1, \mu}=\frac{1}{N}\sum_{j=1}^{N}S_{j}^{1,\mu_{d}}\xi_{j}^{\mu}. \tag{25}\]

The derivation of (22) is reported in Appendix A.2. For the case of fixed points, we have \(\omega_{i}^{\mu}=m_{\mu}\chi_{i}^{\mu}\), coherently with the same observable defined in [9]. Generally speaking, \(\omega_{i}^{\mu}\) can be a function of \(\Delta_{i}^{\mu}\). Since stabilities are self-averaging quantities, by taking \(N,p\gg 1\) we can rewrite equation (22) as

\[\delta\mathcal{L}_{U}\propto\frac{m}{\sqrt{2\pi(1-m^{2})}}\int_{-\infty}^{+ \infty}d\Delta\rho(\Delta)\omega(\Delta)e^{-\frac{m^{2}\Delta^{2}}{2(1-m^{2})}}, \tag{26}\]

where \(\rho(\Delta)\) is the true probability density function of \(\Delta\) at a given step of the algorithm. When \(m\to 1^{-}\) the Gaussian contained in the integral becomes very peaked around \(0\). Since \(\rho\) is a strictly positive function, and we want \(\delta\mathcal{L}\) to decrease, we need

\[\omega(\Delta)<0\hskip 5.690551pt\text{when}\hskip 5.690551pt|\Delta|<\epsilon, \hskip 5.690551pt\epsilon=0^{+} \tag{27}\]

Hence, a coupling update improves the memory performances when condition (27) is satisfied. As a consequence, noisy units are internally constrained by (27), as a difference with the standard training-with-noise algorithm. The more negative \(\omega\) is when \(\Delta\sim 0\) the more powerful a given data-point is at approaching the SVM performances. One should also bear in mind that training is a dynamic process: to reduce \(\mathcal{L}(m=1^{-},J)\) condition (27) should hold during training.
#### About the initialization

We are now interested in studying how the initialization of the couplings influences the choice of training configurations, concentrating in particular on the function \(\omega(\Delta)\). Computing \(\omega\) analytically is a challenging task, but we will introduce some related empirical quantities, which can be useful to probe the structure of noise in the configurations. To do so, we first sample training configurations from a network initialized with the Hebbian rule according to a Monte Carlo dynamics at temperature \(T\). Temperature acts as a control parameter: when \(T=0\) training configurations are stable fixed points of eq. (1), as in standard unlearning. Higher values of \(T\) progressively reduce the structure of noise in training configurations, and in the limit \(T\to\infty\), training configurations are the same as in the training-with-noise algorithm. The Monte Carlo of our choice is of the Kawasaki kind [24], to ensure that all training configurations are at the prescribed overlap \(m_{t}=0^{+}\). Fig. 6, presents the result at four different temperatures. Each panel shows \(\omega\) as a function of \(\Delta\). Data points are collected over 15 realizations of the network, then plotted and smoothed to create a density map. We are interested in the _typical_ behavior of \(\omega(\Delta)\) in the neighborhood of \(\Delta=0\), which can be estimated by a linear fit of the data. We consider the intercept of the best fit line as an _indicator_\(\omega_{emp}(0)\) of the mean value of \(\omega(0)\) over the noise.

We find that the lower the temperature, the more sampled configurations favor both classification and generalization: the line is negative over a larger interval centered around \(\Delta=0\). As temperature becomes too high, \(\omega_{emp}(0)\) gets closer to zero, suggesting low quality in terms of training performance.

Another interesting aspect emerged from the analysis is that, by calling \(a\) the line intercept and \(b\) its angular coefficient, the position of the right extremum of the negative band \(\frac{|a|}{b}\) is independent of the system size \(N\), implying that finite size effects are not strong for this kind of measure. This is likely because both the mean of the distribution of the points along \(\omega\) and their dispersion, i.e. how far they span the y-axis, are decreasing as \(O(N^{-1/2})\).

The same analysis is repeated in the case of a random initialization. We chose the Sherrington-Kirkpatrick (SK) model [25] as a case of study. Panels (a) and (b) in fig. 7 report the smoothed distribution of \(\omega\) versus \(\Delta\) showing a different scenario with respect to the Hebbian one. The distribution looks anisotropic, as in the Hebbian case, yet the stabilities are centered Gaussians, so \(\omega_{emp}(0)\) is positive. In particular, things appear to improve when \(T\) increases, in contrast with the previous case of study, though in accordance with the Hebbian limit of the training-with-noise algorithm explained in Section IV. Panel (c) displays more clearly the mutual dependence between \(\omega\) and stabilities \(\Delta\) by reporting the Pearson coefficient at the various \(T\) between these two quantities: both Hebb's and SK show some mutual dependence, but the distribution in the Hebbian landscape is way more deformed. Furthermore, panel (d) shows \(\omega_{emp}(0)\) in both cases. This quantity is very well correlated with the Pearson coefficient: whereas in the Hebb's initialization \(\omega_{emp}(0)\) remains negative and reaches the lowest values at low temperatures, the random case shows an opposite trend, where the estimated \(\omega_{emp}(0)\) is rather positive. Notice, from both panels (c), (d), the existence of an optimum which does not coincide with the stable fixed points of the dynamics (i.e. \(T=0\)): this aspect will be deepened in Sec. VI.

#### Training with stable fixed points

Let us now consider the instance of a set of noisy states that are also fixed points of the dynamics with a training overlap \(m_{t}=0^{+}\). In this case, one has

\[\Delta J^{N}_{ij}(D)=0^{+}+O\left(\sqrt{\frac{\lambda}{N}}\right). \tag{28}\]

On the other hand, the unlearning contribution to the evolution of the couplings is

\[\Delta J^{U}_{ij}(D)=-\frac{\lambda}{N}\sum_{d=1}^{T}S^{\mu_{d}}_{i}S^{\mu_{d} }_{j}, \tag{29}\]

which is the classic unlearning updating rule. As a result, when \(\lambda/N\to 0\) the training-with-noise algorithm and the unlearning algorithm will converge to the same updating rule for the couplings when stable fixed points of the dynamics are used in the training. The same argument can be applied to the original asymmetric rule (8), however asymmetric networks may have no stable fixed points of (1) that can be learned.

We now perform a numerical test of the argument above, in the case of a symmetric connectivity matrix. At each step of the algorithm, the network is initialized with an initial overlap contained in \((0,N^{-1/2})\) with one memory \(\vec{\xi}^{\mu_{d}}\). Then, asynchronous dynamics (1) is run until convergence, and the final overlap \(m_{t}\) is measured. If \(m_{t}\in(0,N^{-1/2})\), we use the sampled configuration for training, otherwise the process is repeated. Typically, an initial overlap equal to \(0^{+}\) implies a similar order of magnitude for the final overlap, hence no reiteration is needed.

The algorithm (17) is repeated for \(D^{*}=O(N/\lambda)\) steps. The order of magnitude of \(\Delta J^{U}_{ij}(D)\) is supposed to be the same of \(J^{(0)}_{ij}\), in order to see significant modifications to the initial connectivity matrix. The network is initialized according to the Hebb's rule (2), i.e. \(J^{(0)}_{ij}=O(N^{-1/2})\) which implies \(\Delta J^{U}_{ij}(D)=O(N^{-1/2})\) at leading order. The contributions U and N are compared by computing the norm of the relative \(\Delta J\) matrix and evaluating the ratio \(|\Delta J^{U}|/|\Delta J^{N}|\). From our previous considerations we expect \(|\Delta J^{U}|/|\Delta J^{N}|\) to be linear in \(\lambda^{-1/2}\) when corrections vanish. Results are reported in fig. 8:
\(|\Delta J^{U}|/|\Delta J^{N}|\) grows when \(N\) increases and \(\lambda\) decreases, according to the scaling relation predicted by our argument. In addition to this, curves are collapsing on the expected line when \(\lambda\to 0\) and \(N\to\infty\).

We also measured \(\Delta_{\min}\) at its maximum over the course of the algorithm (as described in IV.1). Results are reported in fig. 9. \(\Delta_{\min}\) produced by training-with-noise and unlearning are found to coincide when \(\lambda\) is sufficiently low. Moreover, the number of steps necessary to reach the maximum are the same for both the algorithms, confirming that couplings are transforming in the same way. This last aspect is corroborated by the subplot in fig. 9, representing the set of \(J_{ij}\) obtained with the traditional unlearning algorithm as a function of the one resulting from the training-with-noise algorithm, for one realization of the network. The strong correlation is evident, as predicted from the our pseudo-analytical arguments.

One can also study how \(\omega(\Delta)\) evolves during the training process. Fig. 9(a) shows the value of \(\omega_{emp}(0)\) at different time steps of the training process, for different values of \(\alpha\). The colored halo around the line of data points represents the width of the band centered in \(0\) where the line of the best fit assumes negative values.

Figure 6: Distribution of \(\omega_{i}^{\mu}\) as a function of \(\Delta_{i}^{\mu}\) for training configurations sampled with a Monte Carlo at temperature \(T=0\) i.e. stable fixed points only (a), \(T=0.5\) (b), \(T=1\) (c), \(T=8\) (d), on a Hebbian network. Warmer colors represent denser region of data points. The _full black_ line is the non-weighted best fit line for the points, the _dotted white_ line represents \(\omega=0\), the _red dot_ is the value of the best fit line associated with \(\Delta=0\). Sub-panels to each panel report a zoom of the line around \(\Delta=0\): the _reddish_ region give a measure of the largeness of the negative band centered around \(\Delta=0\). Measures have been collected over \(15\) samples of the network. Choice of the parameters: \(N=500\), \(\alpha=0.5\).

Figure 7: (a), (b): Distribution of \(\omega_{i}^{\mu}\) as a function of \(\Delta_{i}^{\mu}\) for training configurations sampled with a Monte Carlo at temperature \(T=0\) i.e. stable fixed points only (a), and \(T=8\) (b) on a SK model. Warmer colors represent denser region of data points. The _full black_ line is the non-weighted best fit line for the points, the _dotted white_ line represents \(\omega=0\), the _red spot_ is the value of the best fit line associated with \(\Delta=0\). Sub-panels to each panel report a zoom of the line around \(\Delta=0\). (c), (d): Comparison between the Hebbian initialization and the Random one through evaluation of: the Pearson coefficient between \(\omega_{i}^{\mu}\) and \(\Delta_{i}^{\mu}\) (c) and the estimated value of \(\omega_{emp}(0)\) from the dispersion plots (d). Measures have been collected over \(15\) samples of the network. Choice of the parameters: \(N=500\), \(\alpha=0.5\).
We find that \(\omega_{emp}(0)<0\) for \(\alpha\leq 0.6\). The progressive increase of \(\omega_{emp}(0)\) means that the structure of the fixed points is more effective in the starting Hebbian landscape compared to intermediate stages of training. In the last part of training, points reacquire more negative values, but this is not a reliable indication of good performance: as shown in fig. 10c, in this part of the process the standard deviation of the couplings \(\sigma_{i}\) is comparable to \(O(\lambda\cdot N^{-1/2})\), and the expansion of the \(\mathcal{L}\) in eq. (22) is not valid. The last part of the training, where \(\sigma_{i}\simeq 0\ \forall i\), has been neglected from the plot. On the other hand, the width of the colored halos decreases with \(\alpha\). The experiment is thus consistent with the characterization of the unlearning algorithm presented in [9], which showed decreasing classification and generalization when increasing \(\alpha\). This is confirmed by the study of the Pearson correlation coefficient between \(\omega_{i}^{\mu}\) and the associated stabilities \(\Delta_{i}^{\mu}\) (see Fig. 10,b). High values of the Pearson coefficient show a strong dependence of the structure of noise on the relative stabilities. For all \(\alpha\), the Pearson coefficient is highest at \(d=0\), and progressively decreases during training, suggesting that the quality of the training configurations is deteriorating. The final increase in the coefficient is, again, due to the vanishing of the standard deviations \(\sigma_{i}\) of the couplings, and does not indicate good performance.

### \(m_{t}>0\)

When training configurations are fixed points of the dynamics with \(m_{t}=O(1)\), the considerations of the previous section no longer apply. Such configurations can be generated by initializing the network at an overlap \(m=O(1)\) with a memory, and let it evolve according to (1). The connectivity matrix is initialized according to Hebb's learning rule eq. (2). In this setting, if at some point during training \(m\) happens to enter the basin of attraction of the memories, the fixed point reached by the dynamics will be the memory itself, and the noise contribution will cancel exactly the learning contribution, giving \(\delta J=0\). On the other hand, for sufficiently high values of the load \(\alpha\), at the start of the training procedure memories will have zero size basins of attraction and trajectories will drift away from the memories following the dynamics. In this scenario, the two contributions \(\delta J^{N}\) and \(\delta J^{U}\) decorrelate, and \(\delta J^{N}\) will take again a similar role to what described in the previous section. The result is an algorithm which interpolates between unlearning when \(d\) is small and a supervised algorithm when basins increase to a size close to \((1-m)\). In this regime, \(\delta J^{N}\) acts as a breaking term, preventing the algorithm to further modify the coupling matrix \(J\). A similar mechanism has been studied in [26] where a supervised term was added to the standard unlearning update rule, leading to \(\delta J_{ij}\propto-S_{i}^{\mu_{d}}\delta_{j}^{\mu_{d}}+\xi_{i}^{\mu_{d}}\xi_ {j}^{\mu_{d}}\). Notice that the term

Figure 8: Estimates of the ratio \(|\Delta J^{U}|/|\Delta J^{N}|\) as a function of \(\lambda^{-\frac{1}{2}}\) and \(N\) for \(\alpha=0.5\). Measures are averaged over 5 samples. Error bars are not indicated because smaller than the symbols.

Figure 9: The quantity \(\max_{d}\left(\Delta_{\min}\right)\) as a function of \(\lambda^{-1/2}\). Colors are: _red_ for \(N=100\) and _blue_ for \(N=500\). The _gray_ line represents the null value for the stability. Symbols are: _circles_ for the training-with-noise rule, _triangles_ for the unlearning rule. In the subplot on the center right, the couplings obtained through the unlearning algorithm are plotted as a function of the ones resulting from the training-with-noise, at the same amount of iterations, for one sample at \(N=500\) and \(\lambda=5\cdot 10^{-3}\). Measures are averaged over 50 samples. The choice of the parameters is: \(\alpha=0.5\), \(m_{t}=0^{+}\).
\(\xi_{i}^{\mu_{d}}\xi_{j}^{\mu_{d}}\) is deterministically reproducing Hebb's learning rule, while in our study the unlearning is modified by a stochastic term, whose nature we have already discussed. Given a sufficiently small learning rate \(\lambda\), there will exist a characteristic number of steps of the algorithm over which the coupling matrix does not change significantly. We will call this timescale _an epoch_. Averaging the effect of training steps over an epoch, we get a snapshot of how the algorithm is affecting the couplings at a given point during training. This can be used to study the relation between \(\delta J^{N}\) and \(\delta J^{U}\), quantified by the connected correlation coefficient

\[\text{Cov}_{\text{N-U}}:=\frac{2}{N(N-1)}\sum_{i,j>i}\big{(}\overline{\delta J _{ij}^{N}\delta J_{ij}^{U}}-\overline{\delta J_{ij}^{N}}\,\overline{\delta J_{ ij}^{U}}\big{)}, \tag{30}\]

where the average is computed over an epoch. When this quantity equals one, there is no effective update of the couplings over an epoch. Results are presented in fig. 11 for different values of \(m\) and of \(\alpha\), as a function of the number of training steps \(d\). The number of iterations has been rescaled by a factor \(p/\lambda\) for clarity of the plot. At any given \(\alpha\), training with higher \(m\) results in a faster increase of \(\text{Cov}_{\text{N-U}}\), i.e. a faster convergence of the algorithm. If the value of \(m\) is too low, the algorithm never manages to build a big enough basin of attraction, and never stops. As \(\alpha\) is increases, higher and higher values of \(m\) are required for the algorithm to stop, since the typical size of the attraction basins shrinks.

The network performance can be benchmarked by tracking the evolution of the lowest stability \(\Delta_{\text{min}}\) throughout the training procedure. Results are presented in fig. 12 for different values of \(\alpha\) and \(m\). For sufficiently low values of \(\alpha\) and sufficiently high values of \(m\), \(\Delta_{\text{min}}\) surpasses the zero. Once this condition is met, the value \(\Delta_{\text{min}}\) becomes essentially constant, even if \(\text{Cov}_{\text{N-U}}<1\), signaling that the update of the coupling matrix is still in progress. The result is a curve \(\Delta_{\text{min}}(d)\) which barely surpasses zero. For each value of \(m\) there exists a critical

Figure 11: Correlation between _noise_ and _unlearning_ contributions to \(\delta J\) as a function of the rescaled number of training steps \(\frac{d\lambda}{p}\), for two values of the training parameter \(m\), and different values of \(\alpha\). When \(\text{Cov}_{\text{N-U}}=1\), the algorithm stops modifying the coupling matrix. Choice of the parameters: \(N=400\), \(\lambda=10^{-2}\). Results are averaged over 100 samples.

Figure 10: The training-with-noise algorithm is implemented by sampling stable fixed points of the network dynamics with \(m_{t}=0^{+}\). (a) The empirical measure of \(\omega\) for \(\Delta=0\) for the stable fixed points as a function of the rescaled number of iterations of the training-with-structured-noise algorithm: the bars represent the symmetric negative band obtained from the linear fit in fig. 6. (b) Pearson coefficient measured between \(\omega_{i}^{\mu}\) and \(\Delta_{i}^{\mu}\). (c) The standard deviation of the couplings during learning, defined as \(\sigma=\frac{1}{N}\sum\sigma_{i}\). Points are averaged over 50 samples and the choice of the parameters is: \(N=100\), \(\lambda=10^{-2}\).
value \(\alpha_{c}(m)\) beyond which no amount of steps is able to produce \(\Delta_{\min}>0\). Extrapolating empirical results to the \(N\to\infty\) limit, one finds

\[\alpha_{c}(m)=A\cdot(m)^{B}+C,\]

where

\[A=0.35\pm 0.01,\quad B=6.9\pm 0.5,\quad C=0.58\pm 0.01\]

Consistently with what presented in the previous section, in the limit \(m\to 0^{+}\) one finds the critical capacity of the unlearning algorithm [7; 9]. The critical capacity increases up to a value \(\alpha_{c}(1)=0.93\pm 0.01\) when \(m\) reaches unity.

Regardless of whether \(\Delta_{\min}>0\) is obeyed, one can monitor the network performance as an associative memory device by measuring the retrieval map \(m_{f}(m_{0})\). We find that the best performance always corresponds to the number of training step maximizing \(\Delta_{\min}\), hence the curves \(m_{f}(m_{0})\) are all relative to this number of steps. Results are presented in fig. 13. When classification is achieved (i.e. \(\alpha<\alpha_{c}(m)\)), lower values of \(m\) increase the degree of generalization of the network (i.e. enlarge the basins of attraction), at the cost of a lower critical capacity.

## VI A sampling procedure for noisy data-points

Insights from the previous sections on the structure of well performing training data can be used to effectively sample configurations to teach to the model. This can be achieved by means of a supervised Monte Carlo routine that searches for maximally noisy configurations (i.e. \(m_{t}=0^{+}\)) satisfying condition (27). The coupling matrix is initialized according to Hebb's rule (2), and updated recursively according to either eq. (8) or eq. (5). We first introduce the sampling algorithm and then report some numerical results regarding both the training-with-noise and unlearning routines.

### The sampling algorithm

We sample maximally noisy training configurations \(m_{t}=0^{+}\) such that \(E(\vec{\chi}|m,J)<0\), where

\[E(\vec{\chi}|m,J):=\frac{m}{\sqrt{2(1-m^{2})}}\sum_{i,\mu}^{N,p}\omega_{i}^{ \mu}\exp\left(-\frac{m^{2}\Delta_{i}^{\mu^{2}}}{2(1-m^{2})}\right)\!, \tag{31}\]

that is a function of the noisy variables \(\chi_{i}^{\mu}\), conditioned on a reference overlap \(m\) and the couplings \(J\). Sampling is done through the following procedure:

1. The network is initialized in a random configuration and the asynchronous dynamics in eq. (1) is run until convergence on a fixed point. The final state \(\vec{S}^{\mu_{d}}\) must have an overlap \(m_{t}\) in the interval \((0,1/\sqrt{N})\) with one memory \(\mu_{d}\), otherwise the procedure is repeated.
2. A \(T=0\) temperature dynamics in the landscape of \(E(\vec{\chi}|m,J)\) is performed until \(E(\vec{\chi}|m,J)<0\). We

Figure 12: Minimum stability as a function of the rescaled number of training steps \(\frac{d\lambda}{p}\), for two values of the training parameter \(m\), and different values of \(\alpha\). When \(\Delta_{\min}\geq 0\), each memory is a stable fixed point of the dynamics. Choice of the parameters: \(N=400\), \(\lambda=10^{-2}\). Results are averaged over 100 samples.

Figure 13: Retrieval map \(m_{f}(m_{0})\) for two values of \(\alpha\) and different values of the training parameter \(m\). At \(\alpha=0.4\), every \(m\) leads to stable memories, i.e. \(m_{f}(1)=1\). At \(\alpha=0.8\), only the highest values of \(m\) lead to stable memories, while for low values of \(m\) one has \(m_{f}(1)<1\). Choice of the parameters: \(N=400\), \(\lambda=10^{-2}\). Results are averaged over 100 samples.
use a Kawasaki kind of dynamics over the noisy variables \(\vec{\chi}\) to make sure that \(m_{t}\) maintains the prescribed value.

Since \(E(\vec{\chi}|m,J)\) is proportional to \(\delta\mathcal{L}_{U}\) (see eq. (22)), the procedure will lead to a reduction in the Loss eq. (14). In this setting, the classification and generalization properties can be tuned by the parameter \(m\), while the training configurations always have a fixed value of \(m_{t}=0^{+}\). In particular, to require a performance that is most similar to the one of a SVM, we will set \(m\to 1^{-}\).

The sampling procedure starts from fixed points because we know, from the previous sections, that they are close to be the most effective configurations. Interestingly, the described procedure results significantly more effective than a standard minimization of \(\mathcal{L}(m=1^{-},J)\): in the latter case training stops when \(\mathcal{L}=-1\), while the former technique apparently pushes the stabilities further in the positive values. Nevertheless, the goal of this section is not to propose a winning, or more computationally efficient training routine, but rather to search for the most virtuous training data points in the context of maximal training noise, and reveal their identity.

### Algorithm performance

The sampling procedure results in a better performance for both training-with-noise and the unlearning update rules.

Results for training-with-noise are reported in fig. 14. Panel (a) shows that classification is reached up to \(\alpha\simeq 0.8\) for a network of size \(N=100\). Panel (b) shows the retrieval map \(m_{f}(m_{0})\), for different values of \(\alpha\) and for the lowest number of algorithm iterations leading to classification. The high values of \(m_{f}\) for \(m_{0}\sim 1\) indicates that the network, while achieving classification, maintains good generalization properties. Results for a SVM with the same control parameters are also shown, for comparison. The curves obtained from the training-with-noise are higher than ones relative to the SVM, signaling a better recalling performance, even though quantifying this effect will require a more detailed study of finite size effects. Fig. 15 shows analogous plots for the unlearning update rule. Again, the network compares favorably with traditional unlearning in terms of memory capacity, that is increased up to \(\alpha\simeq 0.9\) (see panel (a)) with respect to the maximum capacity of \(\alpha\simeq 0.7\) obtained with fixed points only. Panel (b) represents an indication for the sizes of the basins of attraction at the lowest number of algorithm iterations leading to classification. Results are coherent with fig. 1 concerning \(\alpha=0.35\), yet again basins appear to be larger than the ones from a SVM trained with the same control parameters.
different \(f\) assume similar volumes of the basins of attraction when they are measured at the very first instant they reach classification. The plot also shows that the generalization performance is comparable with the one of a SVM trained with the same choice of the control parameters, consistently with the results of [9] in the matter of the unlearning algorithm.

Figure 14: Performance of the training-with-noise algorithm taught with noisy configurations sampled according to Section VI regarding four progressing values of the load \(\alpha\). (a) Minimum stability as a function of the algorithm time: _full_ line is the unlearning with sampling, _dotted_ line is the traditional unlearning. (b) Retrieval map \(m_{f}(m_{0})\), relatively to the _circles_ in panel (a) for \(\alpha\in[0.35,0.7,0.8]\): _full_ line is the unlearning with sampling, _dashed_ line is a SVM trained with no symmetry constraints with the same control parameters. (c) Saddle index \(f\) as a function of the algorithm steps for \(\alpha\in[0.35,0.7,0.8]\). The sub-panel zooms over \(\alpha=0.35\) alone. All data points have been averaged over \(20\) samples in (a),(b) and \(5\) samples in (c). Errors are neglected for clarity of the image. The choice of the parameters: \(N=100\), \(\lambda=10^{-3}\), \(m=0.9999\).

Figure 15: Performance of the unlearning algorithm trained with noisy configurations sampled according to Section VI regarding four progressing values of the load \(\alpha\). (a) Minimum stability as a function of the algorithm time: _full_ line is the unlearning with sampling, _dotted_ line is the traditional unlearning. (b) Retrieval overlap \(m_{f}(m_{0})\), relatively to the _circles_ in panel (a) for \(\alpha\in[0.7,0.8,0.9]\): _full_ line is the unlearning with sampling, _dashed_ line is a SVM trained with no symmetry constraints with the same control parameters. (c) Saddle index \(f\) as a function of the algorithm steps for \(\alpha\in[0.7,0.8,0.9]\). All data points have been averaged over \(20\) samples in (a),(b) and \(5\) samples in (c). Errors are neglected for clarity of the image. The choice of the parameters: \(N=100\), \(\lambda=10^{-3}\), \(m=0.9999\).
## VII Conclusions

The seminal work of [14] showed that noise could be injected during training to increase generalization, i.e. the capability of neural networks to retrieve classes of data by receiving corrupted stimuli [29; 30; 31; 32]. The same concept recently seems to catch on in biology as well [33; 34], as artificial neural networks appear more frequently to be involved in explaining neuroscience, and viceversa.

In Section IV, we show that the training-with-noise algorithm [14] reproduces the thermodynamics described by Wong and Sherrington in [20; 21]. This implies convergence to a Hebbian matrix [2; 4] or a Support Vector Machine (SVM) [11; 22; 35] when learning random configurations with, respectively, a maximal (i.e. \(m_{t}=0^{+}\)) and a minimal (i.e. \(m_{t}=1^{-}\)) amount of noise. The training overlap \(m_{t}\in(0,1)\) can be then tuned to interpolate between these two models.

In Section V, we exploit a novel degree of freedom to access a new path in coupling space. After setting a specific amount of training noise \(m_{t}\), we constrain the entries of the training configurations to assume a particular internal _structure_. In the case of maximal noise \(m_{t}=0^{+}\), for every training configuration, we define a set of variables \(\omega_{i}^{\mu}\), whose value in the neighborhood of \(\Delta_{i}^{\mu}=0\) allows to infer the classification and generalization power of the training configuration. When this quantity is negative, training drives the network closer to the optimal properties of a SVM.

In this framework, the shape of the initial landscape of attractors where configurations are sampled assumes a crucial role. We find that the Hebbian landscape in the _oblivion_ regime (i.e. \(\alpha>0.14\)) is a very effective starting condition for learning. Furthermore, our analysis shows that the quality of the configurations, in terms of noise structure, increases when we descend towards the metastable minima of a Hebbian landscape, while the inverse trend is observed in a fully random landscape. In order to reach lower states in the Lyapunov function, we focused on a symmetric version of the training-with-noise algorithm [14]. In particular, we proved that the traditional _unlearning_ algorithm [5; 7; 8; 9] (see Section III) is recovered from the training-with-noise algorithm (see Section IV) when the noise is maximal and the training configurations are stable fixed points of the dynamics. Therefore, it is now clear that basins of attraction are close to be maximized by the unlearning algorithm [9] because training data are such as to approach the global minimum of the loss function of a SVM. The same principle is traditionally applied by other unsupervised procedures that are similar to unlearning [36; 37; 10]. As a difference with the latter, such algorithms learn configurations that are higher in the landscape of attractors. Consistently with our study, these techniques achieve small or nonexistent basins of attraction, while the traditional unlearning routine, that makes use of the metastable states, approaches an optimal memory performance [9].

The picture about maximally noisy training data is completed by Section VI, where the measure given by the \(\omega_{i}^{\mu}\) variables is used to sample efficient training data. Simulations suggest that low saddles favor higher performances even more than stable fixed points do. This result can be exploited to outperform the traditional unlearning algorithm both in terms of critical capacity and basins of attraction.

In addition to the maximal noise scenario, we studied an application of the training-with-noise algorithm when stable fixed points with \(m_{t}>0\) are learned by the network, when it is initialized according to the Hebb's rule. The resulting algorithm reaches both classification and a good degree of generalization, proving that the internal structure of the configurations plays a relevant role also for higher values of \(m_{t}\). The critical capacity reached by the algorithm has been numerically estimated.

In light of our work, the full learning and memory consolidation process can be interpreted in terms of a sequential training-with-noise routine in conditions of maximal noise. In the following, we conjecture a new interpretation of learning as it would naturally come from our analysis. During a first _online_ phase external stimuli are learned by the network according to the standard training-with-noise algorithm: such stimuli might be conceived to have a very weak overlap with a set of _concepts_

Figure 16: Minimum stability \(\Delta_{\min}\) as a function of the algorithm steps on a network trained with the symmetric training-with-noise routine that learns saddles of various indices \(f\). The initial matrix is assembled according to the Hebb’s rule. Full dots report the amount of iterations needed to accomplish classification. The subplot depicts the relation \(m_{f}(m_{0})\) as measured on the positions of the dots. A comparison with a SVM trained with the same choice of the parameters is presented. All measures are averaged over 5 samples with the shaded region indicating the experimental errors. The choice of the parameters is: \(N=100\), \(\alpha=0.35\), \(\lambda=10^{-3}\).
or _archetypes_ that should be pseudo-orthogonal with each other and unknowable a-priori. The archetypes are contained in the external environment, i.e. they are hidden in the symmetries and structures of the stimuli. At the end of this phase the initial network, that had been initialized at random, or in the form of a _tabula rasa_, has achieved a Hebbian appearance hiding the unconscious archetypes. The Hebbian network might now be in the non-retrieval regime, hence without any practical utility. A second _offline_ phase follows to the first one. Now the network samples structured noisy neural configurations, still weakly correlated with the archetypes, from the freshly formed landscape of attractors. Such states could be, for instance, lower saddles or stable fixed points of the neural dynamics. Consequently, memory is consolidated by centering the unconscious archetypes into very large basins of attraction. From the biological point of view, this picture supports the importance of both daily and night experience in learning, as hebbian learning acquires higher credibility [38; 39] and the role of sleep in consolidating memory has been repeatedly confirmed by electrophysiology [40; 41]. Moreover, the association between the stored memories and pseudo-orthogonal archetypes may present interesting connections with the efforts of early psychopathology to interpret the content of dreams in humans [42]. Most importantly, it should be noticed that working with maximal noise allows the learning process to go unsupervised, as it is meant to work in real neural networks. Furthermore, learning as composed by two alternating phases is also the object of very recent speculations at the interface between neuroscience and computer science: in analogy with an adversarial network [43; 44], or a Boltzmann Machine [45; 46], the brain would use internal noisy representations to increase the robustness of learning and help generalization. We hope that our results can shed light over the particular structure of noise that is optimal for learning in neural networks, possibly helping to develop a proper theory behind the very empirical techniques of noise injection (i.e. dropout or data augmentation) that are already largely implemented in training deep networks [47; 29; 48] and which can be integrated into a wider concept of memory formation in the brain [49].

###### Acknowledgements.

 The authors are particularly grateful to their mentors Enzo Marinari, Giancarlo Ruocco and Francesco Zamponi for the precious suggestions and support. They also thank Fabian Aguirre Lopez, Aldo Battista, Simona Cocco, Giampaolo Folena, Remi Monasson and Mauro Pastore for useful discussions.
