About the de Almeida-Thouless line in neural networks
In this work we present a rigorous and straightforward method to detect the
onset of the instability of replica-symmetric theories in information
processing systems, which does not require a full replica analysis as in the
method originally proposed by de Almeida and Thouless for spin glasses. The
method is based on an expansion of the free-energy obtained within one-step of
replica symmetry breaking (RSB) around the RS value. As such, it requires
solely continuity and differentiability of the free-energy and it is robust to
be applied broadly to systems with quenched disorder. We apply the method to
the Hopfield model and to neural networks with multi-node Hebbian interactions,
as case studies. In the appendices we test the method on the
Sherrington-Kirkpatrick and the Ising P-spin models, recovering the AT lines
known in the literature for these models, as a special limit, which corresponds
to assuming that the transition from the RS to the RSB phase can be obtained by
varying continuously the order parameters. Our method provides a generalization
of the AT approach, which does not rely on this limit and can be applied to
systems with discontinuous phase transitions, as we show explicitly for the
spherical P-spin model, recovering the known RS instability line.
Linda Albanese, Andrea Alessandrelli, Adriano Barra, Alessia Annibale
# About the de Almeida-Thouless line in neural networks

###### Abstract

In this work we present a rigorous and straightforward method to detect the onset of the instability of replica-symmetric theories in information processing systems, which does not require a full replica analysis as in the method originally proposed by Almeida-Thouless for spin glasses. The method is based on an expansion of the free-energy obtained within one-step of the replica symmetry breaking (RSB) scheme around the replica-symmetric (RS) value. As such, it requires solely continuity and differentiability of the free-energy and it is robust to be applied broadly to systems with quenched disorder. We apply the method to the Hopfield model and to neural networks with multi-node Hebbian interactions, as case studies. In the appendix we test the method on the Sherrington-Kirkpatrick, the Ising \(P\)-spin and the spherical \(P\)-spin models, recovering the AT lines known in the literature for these models, as a special limit, which corresponds to assuming that the transition from the RS to the RSB phase can be obtained by varying continuously the order parameters. Therefore, our method provides a generalization of the AT approach, which can be applied to systems with a discontinuous transition.
###### Contents

* 1 Introduction
* 2 The Hopfield model
* 3 Hebbian networks with \(P\)-node interactions
* 4 Discussion
* A Applications to Spin-glasses models
* A.1 Sherrington-Kirkpatrick Model
* A.2 The Ising P-spin model
* A.3 P-spin spherical model
* B Contributions to sub-leading orders

## 1 Introduction

Replica symmetry breaking in neural networks has attracted increasing attention in recent years [4; 6; 7; 22; 32], however there is, as yet, no general broken replica-symmetry theory for these systems and no simple method to systematically detect their transition from the replica-symmetric (RS) to the replica-symmetry-broken (RSB) phase.

While the first point is still out of reach, the second question can be addressed by adapting approaches originally developed for spin glasses. Indeed, the instability line of the RS phase in the Sherrington-Kirkpatrick (SK) spin glass model was derived by Almeida and Thouless (AT) many decades ago [17], using a method based on replicas. Since their seminal work, rigorous techniques have been developed and tested in archetypical mean-field as well as short-ranged spin glass models, by many researchers (see e.g. [8; 11; 12; 21; 23; 25; 29; 30]).

As neural networks are particular realizations of spin glasses, it is quite natural to ask if we can devise a systematic method to derive the RS instability line also for these systems. In this work we answer affirmatively to this question, using the Hopfield model of neural networks and a model of dense associative memory, which extends Hebbian learning to multi-node interactions, as case studies. To this purpose, we devise a method inspired by the approach proposed by Toninelli in [31], which builds on Guerra's work on broken replica-symmetry bounds [20]. As a technical note we remark that at difference with conventional spin glasses, here we focus on the RS instability in the parameter space \((\alpha,T)\) where \(\alpha\) is the storage load of the network and \(T\) is the noise level, rather than in the space \((h,T)\) (i.e. magnetic field, temperature) conventionally used in spin glasses.

For the Hopfield model, our method recovers the instability line obtained by Coolen [13] using the AT approach, as a special limit, which corresponds to assuming a continuous transition from the RS to the RSB phase, in the order parameters. Therefore, our method provides a generalization of the AT approach, which can be applied to systems with a discontinuous transition from the RS to the 1RSB phase. Another advantage of our method, when compared to the involved calculations of the
AT method in the Hopfield model [13], is its remarkable simplicity. This allows for straightforward application to more complex neural network models, such as dense associative memories with \(P\)-node interactions. We supplement the results in the main text with an appendix where we show our method at work on conventional spin-glasses models, namely the Sherrington-Kirkpatrick model, the Ising \(P\)-spin and the spherical \(P\)-spin model, the latter providing an example of system which exhibits a discontinuous phase transition from the paramagnetic to the spin-glass phase. In all cases, we retrieve the AT lines known for these models [13, 16, 17, 18] in a specific limit, confirming the validity of our approach as a generalization of the AT method. As expected from the decomposition theorem of multi-node Hebbian networks proved in [4], for dense associative memories with \(P\)-node interactions we retrieve the instability line of the Ising \(P\)-spin model derived in the appendix.

## 2 The Hopfield model

In this section we illustrate the method for the Hopfield model with \(N\) Ising neurons \(\sigma_{i}\in\{1,-1\},\ i=1,\ldots,N\) and \(K=\alpha N\) stored patterns \(\mathbf{\xi}^{\mu}\), \(\mu=1,\ldots,K\). Each pattern \(\mathbf{\xi}^{\mu}\) is a sequence of \(N\) Rademacher entries (i.e. Bernoulli variables) \(\xi_{i}^{\mu}\), \(i=1,\ldots,N\), with distribution

\[\mathbb{P}(\xi_{i}^{\mu})=\frac{1}{2}\left(\delta_{\xi_{i}^{\mu},+1}+\delta_{ \xi_{i}^{\mu},-1}\right). \tag{1}\]

The Hamiltonian of the model is

\[H_{N}(\mathbf{\sigma}|\mathbf{\xi})=-\frac{1}{N}\sum_{i,j=1,1}^{N,N}\sum_{\mu=1}^{K} \xi_{i}^{\mu}\xi_{j}^{\mu}\sigma_{i}\sigma_{j}. \tag{2}\]

In the so-called'retrieval' phase, the equilibrium local configurations are correlated only with a single pattern, say \(\nu\). As the couplings \(J_{ij}=\sum_{\mu=1}^{K}\xi_{i}^{\mu}\xi_{j}^{\mu}\) are symmetric w.r.t. to permutations of the patterns, it is assumed without loss of generality that \(\nu=1\). It is thus convenient to define the following order parameters:

\[m\!:=\frac{1}{N}\sum_{i=1}^{N}\xi_{i}^{1}\sigma_{i}\qquad q\!:=\frac{1}{N}\sum _{i=1}^{N}\sigma_{i}^{(1)}\sigma_{i}^{(2)} \tag{3}\]

where the first one, denoted as the _Mattis magnetization_, quantifies the alignment of the network configuration with the retrieval pattern \(\mathbf{\xi}^{1}\) and \(q\) is the standard _two-replica overlap_ quantifying the correlations between two configurations of the network obtained with the same coupling's realization. Denoting their equilibrium values with

\[\bar{m}\!:=\frac{1}{N}\sum_{i=1}^{N}\xi_{i}^{1}\langle\sigma_{i}\rangle\qquad \bar{q}\!:=\frac{1}{N}\sum_{i=1}^{N}\langle\sigma_{i}^{(1)}\rangle\langle \sigma_{i}^{(2)}\rangle\]

where \(\langle\,\cdot\,\rangle\) is the average with respect to the Boltzmann distribution with Hamiltonian (2) and noise level \(T=1/\beta\), the RS analysis assumes that the order parameters \(q\) and \(m\) self-average around their equilibrium values \(\bar{q}\) and \(\bar{m}\), in the thermodynamic limit, namely

\[\lim_{N\rightarrow+\infty}P_{N}^{\prime}(q)= \delta(q-\bar{q}), \tag{4}\] \[\lim_{N\rightarrow+\infty}P_{N}^{\prime}(m)= \delta(m-\bar{m}). \tag{5}\]
Under this assumption, the free-energy, averaged over the pattern distribution (or 'quenched' disorder), \(f\), is given by (see [5])

\[-\beta f_{RS}(\beta,\alpha,\bar{q})= \ln 2-\frac{\alpha}{2}\ln(1-\beta(1-\bar{q}))-\frac{\beta^{2}}{2} \bar{m}^{2}+\frac{\alpha\beta\bar{q}}{2(1-\beta(1-\bar{q}))}\] \[+\frac{\alpha\beta^{2}}{2}\frac{\bar{q}^{2}}{(1-\beta(1-\bar{q}) )}+\mathbb{E}\ln\cosh\left(\beta z\sqrt{\frac{\alpha\bar{q}}{(1-\beta(1-\bar{q} ))^{2}}}+\beta\bar{m}\right), \tag{6}\]

where \(z\) is a random Gaussian variable with zero average and unit variance, \(\mathbb{E}\) denotes the average over \(z\) and \(\alpha\) is the load capacity of the network. In this limit, the order parameters \(\bar{q}\) and \(\bar{m}\) fulfill the celebrated Amit-Gutfreund-Sompolinsky self-consistency equations [5; 14]:

\[\bar{q} =\mathbb{E}\tanh^{2}\left(\beta\bar{m}+\beta J\sqrt{\frac{\alpha \bar{q}}{(1-\beta(1-\bar{q}))^{2}}}z\right), \tag{7}\] \[\bar{m} =\mathbb{E}\tanh\left(\beta\bar{m}+\beta J\sqrt{\frac{\alpha \bar{q}}{(1-\beta(1-\bar{q}))^{2}}}z\right). \tag{8}\]

On the other hand, within one step of the replica-symmetry breaking (1RSB) scheme [15; 28; 1] it is assumed that the distribution of the two-replica overlap \(q\), in the thermodynamic limit, displays two delta-peaks at the equilibrium values \(\bar{q}_{0}\) and \(\bar{q}_{1}>\bar{q}_{0}\) and the concentration on these two values is ruled by the parameter \(\theta\in[0,1]\), while \(m\) self-averages as in the RS case:

\[\lim_{N\to+\infty}P^{\prime}_{N}(q) =\theta\delta(q-\bar{q}_{0})+(1-\theta)\delta(q-\bar{q}_{1}), \tag{9}\] \[\lim_{N\to+\infty}P^{\prime}_{N}(m) =\delta(m-\bar{m}). \tag{10}\]

Within this assumption, the disorder-averaged free-energy is given by (see e.g. [1])

\[-\beta f_{1RSB}(\beta,\alpha,\theta,\bar{m},\bar{q}_{1},\bar{q}_ {0}) =\ln 2-\frac{\alpha}{2}\ln\left(\Delta_{1}(\bar{q}_{1})\right)+ \frac{\alpha}{2\theta}\ln\left(\frac{\Delta_{1}(\bar{q}_{1})}{\Delta_{2}(\bar {q}_{0},\bar{q}_{1})}\right)+\frac{\alpha\beta\bar{q}_{0}}{2\Delta_{2}(\bar{q }_{0},\bar{q}_{1})}\] \[-\frac{\beta}{2}\bar{m}^{2}+\frac{1}{2}\alpha\beta^{2}\theta \frac{\bar{q}_{0}^{2}}{\Delta_{2}^{2}(\bar{q}_{0},\bar{q}_{1})}+\frac{1}{ \theta}\mathbb{E}_{1}\ln\mathbb{E}_{2}\cosh^{\theta}g_{\theta}(\beta,\alpha, \bar{m},\bar{q}_{0},\bar{q}_{1})\] \[-\frac{1}{2}\alpha\beta^{2}\theta\bar{q}_{1}\bigg{(}\frac{\bar{q }_{0}}{\Delta_{2}^{2}(\bar{q}_{0},\bar{q}_{1})}+\frac{\bar{q}_{1}-\bar{q}_{0}} {\Delta_{1}(\bar{q}_{1})\Delta_{2}(\bar{q}_{0},\bar{q}_{1})}\bigg{)}\] \[-\frac{1}{2}\alpha\beta^{2}(1\!-\!\bar{q}_{1})\bigg{(}\frac{\bar{q }_{0}}{\Delta_{2}^{2}(\bar{q}_{0},\bar{q}_{1})}\!+\!\frac{\bar{q}_{1}-\bar{q}_ {0}}{\Delta_{1}(\bar{q}_{1})\Delta_{2}(\bar{q}_{0},\bar{q}_{1})}\bigg{)} \tag{11}\]

where, for mathematical convenience, we defined

\[\Delta_{1}(\beta,\bar{q}_{1}):=1-\beta(1-\bar{q}_{1}) \tag{12}\] \[\Delta_{2}(\beta,\theta,\bar{q}_{0},\bar{q}_{1}):=1-\beta(1- \bar{q}_{1})-\beta\theta(\bar{q}_{1}-\bar{q}_{0})\] (13) \[g_{\theta}(\beta,\alpha,\bar{m},\bar{q}_{0},\bar{q}_{1}):=\beta \bar{m}+\frac{\beta z^{(1)}\sqrt{\alpha\bar{q}_{0}}}{\Delta_{2}(\theta,\bar{q }_{0},\bar{q}_{1})}+\beta z^{(2)}\sqrt{\frac{\alpha(\bar{q}_{1}-\bar{q}_{0})} {\Delta_{1}(\bar{q}_{1})\Delta_{2}(\theta,\bar{q}_{0},\bar{q}_{1})}} \tag{14}\]

and we have denoted with \(\mathbb{E}_{1}\), \(\mathbb{E}_{2}\) the averages w.r.t. the standard normal variables \(z^{(1)}\) and \(z^{(2)}\), respectively. From now on, we imply the dependence of the functions on \(\bar{m},\ \beta,\ \alpha\). By extremizing the 1RSB free-energy w.r.t. its order parameters \(\bar{q}_{0},\ \bar{q}_{1},\ \bar{m}\), it is possible to show that the latter fulfill
the following self-consistency equations

\[\bar{m} = \mathbb{E}_{1}\left[\frac{\mathbb{E}_{2}\cosh^{\theta}g_{\theta}( \bar{q}_{0},\bar{q}_{1})\tanh g_{\theta}(\bar{q}_{0},\bar{q}_{1})}{\mathbb{E}_{ 2}\cosh^{\theta}g_{\theta}(\bar{q}_{0},\bar{q}_{1})}\right],\] \[\bar{q}_{1} = \mathbb{E}_{1}\left[\frac{\mathbb{E}_{2}\cosh^{\theta}g_{\theta}( \bar{q}_{0},\bar{q}_{1})\tanh^{2}g_{\theta}(\bar{q}_{0},\bar{q}_{1})}{\mathbb{ E}_{2}\cosh^{\theta}g_{\theta}(\bar{q}_{0},\bar{q}_{1})}\right], \tag{15}\] \[\bar{q}_{0} = \mathbb{E}_{1}\left[\frac{\mathbb{E}_{2}\cosh^{\theta}g_{\theta} (\bar{q}_{0},\bar{q}_{1})\tanh g_{\theta}(\bar{q}_{0},\bar{q}_{1})}{\mathbb{E} _{2}\cosh^{\theta}g_{\theta}(\bar{q}_{0},\bar{q}_{1})}\right]^{2}.\]

The key idea of our method is to assume that at the onset of the RS instability, one of the two delta-peaks in equation (9) has vanishing weight, i.e. either \(\theta\!\rightarrow\!0\) or \(\theta\!\rightarrow\!1\). Consistency with the RS theory then requires the dominating peak to be located at the value \(\bar{q}\) of the RS order parameter, so either \(\lim_{\theta\to 0}\bar{q}_{1}\!=\!\bar{q}\) or \(\lim_{\theta\to 1}\bar{q}_{0}\!=\!\bar{q}\). As \(q_{1}>q_{0}\geq 0\), the scenario \(\theta\!=\!0\) and \(q_{1}(\theta=0)\!=\!\bar{q}\) can only apply to systems where the RS instability manifests at a temperature \(T_{c}\) below the critical temperature \(T^{\star}\) at which \(\bar{q}\) becomes non-zero (within the RS theory). On the other hand, in systems where \(T_{c}>T^{\star}\), at the onset of RS instability \(\bar{q}=0\), so only the second scenario (\(\theta=1\) and \(q_{0}(\theta=1)=\bar{q}\)) can apply. For the Hopfield model, one can easily check using (15) that \(\lim_{\theta\to 1}\bar{q}_{0}\neq\bar{q}\), while

\[\lim_{\theta\to 0}\bar{q}_{1} = \lim_{\theta\to 0}\mathbb{E}_{1}\mathbb{E}_{2}\tanh^{2} \left(\beta\bar{m}+\beta z^{(1)}\frac{\sqrt{\alpha\bar{q}_{0}}}{\Delta_{2}( \theta,\bar{q}_{0},\bar{q}_{1})}\,+\,\beta z^{(2)}\sqrt{\alpha\frac{\bar{q}_{ 1}-\bar{q}_{0}}{\Delta_{1}(\bar{q}_{1})\Delta_{2}(\theta,\bar{q}_{0},\bar{q}_{ 1})}}\right) \tag{16}\] \[= \mathbb{E}_{1}\mathbb{E}_{2}\tanh^{2}\left(\beta\bar{m}+\beta z^ {(1)}\frac{\sqrt{\alpha\bar{q}_{0}}}{\Delta_{1}(\bar{q}_{1})}+\beta z^{(2)} \frac{\sqrt{\alpha(\bar{q}_{1}-\bar{q}_{0})}}{\Delta_{1}(\bar{q}_{1})}\right)\] \[= \mathbb{E}\tanh^{2}\left(\beta\bar{m}+\beta J\sqrt{\frac{\alpha \bar{q}_{1}}{(1-\beta(1-\bar{q}_{1}))^{2}}}z\right)\]

where we have used that for \(\theta=0\), \(\Delta_{2}(0,\bar{q}_{0},\bar{q}_{1})=\Delta_{1}(\bar{q}_{1})\) and the relation

\[\mathbb{E}_{\lambda,Y}[F(a_{1}+\lambda a_{2}+Ya_{3})]=\mathbb{E}_{x}\left[F \left(a_{1}+Z\sqrt{a_{2}^{2}+a_{3}^{2}}\right)\right], \tag{17}\]

with \(F\) any smooth function, \(a_{1},\ a_{2},\ a_{3}\in\mathbb{R}\), and \(\lambda\), \(Y\) and \(Z\) i.i.d. standard normal random variables. As (16) is identical to (7), in the limit \(\theta\to 0\), \(\bar{q}_{1}\) is equal to the RS order parameter \(\bar{q}\). Furthermore, one can easily verify that \(f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})|_{\theta=0}=f_{RS}(\bar{q})\), as expected from the fact that, for \(\theta=0\), eq. (9) reduces to (5) and one retrieves the RS scheme. Our purpose is then to prove that for small but finite values of \(\theta\) the 1RSB expression of the quenched free-energy is smaller than the RS expression, i.e. \(f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})<f_{RS}(\bar{q})\), below a critical line in the parameters space \((\alpha,\beta)\).

To this purpose, we expand the 1RSB quenched free-energy around \(\theta=0\) -namely around the replica symmetric expression- to the first order, to write

\[f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})= f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})|_{\theta=0}+\theta\partial_{ \theta}f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})|_{\theta=0}, \tag{18}\]

where \(f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})|_{\theta=0}=f_{RS}(\bar{q})\). To determine when the RS solution becomes unstable, i.e. \(f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})<f_{RS}(\bar{q})\) we inspect the sign of \(\partial_{\theta}f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})|_{\theta=0}\). To evaluate the latter, we need to expand the self-consistency equations for \(\bar{q}_{0}\) and \(\bar{q}_{1}\) to linear orders in \(\theta\). Using (16) and denoting

\[g_{0}(\bar{q}_{1},\bar{q}_{0})=\beta\bar{m}+\beta z^{(1)}\frac{\sqrt{\alpha \bar{q}_{0}}}{\Delta_{1}(\bar{q}_{1})}+\beta z^{(2)}\frac{\sqrt{\alpha(\bar{q} _{1}-\bar{q}_{0})}}{\Delta_{1}(\bar{q}_{1})}, \tag{19}\]
we obtain

\[\bar{q}_{1}=\mathbb{E}_{1}\mathbb{E}_{2}\tanh^{2}g_{0}(\bar{q}_{1},\bar{q}_{0})+ \theta A(\bar{q}_{1},\bar{q}_{0}) \tag{20}\]

where \(A(\bar{q}_{1},\bar{q}_{0})\) is a function of \(\bar{q}_{1}\) and \(\bar{q}_{0}\) that will drop out of the calculation, whose expression is provided in (16). It follows from (16) that to \(\mathcal{O}(\theta^{0})\), \(\bar{q}_{1}\) is equal to the RS order parameter \(\bar{q}\) so we can rewrite (20) as

\[\bar{q}_{1}=\bar{q}+\theta A(\bar{q},\bar{q}_{0}). \tag{21}\]

Following the same path for \(\bar{q}_{0}\), and using (21), we have

\[\bar{q}_{0}=:\mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g_{0}(\bar{q}_{0},\bar{q} )\right)^{2}+\theta B(\bar{q}_{1},\bar{q}_{0}) \tag{22}\]

where \(B(\bar{q}_{1},\bar{q}_{0})\) is provided in (16) and will drop out of the calculation. For \(\theta=0\), we have

\[\bar{q}_{0}=\mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g_{0}(\bar{q}_{0},\bar{q} )\right)^{2} \tag{23}\]

which is a self-consistency equation for \(\bar{q}_{0}\), that depends only on \(\bar{q}\). Denoting with \(\tilde{q}_{0}(\bar{q})\) its solution, we can then write (22) as

\[\bar{q}_{0}=\tilde{q}_{0}(\bar{q})+\theta B(\bar{q},\tilde{q}_{0}(\bar{q})) \tag{24}\]

and, finally,

\[\bar{q}_{1}=\bar{q}+\theta A(\bar{q},\tilde{q}_{0}(\bar{q})). \tag{25}\]

Using (25) and (24) to evaluate the derivative of \(f_{1RSB}(\theta,\bar{q}_{1},\bar{q}_{0})\) w.r.t. \(\theta\) and finally setting \(\theta=0\), we obtain:

\[K(\bar{q},\tilde{q}_{0}(\bar{q})):=\partial_{\theta}(-\beta f_{ 1RSB}(\theta,\bar{q}_{1},\tilde{q}_{0}))|_{\theta=0}\] \[=-\frac{\alpha\beta^{2}(\bar{q}^{2}-\tilde{q}_{0}(\bar{q})^{2})} {4\Delta_{1}(\bar{q})^{2}}+\frac{1}{2}\mathbb{E}_{1}\mathbb{E}_{2}\ln^{2} \cosh g_{0}(\bar{q},\tilde{q}_{0}(\bar{q}))-\frac{1}{2}\mathbb{E}_{1}\left( \mathbb{E}_{2}\ln\cosh g_{0}(\bar{q},\tilde{q}_{0}(\bar{q}))\right)^{2} \tag{26}\]

Next, we study the sign of (26), where \(\bar{q}\) and \(\tilde{q}_{0}(\bar{q})\) are the solutions of the self-consistency equations (7) and (23), respectively. To this purpose, it is useful to study the behaviour of the function \(K(\bar{q},x)\) for \(x\in[0,\bar{q}]\). For \(x=\bar{q}\), we have \(K(\bar{q},\bar{q})=0\), while the extremum of \(K(\bar{q},x)\) is found from

\[\partial_{x}K(\bar{q},x)=\frac{\beta^{2}\alpha x}{2\Delta_{1}(\bar{q})^{2}} \left[x-\mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g_{0}(\bar{q},x)\right)^{2} \right]=0\]

as

\[x=\mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g_{0}(\bar{q},x)\right)^{2}\equiv \tilde{q}_{0}(\bar{q}), \tag{27}\]

from eq. (23). Given that \(K(\bar{q},x)\) vanishes for \(x=\bar{q}\), if the extremum \(x=\tilde{q}_{0}(\bar{q})\) is global in the domain considered, we must have that \(K(\bar{q},\tilde{q}_{0}(\bar{q}))>0\) if \(x=\tilde{q}_{0}(\bar{q})\) is a maximum and \(K(\bar{q},\tilde{q}_{0}(\bar{q}))<0\) if \(x=\tilde{q}_{0}(\bar{q})\) is a minimum. Therefore, if

\[\partial_{x}^{2}K(\bar{q},x)|_{x=\tilde{q}_{0}(\bar{q})}=\frac{\beta^{2}\alpha }{2\Delta_{1}(\bar{q})^{2}}\left\{1-\frac{\beta^{2}\alpha}{\Delta_{1}(\bar{q})^ {2}}\mathbb{E}_{1}\left\{\mathbb{E}_{2}\left[\frac{1}{\cosh^{2}g_{0}(\bar{q} _{0}(\bar{q}),\bar{q})}\right]\right\}^{2}\right\} \tag{28}\]

is negative, \(K(\bar{q},\tilde{q}_{0}(\bar{q}))\) is positive and

\[f_{1RSB}(\theta,\bar{q},\tilde{q}_{0}(\bar{q}))=f_{RS}(\bar{q})-\theta\frac{K (\bar{q},\tilde{q}_{0}(\bar{q}))}{\beta}<f_{RS}(\bar{q}), \tag{29}\]
hence the RS theory becomes unstable when the expression in the curly brackets in (28) becomes negative i.e. for

\[(1-\beta(1-\bar{q}))^{2}<\beta^{2}\alpha\mathbb{E}_{1}\left\{\mathbb{E}_{2}\left[ \frac{1}{\cosh^{2}g_{0}(\bar{q},\tilde{q}_{0}(\bar{q}))}\right]\right\}^{2} \tag{30}\]

This expression recovers the result found by Coolen in [13] using the Almeida and Thouless approach [17], in the limit \(\tilde{q}_{0}(\bar{q})\rightarrow\bar{q}\), where (30) reduces to

\[(1-\beta(1-\bar{q}))^{2}<\alpha\beta^{2}\,\mathbb{E}\cosh^{-4}\left[\beta\bar{ m}+\beta z\frac{\sqrt{\alpha\bar{q}}}{1-\beta(1-\bar{q})}\right] \tag{31}\]

While this limit is _a priori_ unjustified, as \(\bar{q}\) and \(\tilde{q}_{0}(\bar{q})\) should be solved from the self-consistency equations (7) and (23), respectively, one can check numerically that the solutions of these equations are indeed very close for any temperature. We anticipate, however, that this will not be the case in Hebbian networks with \(P\)-node interactions, that we will analyse in the next section. In conclusion, our method provides a more general result than the AT approach and it applies to systems where \(\tilde{q}_{0}(\bar{q})\) and \(\bar{q}\) are different. In particular, it carries over to systems with discontinuous phase transition, where \(\bar{q}_{1}\) differs from \(\bar{q}_{0}\) even at the onset of the RS instability, which implies \(\tilde{q}_{0}(\bar{q}_{0})\neq\bar{q}\).

## 3 Hebbian networks with \(P\)-node interactions

In this section we consider generalizations of the Hopfield model, where neurons interact in P-tuples of even \(P\geq 4\) (rather than pairwise, i.e. \(P=2\)). Such networks were shown to store many more patterns than the number of their nodes, so they work as _dense_ associative memories [24]. They are also known to be dual to deep neural networks [3; 4] and to exhibit information processing capabilities that are forbidden in shallow networks, such as the existence of a region in the parameter space where they can retrieve patterns although these are overshadowed by the noise [2]. As before, we consider a network of \(N\) Ising neurons \(\sigma_{i}\in\{1,-1\}\) with \(K\) stored patterns \(\mathbf{\xi}^{\mathbf{\mu}}\), which are now vectors of \(N^{P/2}\) Rademacher entries. The Hamiltonian of this model is

\[H_{N}(\mathbf{\sigma}|\mathbf{\xi})=-\frac{N^{1-P}}{P!}\sum_{\mu=1}^{K}\!\left(\sum_{ i_{1},\ldots,i_{P/2}}\xi_{i_{1}}^{\mu}\ldots\xi_{i_{P/2}}^{\mu}\sigma_{i_{1}} \ldots\sigma_{i_{P/2}}\!\right)^{\!2} \tag{32}\]

The order parameters are still the Mattis magnetization \(m\) and the two-replicas overlap \(q\), as introduced in (3), with their RS distributions given in (4) and (5) and their 1RSB generalizations given in (9) and (10). The quenched free-energy in RS assumption is (see [19])

\[-\beta^{{}^{\prime}}f_{RS}(\beta^{{}^{\prime}},\alpha,\bar{q})= \ln 2-\frac{{\beta^{{}^{\prime}}}^{2}}{2}(P-1)\bar{m}^{P}+\frac{{ \beta^{{}^{\prime}}}^{2}\alpha}{4}(1-\bar{q}^{P})-\frac{{\beta^{{}^{\prime}}}^ {2}\alpha P}{4}\bar{q}^{P-1}(1-\bar{q})\] \[+\mathbb{E}\ln\cosh\left(\beta^{{}^{\prime}}\frac{P}{2}\bar{m}^{ P-1}+\beta^{{}^{\prime}}z\sqrt{\alpha\frac{P}{2}\bar{q}^{P-1}}\right) \tag{33}\]

with \(\beta^{{}^{\prime}}:=2\beta/P!\), where \(\beta\) is the inverse temperature and \(\alpha=\lim_{N\rightarrow\infty}K/N^{P-1}\) is the network load. \(\mathbb{E}\) is the average w.r.t. the standard Gaussian random variable \(z\), and \(\bar{q}\) and \(\bar{m}\) satisfy the
self-consistency equations:

\[\bar{m}=\mathbb{E}\tanh\left(\beta^{{}^{\prime}}\frac{P}{2}\bar{m}^{P -1}+\beta^{{}^{\prime}}J\sqrt{\alpha\frac{P}{2}\bar{q}^{P-1}}z\right),\] \[\bar{q}=\mathbb{E}\tanh^{2}\left(\beta^{{}^{\prime}}\frac{P}{2} \bar{m}^{P-1}+\beta^{{}^{\prime}}J\sqrt{\alpha\frac{P}{2}\bar{q}^{P-1}}z\right). \tag{11}\]

On the other hand, the quenched free-energy within the 1RSB approximation (see [4]), reads as

\[-\beta^{{}^{\prime}}f_{1RSB}(\beta^{{}^{\prime}},\alpha,\theta, \bar{m},\bar{q}_{0},\bar{q}_{1})= \ln 2-\frac{\beta^{{}^{\prime}}{}^{2}}{2}(P-1)\bar{m}^{P}+\frac{ \beta^{{}^{\prime}}{}^{2}\alpha}{4}\left[1-\theta\bar{q}_{0}^{P}+(\theta-1) \bar{q}_{1}^{P}\right]\] \[+\frac{\beta^{{}^{\prime}}{}^{2}\alpha P}{4}\bar{q}_{1}^{P-1}- \frac{\beta^{{}^{\prime}}{}^{2}}{4}P\alpha\left[(\theta-1)\bar{q}_{1}^{P}- \theta\bar{q}_{0}^{P}\right]\] \[+\frac{1}{\theta}\mathbb{E}_{1}\ln\mathbb{E}_{2}\cosh^{\theta}g( \beta^{{}^{\prime}},\alpha,\bar{m},\bar{q}_{0},\bar{q}_{1}) \tag{12}\]

where \(\mathbb{E}_{1}\), \(\mathbb{E}_{2}\) are the average w.r.t. the standard normal random variables \(z^{(1)}\) and \(z^{(2)}\), respectively, and

\[g(\beta^{{}^{\prime}},\alpha,\bar{m},\bar{q}_{0},\bar{q}_{1})= \frac{\beta^{{}^{\prime}}P}{2}\bar{m}^{P-1}+\beta^{{}^{\prime}}z^ {(1)}\sqrt{\frac{P}{2}\alpha\bar{q}_{0}^{P-1}}+\beta^{{}^{\prime}}z^{(2)}\sqrt {\frac{P}{2}\alpha\left(\bar{q}_{1}^{P-1}-\bar{q}_{0}^{P-1}\right)}. \tag{13}\]

From now on, we imply the dependence of the functions on \(\beta^{{}^{\prime}},\ \alpha\) and \(\bar{m}\). In this approximation the self-consistency equations for the order parameter \(\bar{q}_{1}\), \(\bar{q}_{0}\) and \(\bar{m}\) are as in (15), with the argument of the hyperbolic cosine and tangent replaced by (13). As before, for \(\theta=0\), \(\bar{q}_{1}=\bar{q}\) and the 1RSB expression for the quenched free-energy reduces to the RS one.

Our objective is to prove that the 1RSB quenched free-energy is smaller than its replica symmetric counterpart i.e. \(f_{1RSB}(\beta^{{}^{\prime}},\alpha,\theta)<f_{RS}(\beta^{{}^{\prime}},\alpha)\) above a critical value of the effective parameter \(\sqrt{\alpha}\beta^{\prime}\). To this purpose, we proceed as in the Hopfield model: we expand, to the leading order in \(\theta\), the 1RSB quenched free-energy around its RS expression, as shown in (18). Since the self-consistency equations also depend on \(\theta\), we need to expand them too. Following the same steps as in the Hopfield model, we can write \(\bar{q}_{1}\) as in (25), with \(A(\bar{q},\tilde{q}_{0}(\bar{q}))\) given in (10), and \(\bar{q}_{0}\) as given in (24), where \(\tilde{q}_{0}(\bar{q})\) is the solution of the self-consistency equation

\[\bar{q}_{0}=\mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g(\bar{q},\bar{q}_{0}) \right)^{2} \tag{14}\]

and \(B(\bar{q},\tilde{q}_{0}(\bar{q}))\) is given in (10). With the above expressions in hand, we can now calculate the derivative of \(f_{1\text{RSB w.r.t. }\theta}\) when \(\theta=0\), as needed in (18)

\[K(\bar{q},\tilde{q}_{0}(\bar{q})):=\partial_{\theta}(-\beta^{{}^ {\prime}}f_{1\text{RSB}}(\theta,\bar{q}_{1},\bar{q}_{0}))|_{\theta=0}\] \[=-\frac{\alpha\beta^{{}^{\prime}}{}^{2}}{4}(P-1)(\bar{q}^{P}- \tilde{q}_{0}^{P}(\bar{q}))+\frac{1}{2}\mathbb{E}_{1}\mathbb{E}_{2}\ln^{2} \cosh g(\bar{q},\tilde{q}_{0}(\bar{q}))-\frac{1}{2}\mathbb{E}_{1}\left(\mathbb{ E}_{2}\ln\cosh g(\bar{q},\tilde{q}_{0}(\bar{q}))\right)^{2} \tag{15}\]

Again, we have that \(K(\bar{q},\bar{q})=0\) (this follows from the fact that for \(\theta=0\), \(\bar{q}\) is an extremum of the free-energy). Next, we inspect the sign of \(K(\bar{q},\tilde{q}_{0}(\bar{q}))\). To this purpose, we study \(K(\bar{q},x)\) for \(x\in[0,\bar{q}]\) and locate its extrema, which are found from

\[\partial_{x}K(\bar{q},x)= \frac{\beta^{{}^{\prime}}{}^{2}\alpha P(P-1)}{4}x^{P-2}\left[x- \mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g(\bar{q},x)\right)^{2}\right]=0 \tag{16}\]
as

\[x=\mathbb{E}_{1}\left(\mathbb{E}_{2}\tanh g(\bar{q},x)\right)^{2} \equiv\tilde{q}_{0}(\bar{q}) \tag{11}\]

where the last equality follows from (10). Under the assumption that the extremum \(x=\tilde{q}_{0}(\bar{q})\) is global in the domain considered and reasoning as in the Hopfield case, we have that \(K(\bar{q},\tilde{q}_{0}(\bar{q}))>0\) if \(x=\tilde{q}_{0}(\bar{q})\) is a maximum and \(K(\bar{q},\tilde{q}_{0}(\bar{q}))<0\) if it is a minimum. In particular, if

\[\partial_{x}^{2}K(\bar{q},x)|_{x=\tilde{q}_{0}(\bar{q})}=-\frac{ \beta^{{}^{\prime}}{}^{2}\alpha P(P-1)}{4}\tilde{q}_{0}(\bar{q})^{P-2}\left\{1- \frac{\beta^{{}^{\prime}}{}^{2}\alpha P(P-1)\tilde{q}_{0}^{P-2}}{2}\mathbb{E} _{1}\left[\mathbb{E}_{2}\frac{1}{\cosh^{2}g(\bar{q},\tilde{q}_{0}(\bar{q}))} \right]^{2}\right\} \tag{12}\]

is negative, \(K(\bar{q},\tilde{q}_{0}(\bar{q}))>0\) and \(f_{1RSB}<f_{RS}\). This happens when the expression in the curly brackets of the equation above is negative, i.e. when the parameter \(\alpha\beta^{{}^{\prime}}{}^{2}\) satisfies the inequality

\[\frac{\alpha\beta^{{}^{\prime}}{}^{2}P(P-1)\tilde{q}_{0}^{P-2}}{ 2}\mathbb{E}_{1}\left\{\mathbb{E}_{2}\left[\frac{1}{\cosh^{2}g(\bar{q},\tilde{ q}_{0})}\right]\right\}^{2}>1. \tag{13}\]

As noted in [4; 9], Hebbian networks with \(P\)-node interactions are equivalent to Ising \(P\)-spin models under a suitable rescaling of the temperature (13) \(\beta^{{}^{\prime}}\sqrt{\alpha}\to\beta^{\prime}\). With this rescaling, (13) retrieves indeed the RS instability line of the Ising P-spin model, that we have for completeness derived in the Appendix, using our method (see eq. (102)). In the limit \(\tilde{q}_{0}(\bar{q})\to\bar{q}\), (102) retrieves the AT line of the Ising \(P\)-spin model [18], however we note that for the Ising \(P\)-spin model and for Hebbian networks with \(P\)-node interactions, \(\tilde{q}_{0}(\bar{q})\) differs from \(\bar{q}\), with deviations getting more pronounced as \(P\) is increased, hence this limit cannot be justified.

In Fig. 1 (top panels) we plot the difference between \(\tilde{q}_{0}(\bar{q})\) and \(\bar{q}\) (obtained solving numerically the self-consistency equations (23) and (7) for the Hopfield model, and equations (10) and (11) for Hebbian networks with multi-node interactions), as a function of the scaled parameter \(T/\sqrt{\alpha}\), for different values of \(\alpha\). In networks with multi-node interactions, differences can be appreciated at the onset of the RS instability for \(P=4\) and become significant for higher values of \(P\). In bottom panels we show the RS instability lines resulting from our method and the AT lines obtained in the limit \(\tilde{q}_{0}(\bar{q})\to\bar{q}\).

## 4 Discussion

In this work we proposed a simple and systematic method to derive the critical line in the parameter space \((\alpha,\beta)\), below which the 1RSB expression for the free-energy is smaller than the RS expression, in Hebbian neural networks. The same analysis for spin-glass models is carried out in the appendix. For the Hopfield model, our approach recovers the critical line obtained by Coolen using the AT approach [13] as a special limit. Similarly, we recover the known AT lines of all the spin-glass models considered in the appendix, in the same limit, showing that our method provides a generalization of the approach originally devised by Almeida and Thouless. Owing to its simplicity, our method allows for straightforward application to Hebbian networks with multi-node interactions, for which the AT-line was unknown.

The key idea of our method is to regard the 1RSB theory, which assumes two delta-peaks in the overlap distribution \(P(q)\), located at \(\bar{q}_{1}\) and \(\bar{q}_{0}<\bar{q}_{1}\), with weights \(1-\theta\) and \(\theta\), respectively, as departing continuously from the RS theory, which assumes only one peak at \(\bar{q}\). This leads us to assume that at
the onset of the RS instability, the 1RSB overlap distribution is dominated by one peak, so that either \(\theta\to 0\) or \(\theta\to 1\). Then, consistency with the RS theory requires either \(\lim_{\theta\to 0}\bar{q}_{1}\!=\!\bar{q}\) or \(\lim_{\theta\to 1}\bar{q}_{0}\!=\!\bar{q}\). As \(\bar{q}_{1}\) and \(\bar{q}_{0}\) are determined from the self-consistency equations of the 1RSB theory, for each value of \(\theta\), it is straightfoward to determine which scenario applies. Noting that for the Hopfield model and Hebbian networks with \(P\)-node interactions, one has \(\bar{q}_{1}\!=\!\bar{q}\) for \(\theta\!=\!0\), we regard the 1RSB theory as a continuous variation of the RS theory, when the parameter \(\theta\) is increased from zero. Crucially, we do not make any assumption on the location of the peaks of the 1RSB theory, which are fixed by the 1RSB self-consistency equations. We then compare the 1RSB and the RS free-energies when \(\theta\) (the only free-parameter in our analysis) is close to zero, by performing simple expansions to linear orders in \(\theta\). In doing so, we solely require that \(f_{1RSB},\bar{q}_{1},\bar{q}_{0}\) are differentiable up to the first order in a neighborhood of \(\theta\!=\!0\) and that the derivative of \(f_{1RSB}\) exists at \(\theta=0\).

Although our method is similar in spirit to the one introduced by Toninelli in [31], there is a crucial difference, in that the latter relies on the assumption \(\bar{q}_{0}\to\bar{q}\), which is, in our view, unjustified a priori. In fact, while \(\bar{q}_{1}=\bar{q}\) for \(\theta=0\), \(\bar{q}_{0}\) may differ from \(\bar{q}\), even in the limit \(\theta\to 0\). This consideration also leads to a departure of our approach from the method originally devised by Almeida and Thouless, which relies on a variation of the RS free-energy as the _order parameters_ are varied continuously around their RS values. In contrast, we study the variation of the RS free-energy as the _statistical weight_ of the order parameters is varied continuously (rather than the actual value of the order parameters).

This approach allows us to determine the instability line of the RS theory also in spin-glass models

Figure 1: Top panels: Difference between \(\bar{q}_{0}(\bar{q})\) and \(\bar{q}\) versus the scaled parameter \(T/\sqrt{\alpha}\), for different values of \(\alpha\) (as shown in the legend), for the Hopfield model (left) and Hebbian networks with \(P\)-node interactions, for \(P=4\) (mid) and \(P=6\) (right). Dotted lines show the onset of the RS instability. Bottom panels: The corresponding RS instability line (i.e. \(T_{c}\) versus \(\alpha\)) obtained via our method (red curve) and the AT line obtained in the limit \(\bar{q}_{0}(\bar{q})\to\bar{q}\) (blue curve), for the same models. Discrepancies increase with \(P=2,4,6\) from left to right. The black curves show the critical temperature \(T^{\star}\) at which \(\bar{q}\) becomes non-zero within the RS theory, i.e. the spin-glass (SG) transition with the RS theory.
which exhibit a discontinuous phase transition, for which \(\bar{q}_{1}\!\neq\!\bar{q}_{0}\) even at \(T_{c}\). As a prototypical example of this class of models, we consider in the Appendix the spherical \(P\)-spin model [15]. For this model, the 1RSB self-consistency equations show that \(\lim_{\theta\to 1}\bar{q}_{0}\!=\!\bar{q}\) and the RS instability line can be found by comparing the 1RSB and the RS free-energies when \(\theta\) is close to one. Although \(\lim_{\theta\to 0}\bar{q}_{1}\!=\!\bar{q}\)_as well_ in this model, one can show that a similar analysis for \(\theta\) close to zero would give a lower temperature for the RS instability, hence the instability that occurs at higher temperature (i.e. for \(\theta\!\simeq\!1\)) is the physical one.

This work is supported by Ministero degli Affari Esteri e della Cooperazione Internazionale (MAECI) via the BULBUL grant (Italy-Israel), CUP Project n. F85F21006230001 L.A. acknowledges E. Zegna Founder's Scholarship and UMI (Unione Matematica Italiana) for financial support and the Department of Mathematics at King's College London for the kind hospitality. L.A. and A.B. acknowledge INDAM (Istituto Nazionale d'Alta Matematica) for support. All the authors acknowledge the stimulating research environment provided by the Alan Turing Institute's Theory and Methods Challenge Fortnights event "Physics-informed Machine Learning".
