# Dense Hopfield Networks in the Teacher-Student Setting

Robin Thériault  Scuola Normale Superiore di Pisa, Piazza dei Cavalieri 7,
56126, Pisa (PI), Italy  Daniele Tantari  Department of Mathematics,
University of Bologna, Piazza di Porta San Donato 5, 40126, Bologna (BO),
Italy

###### Abstract

Dense Hopfield networks are known for their feature to prototype transition
and adversarial robustness. However, previous theoretical studies have been
mostly concerned with their storage capacity. We bridge this gap by studying
the phase diagram of $p$-body Hopfield networks in the teacher-student setting
of an unsupervised learning problem, uncovering ferromagnetic phases
reminiscent of the prototype and feature learning regimes. On the Nishimori
line, we find the critical size of the training set necessary for efficient
pattern retrieval. Interestingly, we find that that the paramagnetic to
ferromagnetic transition of the teacher-student setting coincides with the
paramagnetic to spin-glass transition of the direct model, i.e. with random
patterns. Outside of the Nishimori line, we investigate the learning
performance in relation to the inference temperature and dataset noise.
Moreover, we show that using a larger $p$ for the student than the teacher
gives the student an extensive tolerance to noise. We then derive a closed-
form expression measuring the adversarial robustness of such a student at zero
temperature, corroborating the positive correlation between number of
parameters and robustness observed in large neural networks. We also use our
model to clarify why the prototype phase of modern Hopfield networks is
adversarially robust.

##  1 Introduction

Hopfield networks are artificial neural networks that model associative memory
[1]. In the Hopfield model, examples $\sigma\in\left\\{-1,1\right\\}^{N}$ of
memories $\xi^{\mu}\in\left\\{-1,1\right\\}^{N}$, $\mu=1,\ldots,M$, are
retrieved by sampling the Gibbs distribution of a $2$-body Hamiltonian
$H\left[\sigma|\xi\right]$ at a given temperature $T$ [2]. Hopfield networks
can be trained in a biologically plausible way using Hebb’s rule [1, 3], which
leads to
$H\left[\sigma|\xi\right]=-\frac{1}{N}\sum_{\mu=1}^{M}\left(\sum_{i=1}^{N}\xi^{\mu}_{i}\sigma_{i}\right)^{2}$.
However, they can only store up to $M\sim\mathcal{O}\left(N\right)$ random
memories in the limit of large $N$ [1, 4, 5]. One way to find this scaling is
to study the phase diagram of $H\left[\sigma|\xi\right]$ as a function of the
temperature $T$ and load $\alpha=\frac{M}{N}$ [5], where the so-called
ferromagnetic phase, which extends up to $\alpha\approx 0.14$, corresponds to
accurate retrieval.

Since Hopfield’s seminal work, several generalizations have been investigated
in relation to their critical storage capacity and retrieval capabilities. For
example, parallel retrieval has been studied in relation to pattern sparsity
[6, 7, 8, 9, 10] or hierarchical interactions [11, 12, 13, 14, 15], and non-
universality has been shown with respect to more general pattern entries and
unit priors [16, 17, 18, 19, 20, 21, 22]. Efforts to overcome the
$\mathcal{O}\left(N\right)$ limitation of the capacity led to the development
of a novel class of modern Hopfield networks [23, 24, 25], which are sometimes
called dense due to their faculty to store much more memories than the
original Hopfield model [26]. These neural networks surpass
$\mathcal{O}\left(N\right)$ storage capacity by using higher-order
interactions instead of the original $2$-body couplings [27, 28, 29, 30, 31,
32]. In particular, Gardner [30] calculated the replica-symmetric (RS) phase
diagram of the Hamiltonian
$H\left[\sigma|\xi\right]=-\sum_{i_{1}<...<i_{p}=1}^{N}J_{i_{1}...i_{p}}\sigma_{i_{1}}...\sigma_{i_{p}}$
with $p$-body interactions
$J_{i_{1}...i_{p}}=\frac{p!}{N^{p-1}}\sum_{\mu=1}^{M}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}$
conditioned on random memories $\xi^{\mu}\in\left\\{-1,1\right\\}^{N}$,
finding a $M=\mathcal{O}\left(N^{p-1}\right)$ storage capacity. These
calculations were later extended to include the effects of one-step replica
symmetry breaking (1RSB) [33].

Although they draw a rather detailed picture of the retrieval of individual
random memories, these results are not the end of the story. First of all,
there is still significant uncertainty on the location of the paramagnetic to
spin-glass phase transition because of numerical instability. Second of all,
dense Hopfield networks have been rapidly gaining a renewed attention for
reasons other than their storage capacity since Krotov’s recent paper [26],
where they were used as a trainable machine learning architecture. For
instance, they have been related to transformers [23, 34] and diffusion models
[35, 36], and they were found to be significantly more explainable and
adversarially robust than feedforward neural networks with ReLU activation
functions [26, 37].

One such aspect of dense Hopfield networks that is still poorly understood is
their performance as generative models for unsupervised learning, where they
are trained over some given dataset to reproduce its probability distribution.
As far as we are aware, this problem has not yet been studied theoretically
for $p$-body models with $p\geq 3$. However, it was studied for the original
$2$-body Hopfield network by using the teacher-student setting [38] first
described in [16, 17, 39]. In the teacher-student setting, which is also
called inverse problem in opposition to the direct problem of random pattern
retrieval, a student model $H\left[\xi|\sigma\right]$ is trained with $M$
teacher examples $\sigma^{a}\sim H\left[\sigma^{a}|\xi^{*}\right]$ conditioned
on the planted pattern $\xi^{*}$. In other words, the student tries to infer
the pattern $\xi^{*}$ of the teacher using a structured set of examples
$\sigma^{a}$.

At finite load $\alpha=\frac{M}{N}$, two regimes of pattern retrieval were
found: example retrieval ($eR$) and signal retrieval ($sR$). In the $eR$
phase, the student tries to reconstruct $\xi^{*}$ by directly retrieving the
examples $\sigma^{a}$, which is a good strategy provided that they are
strongly correlated with $\xi^{*}$. In the $sR$ phase, on the other hand,
retrieval is done by extracting subtle cues from weakly correlated examples.
The two types of examples used in these two retrieval strategies are
respectively called prototypes and features of $\xi^{*}$ [26]. Interestingly,
a prototype regime and a feature regime were also observed by Krotov in dense
Hopfield networks trained to classify real data [26], where it was found that
the prototype regime is significantly more adversarially robust than the
feature regime. In other words, the prototype regime is more resistant than
the feature regime to small data perturbations that are specifically designed
to cause incorrect classification [40, 41]. This prototype approach is
arguably a big step toward designing adversarially robust neural networks, a
long-standing problem that still lacks a fully satisfying solution [42, 43,
44].

In this work, we study the performance of $p$-body Hopfield networks in the
teacher-student setting, revealing a prototype regime and a feature regime as
in the $2$-body model. In Section 2, we review Gardner’s main results in
studying $p$-body Hopfield models. In Section 3, we compute the phase diagram
of these $p$-body models in the teacher-student setting. In Section 4.1, we
discuss the transition to the retrieval phase in the inverse problem and
compare it against the transition to the spin-glass phase in the direct
problem. Despite their different nature, we show that these two transitions
are equivalent on the Nishimori line where the teacher and the student have
the same $p$ and $T$ [45, 46, 47, 48]. In Section 4.2, we discuss the phase
diagram on the Nishimori line in more details. In Section 4.3 and Section 4.4,
we discuss the phase diagram outside of the Nishimori line. First of all, we
investigate the effect of using an inference temperature different from the
dataset noise. Second of all, we reveal that using a larger $p$ for the
student than the teacher gives the student an extensive tolerance against both
teacher noise and pattern interference. Finally, in Section 4.5, we derive a
closed-form expression that measures the adversarial robustness of the student
at zero temperature and explain what our results reveal about the nature of
adversarial attacks.

##  2 Overview of Gardner’s results

Consider the $p$-body Hamiltonian

$H\left[\sigma|\xi\right]=-\sum_{i_{1}<...<i_{p}=1}^{N}J_{i_{1}...i_{p}}\sigma_{i_{1}}...\sigma_{i_{p}}=-\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}=1}^{N}\sum_{\mu=1}^{M}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma_{i_{1}}...\sigma_{i_{p}}$
(1)

conditioned on a set of $M=\frac{\alpha N^{p-1}}{p!}$ quenched memories
$\xi^{\mu}\in\\{-1,1\\}^{N}$, $\mu=1,\ldots,M$, sampled i.i.d. from the
Rademacher distribution
$\frac{1}{2}\left[\delta\left(\xi^{\mu}_{i}-1\right)+\delta\left(\xi^{\mu}_{i}+1\right)\right]$.
In the direct model, patterns $\sigma$ are in turn sampled from the
equilibrium Gibbs distribution $P(\sigma|\xi)=Z^{-1}e^{-\beta H[\sigma|\xi]}$,
where $\beta\geq 0$ is the inverse temperature and $Z=\sum_{\sigma}e^{-\beta
H[\sigma|\xi]}$ is the system’s partition function. The so-called direct
problem studied by Gardner [30] consists of quantifying the performance of
this model as a method of memory retrieval. In that context, the overlap
$\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma_{i}$ is a good measure of retrieval
accuracy, and its expected value can be derived from the quenched free entropy
$f=\frac{1}{N}\left\langle\log Z\right\rangle_{\xi}$ in the thermodynamic
limit $N\to\infty$. At finite $p$, Gardner used the (non-rigorous) replica
trick [49] to evaluate the RS approximation of $f$ (see also Appendix B) in
terms of a variational principle of the form

$\displaystyle f=\lim_{N\to\infty}\frac{1}{N}\left\langle\log
Z\right\rangle_{\xi}$ $\displaystyle=\lim_{N\to\infty,L\rightarrow
0}\left(\frac{\partial}{\partial L}\left[\frac{1}{N}\log\left\langle
Z^{L}\right\rangle_{\xi}\right]\right)=\operatorname*{Extr}_{m,k,q,k,r}\
f(m,k,q,r),$

whose solution is

$\displaystyle q$
$\displaystyle=\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}x^{2}\right)\tanh^{2}\left(\beta\left[\sqrt{\alpha
r}x+k\right]\right)$ $\displaystyle m$
$\displaystyle=\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}x^{2}\right)\tanh\left(\beta\left[\sqrt{\alpha
r}x+k\right]\right)$ $\displaystyle r$ $\displaystyle=pq^{p-1}$ $\displaystyle
k$ $\displaystyle=pm^{p-1},$

and the order parameters $m$ and $q$ are to be interpreted as expected
overlaps. To be more precise, $m$ can be shown to be the expected overlap of a
retrieval attempt $\sigma$ against one memory in the thermodynamic limit, i.e.
$m=\lim_{N\to\infty}\left\langle\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma_{i}\right\rangle_{\xi,\sigma}$.
Similarly, $q$ is the expected overlap between two retrieval attempts
$\sigma^{1}$ and $\sigma^{2}$, i.e.
$q=\lim_{N\to\infty}\left\langle\frac{1}{N}\sum_{i}\sigma^{1}_{i}\sigma^{2}_{i}\right\rangle_{\xi,\sigma}=\lim_{N\to\infty}\left\langle\frac{1}{N}\sum_{i}\left\langle\sigma_{i}\right\rangle^{2}_{\sigma}\right\rangle_{\xi}$.
Intuitively, $q$ measures the tendency of the system to stay frozen in
specific configurations rather than visiting all possible values of $\sigma$.

The resulting RS phase diagram (see Fig. 1) can be derived from the value of
the order parameters as a function of three hyperparameters: the interaction
order $p$, temperature $T=1/\beta$ and load $\alpha=\frac{Mp!}{N^{p-1}}$.
There are four different phases:

  * •

In the Paramagnetic phase ($P$), the overlaps $m$ and $q$ both vanish. The
network does not retrieve any specific pattern: sampled configurations are
completely random.

  * •

In the Spin-Glass phase ($SG$), $m$ vanishes but $q>0$. In other terms, the
network does not retrieve individual stored memories but rather converges to
spurious patterns depending on all the memories in a non-trivial way.

  * •

In the signal Retrieval phases ($lR$ and $gR$), $m\neq 0$ and $q>0$, which
means that the network is able to retrieve the stored memories. $lR$ and $gR$
are respectively locally stable and globally stable. In other words, local
retrieval $lR$ is only attainable from initial conditions in a limited
neighborhood of a memory $\xi^{\mu}$, while global retrieval $gR$ is
accessible from any initial conditions given enough time. These two phases are
said to be ferromagnetic.

Figure 1: RS phase diagrams of the direct models with $p=3$ on the left and
$p=10$ on the right. Accurate pattern retrieval is not possible in the
paramagnetic phase ($P$) or in the spin-glass phase ($SG$), but it is possible
in the local retrieval phase ($lR$) and in the global retrieval phase ($gR$).
The ferromagnetic fixed point corresponding to accurate pattern retrieval is
globally stable in the $gR$ phase, but locally stable in the $lR$ phase. The
phase diagrams are inexact below the white dashed line where the total entropy
of the paramagnetic phase becomes negative.

Gardner also calculated the exact $p\rightarrow\infty$ phase diagram without
making any assumptions about replica symmetry [30]. The resulting paramagnetic
to spin-glass ($P$-$SG$) phase transition line, given by
$\beta^{2}\alpha=2\log 2$, coincides with the boundary of the region where the
total entropy of the paramagnetic phase becomes negative. Therefore, the total
entropy is of the system is always positive, as expected. Conversely, the
finite $p$ phase diagrams obtained under the RS approximation have a limited
scope because RSB must be taken into account in the $\beta^{2}\alpha>2\log 2$
region where the RS approximation of the total entropy becomes negative [30,
50]. Although this phase diagram was recently extended by including the
effects of 1RSB [33], there is still significant uncertainty on the location
of the large-$p$ $P$-$SG$ transition because of numerical instability issues.
In fact, the finite-$p$ transition evaluated in [33] does not seem to get
closer to the exact $p\rightarrow\infty$ transition as $p$ gets larger.

##  3 Teacher-student setting

On our end, we study a dense Hopfield network with Hamiltonian (1) as a
generative model for unsupervised learning. In that context, the memories
$\xi$ are model parameters that have to be trained in such a way that the
examples of a given dataset $\\{\sigma^{a}\\}_{a=1}^{M}$ result as typical
network configurations.

In particular, we study a controlled teacher-student setting in which the
examples are sampled from the probability distribution
$P\left(\sigma^{a}|\xi^{*}\right)$ of a so-called teacher dense Hopfield
network conditioned on a single planted pattern
$\xi^{*}\in\left\\{-1,1\right\\}^{N}$ whose entries are quenched Rademacher
random variables. A student dense Hopfield network, also known as the inverse
model, then samples its own student pattern $\xi$ from the posterior
distribution

$\displaystyle P\left(\xi|\sigma\right)$
$\displaystyle=\frac{P(\xi)\prod_{a=1}^{M}P\left(\sigma^{a}|\xi\right)}{P(\sigma)}=\frac{P(\xi)}{P(\sigma)}\prod_{a=1}^{M}Z^{-1}\exp\left(-\beta
H[\sigma^{a}|\xi]\right),$

where $P\left(\sigma^{a}|\xi\right)$ is the Gibbs distribution of the direct
model with a single memory $\xi$, and $P\left(\xi\right)$ is the prior on
$\xi$ that is chosen to be uniform. Since the direct model has only a single
pattern, $Z$ does not depend on $\xi$ (see Appendix C), and the posterior
simplifies to

$\displaystyle P\left(\xi|\sigma\right)$
$\displaystyle=\mathcal{Z}^{-1}(\sigma)\exp\left(-\beta H[\xi|\sigma]\right).$

In sum, the student posterior distribution is that of a dense Hopfield network
where $\xi$ plays the role of the sampled pattern and the examples $\sigma$
act like the $M$ quenched memories. Our task, called the inverse problem,
consists of quantifying the student’s capability to infer the teacher pattern,
which we will also call the signal. Like Gardner, we calculate a free entropy
of the form $f=\frac{1}{N}\left\langle\log\mathcal{Z}\right\rangle_{\sigma}$
in the thermodynamic limit $N\to\infty$. This time, however, the average
$\left\langle\cdot\right\rangle_{\sigma}$ is over a structured set of examples
$\sigma$. In fact, we recall that, unlike the i.i.d. memories studied by
Gardner, the examples $\sigma^{a}$ are sampled from the teacher distribution
$P\left(\sigma^{a}|\xi^{*}\right)$.

In general, the student does not have access to the teacher generative model.
In our controlled teacher-student setting, the student knows that the correct
model for $P\left(\sigma^{a}|\xi\right)$ is a dense Hopfield network.
Nevertheless, it does not necessarily have access to the interaction order
$p^{*}$ and inverse temperature $\beta^{*}$ used by the teacher. Therefore, we
denote the student hyperparameters by $p$ and $\beta$ and emphasize that they
are not necessarily equal to $p^{*}$ and $\beta^{*}$. As previously stated, we
calculate the free entropy

$\displaystyle f=\frac{1}{N}\left\langle\log\mathcal{Z}\right\rangle_{\sigma}$
$\displaystyle=2^{-N}\sum_{\xi^{*}}\sum_{\sigma}\left[Z^{*}\right]^{-M}\exp\left(\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a=1}^{M}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)$
$\displaystyle\quad\times\log\sum_{\xi}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{a=1}^{M}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)$
(2)

in the thermodynamic limit $N\to\infty$. We then draw phase diagrams of the
inverse problem as a function of $p^{*}$, $T^{*}=1/\beta^{*}$, $p$,
$T=1/\beta$ and $\alpha$.

We first consider the case where $p^{*}=p$ and the only possible mismatch
between the teacher and student networks is in the inverse temperature, i.e.
$\beta^{*}\neq\beta$. At low $T^{*}$, the student’s task is easy. In fact,
below the critical temperature $T_{\text{crit}}$ of the direct problem with
one pattern (see Fig. 1), the teacher produces examples $\sigma^{a}$ that
cluster around $\xi^{*}$. Therefore, the student can infer $\xi^{*}$ by
aligning its pattern $\xi$ with the examples $\sigma^{a}$. This retrieval
strategy works even when using a very small amount of examples (see [38]).
Since the size of our dataset is extensive, the retrieval accuracy is maximum
in the thermodynamic limit. We call this region the (accurate) example
Retrieval phase ($eR$).

Conversely, when $T^{*}$ is above $T_{\text{crit}}$, the examples in the
training set are very noisy and we do not observe a finite overlap between
$\sigma^{a}$ and $\xi^{*}$. In this regime, we find that the RS approximation
of the free entropy can be computed (see Appendix D) in terms of the
variational principle

$\displaystyle f$
$\displaystyle=\operatorname*{Extr}_{m,k,q,r,q^{*},r^{*}}\Bigg{\\{}\beta^{*}\beta\alpha\left[q^{*}\right]^{p}-\frac{1}{2}\beta^{2}\alpha
q^{p}+\beta m^{p}-\beta^{*}\beta\alpha r^{*}q^{*}$ (3)
$\displaystyle\quad+\frac{1}{2}\beta^{2}\alpha rq-\frac{1}{2}\beta^{2}\alpha
r-\beta mk+\frac{1}{2}\beta^{2}\alpha+\log 2$ $\displaystyle\quad+\int
dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\Big{\langle}\log\left[\cosh\left(\beta\left[\sqrt{\alpha
r}x+\beta^{*}\alpha r^{*}+kz\right]\right)\right]\Big{\rangle}_{z}\Bigg{\\}},$

whose solution is the saddle-point equations

$\displaystyle q^{*}$
$\displaystyle=\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}x^{2}\right)\Big{\langle}\tanh\left(\beta\left[\sqrt{\alpha
r}x+\beta^{*}\alpha r^{*}+kz\right]\right)\Big{\rangle}_{z}$ $\displaystyle q$
$\displaystyle=\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}x^{2}\right)\Big{\langle}\tanh^{2}\left(\beta\left[\sqrt{\alpha
r}x+\beta^{*}\alpha r^{*}+kz\right]\right)\Big{\rangle}_{z}$ $\displaystyle m$
$\displaystyle=\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}x^{2}\right)\Big{\langle}z\tanh\left(\beta\left[\sqrt{\alpha
r}x+\beta^{*}\alpha r^{*}+kz\right]\right)\Big{\rangle}_{z}$ (4)
$\displaystyle r^{*}$ $\displaystyle=p\left[q^{*}\right]^{p-1}$ $\displaystyle
r$ $\displaystyle=pq^{p-1}$ $\displaystyle k$ $\displaystyle=pm^{p-1},$

where $z$ is a Rademacher random variable. As in the direct model described in
Section 2, the order parameters $m$ and $q$ have a clear interpretation in
terms of expected overlaps.
$m=\lim_{N\to\infty}\left\langle\frac{1}{N}\sum_{i}\xi_{i}\sigma^{a}_{i}\right\rangle_{\xi^{*},\sigma,\xi}$
is the expected overlap of a retrieval attempt with an example $\sigma^{a}$,
and
$q=\lim_{N\to\infty}\left\langle\frac{1}{N}\sum_{i}\left\langle\xi_{i}\right\rangle^{2}_{\xi}\right\rangle_{\xi^{*},\sigma}$
is the expected overlap between two retrieval attempts. Similarly, $q^{*}$ is
the expected overlap between the teacher and student patterns, i.e.
$q^{*}=\lim_{N\to\infty}\left\langle\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi_{i}\right\rangle_{\xi^{*},\sigma,\xi}$.
Therefore, it is a good measure of inference performance. The free entropy
(Eq. 3) is expected to be exact in absence of mismatch between the teacher and
the student, i.e. $\beta^{*}=\beta$. This condition is known as the Nishimori
line [45, 46, 47, 48]. Outside of the Nishimori region, RSB corrections are
expected. Like in the direct problem, there are different phases characterized
by the values of the order parameters:

  * •

In the Paramagnetic phase ($P$), the overlaps $m$, $q^{*}$ and $q$ all vanish.

  * •

In the signal Retrieval phases ($lR$ and $gR$), $m=0$ but $q^{*}\neq 0$ and
$q>0$. $lR$ and $gR$ are respectively locally stable and globally stable. In
other words, local retrieval $lR$ is only attainable from initial conditions
in a limited neighborhood of $\xi^{*}$, while global retrieval $gR$ is
accessible from any initial conditions given enough time. These two phases are
also said to be ferromagnetic.

  * •

In the (inaccurate) example Retrieval phase ($eR$), $m\neq 0$ and $q>0$ but
$q^{*}=0$.

  * •

In the Spin-Glass phase (SG), $q>0$ but $q^{*}$ and $m$ vanish.

In sum, when $T^{*}$ is above $T_{\text{crit}}$, the student can only learn
the teacher pattern in the signal retrieval phases. In all the other phases,
the student pattern is uncorrelated with the signal, being either a random
guess ($P$ phase), aligned with a noisy example (inaccurate $eR$ phase), or
aligned with a spurious low energy state ($SG$ phase).

We also investigate the $T^{*}>T_{\text{crit}}$ regime in the presence of a
mismatch between the interaction orders of the teacher and student networks,
i.e. $p^{*}\neq p$. We focus on the case of $p^{*}=2$ and even $p\geq 3$ to
study the consequences of fitting the teacher of [38] using a student with
higher order interactions. We find two different scaling regimes of the
training set size $M$ and inverse temperature $\beta^{*}$ that make retrieval
possible (see Appendix D):

  * •

a large-noise scaling where $\beta^{*}\sim\mathcal{O}\left(N^{2/p-1}\right)$
and $M\sim\mathcal{O}\left(N^{p-1}\right)$, such that
$\alpha=\frac{Mp!}{N^{p-1}}$ and
$\lambda=\frac{\left[\beta^{*}\right]^{p/2}}{(p/2)!}N^{p/2-1}$ are finite;

  * •

a finite-noise scaling where $\beta^{*}\sim\mathcal{O}\left(1\right)$ and
$M\sim\mathcal{O}\left(N^{p/2}\right)$, such that
$\alpha=\frac{M(p/2+1)!}{N^{p/2}}$ is finite.

In the large-noise scaling, we obtain saddle point equations similar to Eqs.
(3) but with $\beta^{*}$ replaced by $\lambda$ (see Appendix D). Conversely,
the finite noise scaling leads to

$\displaystyle q^{*}$
$\displaystyle=\Big{\langle}\tanh\left(\beta\left[\eta\alpha
r^{*}+kz\right]\right)\Big{\rangle}_{z}$ $\displaystyle m$
$\displaystyle=\Big{\langle}z\tanh\left(\beta\left[\eta\alpha
r^{*}+kz\right]\right)\Big{\rangle}_{z}$ (5) $\displaystyle r^{*}$
$\displaystyle=p\left[q^{*}\right]^{p-1}$ $\displaystyle k$
$\displaystyle=pm^{p-1},$

where $\eta$ generally depends on $\beta^{*}$ and $p$ in a non-trivial way,
but we find that
$\eta=\frac{2\left[\beta^{*}\right]^{2}}{\left(1-2\beta^{*}\right)^{2}}$ when
$p=4$ (see Appendix D). These equations can also be derived by extrapolating
the large-noise equations equations to $\alpha_{\text{large noise}}\rightarrow
0$ and $\lambda\rightarrow\infty$ with fixed $\lambda\alpha_{\text{large
noise}}=\eta\alpha_{\text{finite noise}}$.

##  4 Results and Discussion

###  4.1 Transition to the ordered phases: universality

The paramagnetic solution of Eqs. (3) always exists and is globally stable in
the part of the phase diagram where the $T$ is relatively large and $\alpha$
is relatively small. On the other hand, the $gR$ phase exists when
$\beta^{2}\alpha p$ and $\beta^{*}\beta\alpha p$ are both large. In fact, in
that limit, $q^{*}=q=1$ is a fixed point of Eqs. (3). The critical line where
$gR$ becomes globally stable instead of $P$ is not clear from this analysis
alone, but we can at least find it analytically in the limit of infinite $p$.
As for the direct model, the free entropy and the total entropy of the
paramagnetic phase are respectively $\frac{1}{2}\beta^{2}\alpha+\log 2$ and
$-\frac{1}{2}\beta^{2}\alpha+\log 2$ [30]. At the same time, the
$p\rightarrow\infty$ free entropy takes the form

$\displaystyle f$
$\displaystyle=\operatorname*{Extr}\Bigg{\\{}\beta^{*}\beta\alpha\
\theta\left[q^{*}\right]-\frac{1}{2}\beta^{2}\alpha\
\theta\left[q\right]-\beta^{*}\beta\alpha
r^{*}q^{*}+\frac{1}{2}\beta^{2}\alpha rq-\frac{1}{2}\beta^{2}\alpha
r+\frac{1}{2}\beta^{2}\alpha+\log 2$ $\displaystyle\quad+\int
dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\log\left[\cosh\left(\sqrt{\beta^{2}\alpha
r}x+\beta^{*}\beta\alpha r^{*}\right)\right]\Bigg{\\}},$

where $\theta\left[q\right]:=\lim_{p\rightarrow\infty}q^{p}$,
$q\in\left[0,1\right]$, is the Heaviside step function jumping at $q=1$, i.e.
$\theta(1)=1$ and $\theta(q)=0$ $\forall q\in[0,1)$. In this limit, the
ferromagnetic phase is characterized by $q=q^{*}=1$, and its free entropy is
then

$\displaystyle f$ $\displaystyle=\beta^{*}\beta\alpha-\beta^{*}\beta\alpha
p+\int
dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\log\left[2\cosh\left(\sqrt{\beta^{2}\alpha
p}x+\beta^{*}\beta\alpha p\right)\right]$
$\displaystyle\approx\beta^{*}\beta\alpha-\beta^{*}\beta\alpha p+\int
dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\left(\sqrt{\beta^{2}\alpha
p}x+\beta^{*}\beta\alpha p\right)$ $\displaystyle=\beta^{*}\beta\alpha.$

The corresponding total entropy is $s=f-\beta\frac{\partial
f}{\partial\beta}=0$, as expected from a ferromagnetic phase with $q^{*}=q=1$.
On the Nishimori line, $f=\beta^{*}\beta\alpha$ becomes larger than the free
entropy of the paramagnetic phase, which triggers a phase transition, if and
only if

$\displaystyle T$ $\displaystyle<\sqrt{\frac{\alpha}{2\log 2}},$ (6)

where $T=\sqrt{\frac{\alpha}{2\log 2}}$ is also the temperature where the
entropy of the paramagnetic phase becomes negative. Outside of the Nishimori
line, this inequality generalizes to
$\beta^{*}\beta\alpha>\frac{1}{2}\beta^{2}\alpha+\log 2$, leading to

$\displaystyle\beta^{*}-\sqrt{\left[\beta^{*}\right]^{2}-\frac{2\log
2}{\alpha}}<\beta$
$\displaystyle<\beta^{*}+\sqrt{\left[\beta^{*}\right]^{2}-\frac{2\log
2}{\alpha}},$

while the temperature where the paramagnetic free entropy becomes negative
stays the same.

In the $p\rightarrow\infty$ limit, the transition towards $gR$ of the inverse
model on the Nishimori line is identical to the exact $P$-$SG$ transition of
the direct model [30]. We claim that these two critical lines are actually
closely related for any $p$. In the Hopfield model with $p=2$, they were
already shown to be identical [38]. We will now argue that they overlap for
any $p$ and $\beta$ such that the inverse model is outside of the $eR$ phase.
In the case of $p=2$, both lines can be obtained exactly from the RS
approximation of either the direct model or the inverse model, so there is no
obvious advantage to using this equivalence in calculations. In general, while
the inverse problem on the Nishimori line is replica symmetric, the direct
problem is not, and the $p\geq 3$ replica symmetric $P$-$SG$ transition is not
exact. Moreover, even the critical line calculated using 1RSB may be
inaccurate due to numerical instability [33]. In this situation, the knowledge
of the $gR$ transition in the replica-symmetric inverse problem can be used to
locate the exact $P$-$SG$ transition of the direct problem, where symmetry
breaking occurs.

For that purpose, we will argue that the direct model is in the paramagnetic
phase if and only if the inverse model is in the paramagnetic phase.

The converse implication comes from the fact that since (see Appendix C)

$P\left(\sigma\right)=\frac{1}{2^{MN}}\frac{\mathcal{Z}(\sigma)}{\left\langle\mathcal{Z}\right\rangle},$
(7)

the example distribution $P\left(\sigma\right)$ of the inverse problem is
contiguous [51] to the uniform distribution, i.e. the memory distribution of
the direct problem, when

$\lim_{N\rightarrow\infty}\left\\{\frac{\log\mathcal{Z}-\log\left\langle\mathcal{Z}\right\rangle}{N}\right\\}=0.$
(8)

As determined in Appendix C and D, the annealed expression
$\frac{1}{N}\log\left\langle\mathcal{Z}\right\rangle$ is equal to the free
entropy of the paramagnetic phase. Therefore, when the inverse model is in the
paramagnetic phase, $P\left(\sigma\right)$ is contiguous to the uniform
distribution. This property is called quiet planting and is known to occur
more generally in mean-field paramagnets [52, 53, 54, 55]. In our problem
setting, it means that if the inverse model is in the paramagnetic phase, then
it is equivalent to the direct model. In particular, if the inverse model is
in the paramagnetic phase, then so is the direct model. In more intuitive
terms, the $gR$ transition temperature of the inverse model must be greater
than or equal to the $P$-$SG$ transition temperature of the direct model
because the ensemble of examples $\sigma^{a}$ generated by the teacher model
is on average at least as structured as the set of random memories stored in
the direct model.

For the direct implication, notice that the average replicated partition
function of the direct model in the paramagnetic phase can be approximated as
(see Appendix E)

$\displaystyle\left\langle Z^{L}\right\rangle$
$\displaystyle\approx\frac{1}{\left\langle
Z\right\rangle}\Bigg{\langle}\sum_{\sigma}\exp\Bigg{(}\beta
N\sum_{\gamma}\sum_{\mu\in\Gamma_{\gamma}}\left[\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma^{\gamma}_{i}\right]^{p}$
$\displaystyle\quad+\beta\sum_{\gamma}\sum_{\mu\in\bar{\Gamma}}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\Bigg{)}$
$\displaystyle\quad\sum_{\sigma_{0}}\exp\left(\beta\sum_{\mu\in\bar{\Gamma}}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{0}_{i_{1}}...\sigma^{0}_{i_{p}}\right)\Bigg{\rangle}.$

This expression is identical to the replicated partition function of the
inverse model outside of the $eR$ phase, which therefore must also be in the
paramagnetic phase.

As a consequence, the $P$-$SG$ transition line of the direct model must be
identical to the $gR$ transition line of the inverse model on the Nishimori
line.

###  4.2 Phase diagram on the Nishimori line

On the Nishimori line, the student is fully informed about the teacher
generative model and uses $\beta=\beta^{*}$ and $p=p^{*}$. In this scenario,
thanks to the Nishimori identities [46], it is well known that $\xi^{*}$ and
$\xi$ play symmetric roles and that $q^{*}=q$. For the same reason, the
overlaps $\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi_{i}$ and
$\frac{1}{N}\sum_{i}\xi^{1}_{i}\xi^{2}_{i}$ have the same distribution. From
the self-averaging of $\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi_{i}$, it follows that
the system is expected to be replica symmetric, and Eqs. (3) and (3) are
expected to hold. Fig. (2) shows the phase diagrams obtained by solving the
saddle-point equations numerically on the Nishimori line. Both $q^{*}=q$ and
the replica symmetry condition are verified. In particular, numerical
solutions of a few values of $p\geq 3$ show that the $gR$ transition occurs at
a higher $T$ than the line $\beta^{2}\alpha=2\log 2$ where the total entropy
of the paramagnetic phase becomes negative. In other terms, the phase
transition towards $gR$ prevents the total entropy from becoming negative when
$T$ decreases below $\sqrt{\frac{\alpha}{2\log 2}}$, which is consistent with
the RS solution being exact on the Nishimori line.

Figure 2: Exact replica-symmetric phase diagrams of the inverse models with
$p=3$ on the left, $p=4$ in the center and $p=10$ on the right. Accurate
pattern retrieval is not possible in the paramagnetic phase ($P$), but it is
possible in the local retrieval phase ($lR$), in the global retrieval phase
($gR$) and in the example retrieval phase ($eR$). The ferromagnetic fixed
point corresponding to accurate pattern retrieval is globally stable in the
$gR$ phase, but locally stable in the $lR$ phase. The black dashed lines mark
the spurious continuation of the $lR$ and $gR$ phase boundaries through the
$eR$ phase. The white dashed line is the $p\rightarrow\infty$ $gR$ critical
line calculated analytically at the end of Section 3. It matches the
corresponding numerical phase boundary increasingly well as $p$ grows larger.

At low $T$, the student can learn efficiently within the accurate $eR$ regime.
In this phase, learning is possible ($q^{*}\neq 0$) because the examples are
correlated with the signal and the student can retrieve it by simply being
aligned with them ($m\neq 0$).

At high $T$, learning is possible only if the amount of examples, i.e. the
size of the dataset, is sufficiently large. When $\alpha$ is too small, Eqs.
(3) have only a paramagnetic fixed point because the amount of information
carried by the dataset is not large enough. Numerical solutions suggest that
the paramagnetic fixed point always exist and it is actually locally stable in
the whole high-temperature regime. When $\alpha$ is sufficiently large, the
signal retrieval fixed point appears as a locally stable attractor. It becomes
globally stable as the size of the dataset is increased further.

The critical boundary of the $gR$ phase can be obtained by solving the saddle-
point equations numerically (Eqs. 3), and the result is consistent with the
analytical $p\rightarrow\infty$ $gR$ boundary of Eq. (6). In fact, we find
that the analytical boundary closely agrees with the numerical solution of the
saddle-point equations with $p^{*}=p=10$ and remains a good approximation even
down to $p^{*}=p=4$.

In the student model, $\sigma$ plays a similar role as the weights of the
trainable dense Hopfield network model that Krotov designed for classification
of data [26]. In that context, $\xi$ is analogous to the test data whose
labels are being predicted (see Fig. 3). In fact, the computation performed by
Krotov’s model to recover labels is similar to the update rule used by the
student to infer the teacher pattern (see Appendix A). Moreover, the $eR$ and
$gR$ phases are respectively reminiscent of the prototype and feature regimes
of Krotov’s networks. Therefore, we believe that the student can act as a toy
model of label prediction in these two regimes.

ptTeacher pattern $\xi^{*}$Examples $\sigma$Student pattern $\xi$Data $x$with
known labels $y$ptWeights $w$Data $x$with predicted labels
$y^{\prime}$pt$H\left(\sigma\big{|}\xi^{*}\right)$pt$H\left(\xi^{b}\big{|}\sigma\right)$ptBackward
propagationptForward propagation Figure 3: The first row of this diagram
sketches how a $p$-body Hopfield network in the teacher-student setting can
reconstruct an incomplete pattern $\xi^{b}$ to match the teacher pattern
$\xi^{*}$ by relying on the examples $\sigma$ obtained from $\xi^{*}$. The
second row summarizes how a dense neural network trained by Krotov can recover
the labels $y^{\prime}$ of the data $x$ given the weights $w$ learned from $x$
[26]. Both models tackle similar tasks using an approach where $\sigma$ and
$\xi^{b}$ respectively play the same roles as $w$ and $(x,y^{\prime})$. The
forward propagation algorithm used to generate $y^{\prime}$ is similar to the
update rule of the student (see [26] and Appendix A), but the backpropagation
algorithm used to learn $w$ is very different from the update rule of the
teacher.

Comparing instead the phase diagrams of our inverse model with that of the
inverse $2$-body Hopfield model, we see that the $eR$ and $gR$ phases of the
inverse $p$-body model with $p\geq 3$ are respectively analogous to the $eR$
and $sR$ (signal Retrieval) phases presented in [38]. One of the key
differences between $p=2$ and $p\geq 3$ is that the paramagnetic to signal
retrieval phase transition of the $p$-body model is second order for $p=2$ but
first order for $p\geq 3$. On the one hand, the second order phase transition
of $p=2$ indicates that its paramagnetic fixed point is never locally stable
and sets an unambiguous boundary between the $sR$ phase where $\xi^{*}$ can be
recovered starting from any initial conditions and the paramagnetic phase
where pattern retrieval is impossible [55]. On the other hand, the first order
phase transition of $p\geq 3$ allows the retrieval and paramagnetic regimes to
coexist. The $lR$ phase is locally stable precisely because it coexists with
the paramagnetic phase and has a lower free entropy. Meanwhile, the $gR$ phase
also coexists with the paramagnetic phase, but has a larger free entropy. In
the presence of phase coexistence, an algorithm trying to retrieve $\xi^{*}$
starting from random initial conditions can get stuck in the paramagnetic
phase instead. In fact, it has been conjectured that there is no algorithm
with random initial conditions that can find such a ferromagnetic fixed point
in a tractable amount of time [55, 56]. That kind of metastable region was
thus given the name hard phase [55, 57]. In summary, we expect that $p\geq 3$
models in the $gR$ phase can only recover partially corrupted patterns whereas
$p=2$ can recover them entirely.

Figure 4: Monte-Carlo simulations of the $p=3$ inverse model compared against
RS saddle-point solutions. The $lR$ phase is included on the left and central
plots, but not on the right one. The left plot has $\varepsilon=0$, and the
two other ones have a handpicked $\varepsilon$ such that the simulations are
initalized near the saddle-point solutions. The dots are simulation data at a
few values of $\alpha$, and the lines are slices of the saddle-point solutions
at the same $\alpha$. The teacher generates $M=\frac{\alpha N^{p-1}}{p!}$
examples $\sigma^{a}$ with $N=512$ components each, and the simulation results
are then averaged over $L=100$ student patterns. The simulation data is
sometimes systematically shifted up with respect to the saddle-point solution.
This difference is notably visible on the central plot, right after the fall
from $eR$ to $gR$ when $\alpha=3$.

Fig. (4) shows results from Monte Carlo simulations with $p=3$, where $L$
replicas of the student pattern $\\{\xi^{b}\\}_{b=1}^{L}$ are initialized to
the teacher pattern $\xi^{*}$ corrupted by some Rademacher noise
$\varepsilon$. In other words, the initial values of $\xi^{b}_{i}$ are sampled
from
$\left(1-\varepsilon\right)\delta\left(\xi_{i}-\xi^{*}_{i}\right)+\frac{\varepsilon}{2}\left[\delta\left(\xi_{i}+1\right)+\delta\left(\xi_{i}-1\right)\right]$
with $\varepsilon\in[0,1]$. The value of $\varepsilon$ is tuned so that the
simulations start relatively close to the saddle-point solutions. As explained
previously, $gR$ is a hard phase, so this initialization is necessary to make
$\xi^{b}$ converge to $gR$ in a reasonable amount of time. Additionally, it is
also used to make $\xi^{b}$ converge to the $lR$ phase rather than the $P$
phase when desired. Once the simulations are over, the overlaps are averaged
over all $L$ replicas. If we fix $\varepsilon=0$, then the simulations
generally converge to the $lR$ phase when it is a fixed point. If instead we
initialize them to the saddle-point solutions by handpicking $\varepsilon$,
then they stay near the initial overlaps. In either case, the simulations
converge to $eR$ when it is globally stable. Some simulation data points might
be systematically shifted up with respect to the saddle-point solutions.
However, this difference decreases with the system size $N$, so finite size
effects seem sufficient to explain it (see Fig. 9 in Appendix F). Overall, the
Monte-Carlo simulations are in very good agreement with the $p=3$ overlap
landscape obtained by solving the saddle-point equations numerically.

###  4.3 Inference temperature vs dataset noise

In the two next Sections, we will discuss the phase diagram when the student
is only partially informed about the teacher generative model, i.e. when the
Nishimori conditions do not hold. We start with the case where $p=p^{*}$ but
$\beta\neq\beta^{*}$, i.e. the inference temperature $T$ is different from the
dataset noise $T^{*}$. As we argued in Section 3, the student accurately
retrieves $\xi^{*}$ when $T^{*}<T_{\text{crit}}$. On the other hand, we must
solve the saddle-points equations (see Eqs. 3) to study
$T^{*}>T_{\text{crit}}$.

We show the phase diagram of this region on Fig. (5). At high inference
temperature $T$, the situation is similar to Fig. (2): retrieval is possible
if the data load $\alpha$ is sufficiently large, but the paramagnetic phase is
always locally stable. The situation is different when the inference
temperature is low. In that case, there are two phases that we did not see for
$\beta=\beta^{*}$: the inaccurate $eR$ phase and the $SG$ phase. When $\alpha$
is relatively small, the student falls in the inaccurate $eR$ phase. In this
regime, it has finite overlap with one of the noisy examples and cannot
retrieve the signal $\xi^{*}$. When $\alpha$ is larger, the interference among
the noisy examples prevents the student to be aligned with them. In this
regime, the $SG$ phase, the student locally converge to spurious patterns that
are uncorrelated with the signal.

Figure 5: RS phase diagrams of inverse models with $p^{*}=p$ and fixed
$\beta^{*}$. The top and bottom rows of plots respectively have $p^{*}=p=3$
and $p^{*}=p=4$. In the same way, the left, central and right columns
correspond to $T^{*}=1.5$, $T^{*}=1.6$ and $T^{*}=1.7$. Accurate pattern
retrieval is not possible in the paramagnetic phase ($P$), in the spin-glass
phase ($SG$) or in the example retrieval phase ($eR$), but it is possible in
the local retrieval phase ($lR$) and in the global retrieval phase ($gR$). The
ferromagnetic fixed point corresponding to accurate pattern retrieval is
globally stable in the $gR$ phase, but locally stable in the $lR$ phase.
Conversely, the $SG$ fixed point is always locally stable and leads the
student to a frozen spurious signal. The white dashed line indicates the
Nishimori line $\beta^{*}=\beta$. The black dashed lined is the $gR$ phase
boundary on the Nishimori line. As explained in Section 4.2, we expect it to
overlap the exact $SG$ phase transition.

Accurate pattern retrieval is only possible in the $lR$ and $gR$ phases where
$\alpha$ is so large that the student can gather enough information from the
dataset to become very close to $\xi^{*}$. The phase diagrams indicate that
pattern retrieval is optimal on the Nishimori line in the sense that
$\beta=\beta^{*}$ is the inverse temperature where the student needs the least
examples to recover $\xi^{*}$. In other words, the student’s performance is
non-monotonic in $T$ and peaks at $T=T^{*}$. These properties were also
observed in the teacher-student setting of the $p=2$ Hopfield network [38].

Contrary to what one would expect to see on the exact phase diagram [45, 46],
the Nishimori line $T=T^{*}$ does not to cross a triple point on the RS phase
diagram. The issue is that the RS phase diagram is not exact outside of the
Nishimori line. In particular, the $SG$ phase boundary is not exact. Outside
of the retrieval regime, the free entropy of the inverse model is the same as
the direct model. Since the transition towards $gR$ of the inverse model on
the Nishimori line overlaps the exact $P$-$SG$ transition of the direct model
(see Section 4.2), we deduce that it must also overlap the exact $P$-$SG$
transition of the inverse model outside of the $gR$ phase. Plotting it on the
RS phase diagrams, we see that it indeed crosses the Nishimori line and the
$gR$ phase boundary at the same point, which therefore becomes a triple point,
as expected.

###  4.4 Interaction order and noise tolerance

So far, we assumed that the student is informed about the interaction order
used by the teacher, i.e. $p=p^{*}$. In this Section, we investigate the role
of the student’s choice of $p$ when the task is to learn from a dataset
sampled by a $2$-body Hopfield network, i.e. $p^{*}=2$. We study two different
non trivial scalings regimes of $M$ and $\beta^{*}$ that make pattern
inference possible (see Appendix D).

We first consider a large noise scaling where
$\beta^{*}\sim\mathcal{O}\left(N^{2/p-1}\right)$ and
$M\sim\mathcal{O}\left(N^{p-1}\right)$, such that

$\alpha=\frac{Mp!}{N^{p-1}}\ \ \ \hbox{and}\ \
\lambda=\frac{\left[\beta^{*}\right]^{p/2}}{(p/2)!}N^{p/2-1}$

are finite. In this scaling, a $p\geq 3$ network requires
$\mathcal{O}\left(N^{p-2}\right)$ more training examples than a $p=2$ network
with finite load $\gamma=\frac{M}{N}$, but also has a higher tolerance to
teacher noise. For instance, a student with $p=4$ interactions is able to
retrieve the pattern of a teacher with
$T^{*}\sim\mathcal{O}\left(N^{1/2}\right)$ noise when it is shown enough
examples $M\sim\mathcal{O}\left(N^{3}\right)$ to be in the $gR$ phase (see
Fig. 6).

Figure 6: RS phase diagrams of inverse models with $p^{*}=2$ and $p=4$. The
left plot is for $\alpha=\frac{M\left(p/2+1\right)!}{N^{p/2}}$, and
$\beta^{*}=1-\frac{1}{\sqrt{2}}$ such that $\eta=1$ and the right plot is for
$\alpha=\frac{Mp!}{N^{p-1}}$ and $\beta^{*}=\sqrt{\frac{2\lambda}{N}}$ with
$\lambda=\beta$. Accurate pattern retrieval is not possible in the
paramagnetic phase ($P$) or in the example retrieval phase ($eR$), but it is
possible in the local retrieval phase ($lR$) and in the global retrieval phase
($gR$). The ferromagnetic fixed point corresponding to accurate pattern
retrieval is globally stable in the $gR$ phase, but locally stable in the $lR$
phase. The black dashed lines mark the metastable continuation of the $eR$,
$lR$ and $gR$ phase boundaries through neighboring phases with a larger free
entropy. The paramagnetic total entropy becomes negative below the white
dashed line drawn on the right plot. However, the paramagnetic phase is no
longer globally stable at that temperature.

$\mathcal{O}\left(N^{1/2}\right)$ noise tolerance was also observed in the
$p=4$ direct model, where it is a consequence of the redundancy stemming from
storing $\mathcal{O}\left(N\right)$ memories rather than the
$\mathcal{O}\left(N^{3}\right)$ needed to saturate the storage capacity [58].

Figure 7: Monte-Carlo simulations (dashed lines) and RS saddle-point solutions
(full lines) of the inverse model in the large-noise scaling with $p^{*}=2$
and $p=4$. The teacher generates $M=\frac{\alpha N^{p-1}}{p!}$ examples
$\sigma^{a}$ with $N=256$ components each, and the simulation results are then
averaged over $L=100$ student patterns. The student patterns are all
initialized to $\xi^{*}$.

Our $p=4$ inverse model exploits a different kind of redundancy by learning
from $\mathcal{O}\left(N^{3}\right)$ examples whereas $p=2$ only needs
$\mathcal{O}\left(N\right)$. In other terms, both storing extensively less
memories than the maximum allowed amount and generating extensively more
examples than the minimum required amount provide enough redundancy to recover
a pattern muddled in an extensive amount of noise. In both cases, there is an
$\mathcal{O}\left(N^{2}\right)$ gap between the number of patterns used in the
noise-tolerant and noise-susceptible regimes. Going beyond $p=4$, the inverse
model has $\mathcal{O}\left(N^{1-2/p}\right)$ noise tolerance as a function of
$p$. In particular, our theory predicts that the tolerance saturates at
$T^{*}\sim\mathcal{O}\left(N\right)$ as $p\rightarrow\infty$, but at the cost
of using an intractable number of examples. This behavior is different from
the $\mathcal{O}\left(N^{1/2-p/4}\right)$ tolerance of the direct $p$-body
model in the noisy-learning regime studied in [59]. In other terms, the
dataset noise that we are facing is of a different nature than the learning
noise of [59]. In any case, it is interesting that both the direct and inverse
models are able to tolerate an extensive amount of noise. Overall, our results
suggest that it could be advantageous to use a student network with a
relatively large $p$ to learn from a large but noisy dataset when the $p^{*}$
of the teacher generative model is unknown.

An unavoidable drawback of large teacher noise is that it always lead to
uncorrelated examples, which makes accurate example retrieval impossible.
Instead, it is replaced by the inaccurate example retrieval phase where the
student has finite overlap $m$ with a noisy example generated by the teacher
but no overlap with the signal (see Fig. 6). Depending on $T$ and $\alpha$,
this phase can be either globally stable or locally stable. For the sake of
clarity, we plot only the globally stable phase on our phase diagram in Fig.
(6). The locally stable phase is arguably less important to plot because it is
identical to the locally stable ferromagnetic phase previously reported in the
direct model when assuming replica symmetry (see [33] and Fig. 1).

Given $m=0$, the free entropy of the inverse model with $p\geq 3$, $p^{*}=2$
and $\beta=\lambda$ is the same as on the Nishimori line (see Eq. 3 and
Appendix D). As a direct consequence, the total entropy is positive outside of
the $eR$ phase (see Fig. 6). Additionally, the $p^{*}=2$, $p\geq 3$ phase
diagrams with $\beta\neq\lambda$ are identical to the $p=p^{*}$ phase diagrams
with $\beta\neq\beta^{*}$, which suggests that $\beta=\lambda$ is optimal for
$p^{*}=2$, $p\geq 3$ in the same sense as $\beta=\beta^{*}$ is optimal for
$p=p^{*}$ (see Fig. 5). Monte-Carlo simulations confirm that a student with
$p\geq 3$ is able to retrieve the pattern of a teacher with $p=2$ and
$T^{*}\sim\mathcal{O}\left(N^{1/2}\right)$ (see Fig. 7). However, the $lR$
phase transition is at a higher $T$ in the simulations than on the
$\beta=\lambda$ RS phase diagram (see Fig. 5), which means that RSB is
necessary to describe it accurately. One could check where replica symmetry
holds by evaluating the stability of the RS saddle point throughout the phase
diagram.

We also consider a different scaling regime where
$\beta^{*}\sim\mathcal{O}\left(1\right)$ and
$M\sim\mathcal{O}\left(N^{p/2}\right)$, such that

$\alpha=\frac{M(p/2+1)!}{N^{p/2}}$

is finite. In this finite-noise scaling, $p\geq 3$ requires
$\mathcal{O}\left(N^{p/2-1}\right)$ more training examples than $p=2$, which
is a lot less than the first scaling. For instance, a student with $p=4$ needs
$\mathcal{O}\left(N^{2}\right)$ examples to retrieve $\xi^{*}$. As before, the
phase transitions are all first order, the overlap $q^{*}$ stays high
throughout the $gR$ and $lR$ phase of $p=4$ and $gR$ is a hard phase. The
saddle-point equations (see Eqs. 3) are free from the pattern interference
term $\sqrt{\alpha r}x$ present in their $p^{*}=p$ counterparts (see Eqs. 3)
until $\beta^{*}$ becomes so small that is approaches
$\mathcal{O}\left(N^{2/p-1}\right)$. Therefore, contrary to $p^{*}=p=2$, the
network is never in the $SG$ phase. Practically, it means that $p\geq 3$ gives
more freedom than $p=2$ for tuning $\beta$ and $\alpha$. The only remaining
restriction is that choosing $\alpha$ and $T$ too small puts the network into
the inaccurate $eR$ phase resulting from the $kz$ term (see Fig. 6). The
saddle point equations can be derived without the RS ansatz because they do
not involve $q$ and $r$. Consequently, we expect them to yield an exact
solution. Like on the Nishimori line, the total entropy of the paramagnetic
phase is always positive, which is consistent with the solution being exact.

###  4.5 Robustness against adversarial attacks

Inverse models with $p^{*}=2$ and $p\geq 3$ offer an opportunity to study
adversarial attacks in a simple setting because their phase diagrams have
regions where the signal retrieval phases ($gR$ and $lR$) overlap with the
inaccurate $eR$ phase. Recall that, in the $lR$ phase, a noisy student pattern
$\xi$ either converges to $\xi^{*}$ or falls in the paramagnetic phase,
depending on the amount of noise that $\xi$ contains initially. The quantity
of noise needed to prevent pattern retrieval becomes smaller as one approaches
the $lR$ to $P$ phase transition and the basin of attraction of $lR$ shrinks.
Similarly, in the region of inaccurate $eR$ where signal retrieval is
metastable, patterns $\xi$ that are corrupted by replacing some of their
entries $\xi_{i}$ by the components $\sigma^{a}_{i}$ of an example
$\sigma^{a}$ may converge to $\sigma^{a}$ when enough entries are replaced.
The fraction $\varepsilon$ of entries that need to be replaced becomes smaller
as the basin of attraction of inaccurate $eR$ expands and overtakes that of
signal retrieval. In practice, an adversary can use this strategy to trick the
student into converging to a pattern other than $\xi^{*}$. This scenario is
similar to an adversarial attack targeting the input of Krotov’s dense
Hopfield network model because the student pattern $\xi$ plays a similar role
in the inverse model as the test data in Krotov’s dense Hopfield networks (see
Fig. 3, Section 4.2 and Appendix A). In that analogy, the examples $\sigma$
are acting like the neural network weights rather than taking the role of the
training data.

We will now investigate what values of the perturbation size $\varepsilon$ are
a threat by deriving a formula for the largest $\varepsilon$ such that the
student converges to the signal at zero temperature. This largest
$\varepsilon$ will be denoted $\varepsilon^{*}$, and we expect it to be a good
measure of adversarial robustness. The saddle-point equations with $T=0$
indicate that the student converges to one of the signal retrieval phases if
and only if $k<\eta\alpha r^{*}$ (see Eqs. 3). Sampling the initial conditions
of $\xi_{i}$ from
$\left(1-\varepsilon\right)\delta\left(\xi_{i}-\xi^{*}_{i}\right)+\varepsilon\
\delta\left(\xi_{i}-\sigma^{a}_{i}\right)$ with
$\varepsilon\in\left[0,1\right]$, we get

$\displaystyle r^{*}$
$\displaystyle=p\left[\frac{1}{N}\sum_{i=1}^{\left(1-\varepsilon\right)N}\xi^{*}_{i}\xi^{*}_{i}+\frac{1}{N}\sum_{i=1}^{\varepsilon
N}\xi^{*}_{i}\sigma^{a}_{i}\right]^{p-1},$ $\displaystyle\quad k$
$\displaystyle=p\left[\frac{1}{N}\sum_{i=1}^{\left(1-\varepsilon\right)N}\xi^{*}_{i}\sigma^{a}_{i}+\frac{1}{N}\sum_{i=1}^{\varepsilon
N}\sigma^{a}_{i}\sigma^{a}_{i}\right]^{p-1}.$

By the law of large numbers, $\frac{1}{\varepsilon N}\sum_{i=1}^{\varepsilon
N}\xi^{*}_{i}\sigma^{a}_{i}$ and
$\frac{1}{\left(1-\varepsilon\right)N}\sum_{i=1}^{\left(1-\varepsilon\right)N}\xi^{*}_{i}\sigma^{a}_{i}$
are both typically close to
$m^{*}=\frac{1}{N}\sum_{i}^{N}\xi^{*}_{i}\sigma^{a}_{i}\approx 0$ as
$N\rightarrow\infty$. If we take $\sigma^{a}$ to be a typical example, then
$r^{*}$ and $k$ reduce to

$\displaystyle r^{*}$ $\displaystyle\approx p\left(1-\varepsilon\right)^{p-1}$
$\displaystyle k$ $\displaystyle\approx p\varepsilon^{p-1}.$

Substituting these expressions back in $k<\eta\alpha r^{*}$ yields

$\displaystyle\varepsilon^{p-1}$
$\displaystyle<\eta\alpha\left(1-\varepsilon\right)^{p-1}$
$\displaystyle\varepsilon$
$\displaystyle<\frac{\left[\eta\alpha\right]^{\frac{1}{p-1}}}{\left[\eta\alpha\right]^{\frac{1}{p-1}}+1}.$

In other terms, the inverse model with $p^{*}=2$ and even $p\geq 3$ is
resistant to adversarial attacks of size
$\varepsilon^{*}=\frac{\left[\eta\alpha\right]^{\frac{1}{p-1}}}{\left[\eta\alpha\right]^{\frac{1}{p-1}}+1}$
and smaller. For $p=4$, $\varepsilon^{*}$ is in good agreement with Monte-
Carlo simulations of the inverse model corrupted by a typical example (see
Fig. 8). This comparison is good evidence that our solution of the finite-
noise scaling is indeed exact. Additionally, $\varepsilon^{*}$ is a decent
approximation of empirical robustness even when the inverse model is corrupted
by the example that has the largest overlap with $\xi^{*}$. Just like
adversarial attacks targeting more complicated neural networks [40, 41], our
example-based attack can be hard to detect at low $\varepsilon$ because a few
adversarially perturbed entries $\xi_{i}$ do not look very different from a
low amount of meaningless noise.

Figure 8: Monte-Carlo simulations of the overlap $q^{*}$ as a function of
$\alpha$ and $\varepsilon$ in the inverse model with $p^{*}=2$,
$\beta^{*}=1-\frac{1}{\sqrt{2}}$, $p=4$, $\beta=\infty$ and $N=1024$. The
simulation results are averaged over $L=100$ student patterns. On the left
plot, the inverse model is corrupted by an example $\sigma^{a}$ that has a
small overlap with $\xi^{*}$ in absolute value. On the right plot, it is
corrupted by the example that has the largest overlap with $\xi^{*}$. The
black line $\varepsilon^{*}=\frac{\alpha^{1/3}}{\alpha^{1/3}+1}$ is our
analytical formula for the largest adversarial perturbation $\varepsilon$ such
that the student retrieves $\xi^{*}$ rather than the example $\sigma^{a}$.

Interestingly, $\varepsilon^{*}$ grows monotonically with $\alpha$, which is
consistent with the common observation that larger neural networks are also
more adversarially robust [43, 60, 61, 62, 63, 64, 65]. At first glance, this
effect can be counter-intuitive because adversarial vulnerability looks like a
form of overfitting [42]. In our model, however, all examples work together to
stabilize the $lR$ phase, and the best way to push the student into the $eR$
phase is to perturb it with a single example. Therefore, it is not surprising
that increasing $\alpha$ makes the student more robust. We recall that the
examples $\sigma$ are a feature-based representation of $\xi^{*}$.
Interestingly, it means that the underlying mechanism of our example-based
attack is conceptually similar to gradient-based attacks targeting many common
types of neural networks [42]. In fact, gradient-based attacks find features
stored in neural network weights and add them to the data in order to fool the
network [42, 66, 67, 68]. It would be interesting to investigate, both
empirically and theoretically, if only a small number of weights are involved
in constructing these adversarial attacks. If it is the case, it could explain
why larger neural networks are often more robust. In general, we expect this
kind of one-example attack to be possible in any region of signal retrieval
that overlaps with the inaccurate $eR$ phase. Using $p\neq p^{*}$ may not be a
necessary ingredient of adversarial vulnerability in more general models with
other sources of mismatch, but in our case it ensures that the signal
retrieval phases intersect the inaccurate $eR$ phase. Conversely, the accurate
$eR$ phase is by definition robust to adversarial attacks since retrieving an
example $\sigma^{a}$ is the same as recovering $\xi^{*}$. This distinction
clarifies why the dense Hopfield networks designed by Krotov are adversarially
robust in the prototype phase despite being adversarially vulnerable in the
feature phase. In fact, Krotov observed that adversarial attacks are
unsuccessful in the prototype phase specifically because they retrieve stored
examples that are semantically meaningful [37]. In summary, our model yields
two main results concerning adversarial examples. First of all, it suggests a
reason why large feature-based neural networks are more adversarially robust
than smaller ones. Second of all, it clarifies why dense Hopfield networks are
much more robust in the prototype phase than in the feature phase.

##  5 Conclusion

In this work, we derive the exact phase diagram of the $p$-dense networks in
the teacher-student setting [16, 17, 30, 38]. On the Nishimori line, we find
an example retrieval phase ($eR$) and a global retrieval phase ($gR$)
reminiscent of the prototype and feature regimes observed empirically in dense
Hopfield networks [26]. We show that the phase transition towards $gR$ of the
inverse model overlaps the paramagnetic to spin-glass ($P$-$SG$) transition of
the direct model, which allows us to locate the $P$-$SG$ transition much more
precisely than before [30, 33]. On the other hand, we discover that inverse
models outside of the Nishimori line are able to resist an extensive amount of
noise. In fact, a student with $p\geq 3$ is able to learn from a teacher with
$p^{*}=2$ even when the teacher’s inverse temperature $\beta^{*}$ is as low as
$\mathcal{O}\left(N^{2/p-1}\right)$. Moreover, such a student is immune to
pattern interference until $\beta^{*}$ reaches
$\mathcal{O}\left(N^{2/p-1}\right)$. In this setting, we derive a formula
measuring the adversarial robustness of the student with $p\geq 3$ and $T=0$.
We then use this formula to describe how making a neural network larger can
potentially increase its robustness to adversarial attacks constructed with
only a few learned weights [43, 60, 61, 62, 63, 64, 65]. Our model also
clarifies why the prototype phase of dense Hopfield networks is adversarially
robust [37]. We compare our key results against Monte-Carlo simulations.

Dense networks with exponential interactions have been argued to be the
$p\rightarrow\infty$ limit of the $p$-body models [69]. It would be
interesting to see if they can achieve $\mathcal{O}\left(N\right)$ noise
tolerance at the cost of an exponential number of training examples. More
generally, studying exponential models in the teacher-student setting would be
an interesting extension of this work and could be used to complement existing
studies of the direct model [69, 70]. A caveat of our model is that the
teacher has only one pattern. In fact, we would need to use a teacher with at
least two patterns to describe more completely the kind of adversarial attack
aiming to misclassify data. It should be possible to study this kind of
teacher by using an approach similar to [71]. On the practical side, we
highlight the untapped benefits of using $p$-body models to either resist an
extensive amount of noise in the feature phase or improve adversarial
robustness in the prototype phase. Overall, we stress that further
investigations of dense Hopfield networks could unlock their true potential.

## Acknowledgements

This work was partially supported by project SERICS (PE00000014) under the MUR
National Recovery and Resilience Plan funded by the European Union -
NextGenerationEU. The work was also supported by the project PRIN22TANTARI
”Statistical Mechanics of Learning Machines: from algorithmic and information-
theoretical limits to new biologically inspired paradigms” 20229T9EAT – CUP
J53D23003640001. DT also acknowledges GNFM-Indam.

## References

  * [1] J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities.” _Proceedings of the national academy of sciences_ , vol. 79, no. 8, pp. 2554–2558, 1982.
  * [2] D. J. Amit, H. Gutfreund, and H. Sompolinsky, “Spin-glass models of neural networks,” _Physical Review A_ , vol. 32, no. 2, p. 1007, 1985.
  * [3] D. O. Hebb, _The organization of behavior: A neuropsychological theory_. Psychology press, 2005.
  * [4] T. M. Cover, “Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition,” _IEEE transactions on electronic computers_ , no. 3, pp. 326–334, 1965.
  * [5] D. J. Amit, H. Gutfreund, and H. Sompolinsky, “Storing infinite numbers of patterns in a spin-glass model of neural networks,” _Physical Review Letters_ , vol. 55, no. 14, p. 1530, 1985.
  * [6] E. Agliari, A. Barra, A. Galluzzi, F. Guerra, and F. Moauro, “Multitasking associative networks,” _Physical review letters_ , vol. 109, no. 26, p. 268101, 2012.
  * [7] E. Agliari, A. Annibale, A. Barra, A. Coolen, and D. Tantari, “Immune networks: multi-tasking capabilities at medium load,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 46, no. 33, p. 335101, 2013.
  * [8] ——, “Immune networks: multitasking capabilities near saturation,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 46, no. 41, p. 415003, 2013.
  * [9] P. Sollich, D. Tantari, A. Annibale, and A. Barra, “Extensive parallel processing on scale-free networks,” _Physical review letters_ , vol. 113, no. 23, p. 238106, 2014.
  * [10] E. Agliari, A. Annibale, A. Barra, A. C. Coolen, and D. Tantari, “Retrieving infinite numbers of patterns in a spin-glass model of immune networks,” _Europhysics Letters_ , vol. 117, no. 2, p. 28003, 2017.
  * [11] E. Agliari, A. Barra, A. Galluzzi, F. Guerra, D. Tantari, and F. Tavani, “Retrieval capabilities of hierarchical networks: From dyson to hopfield,” _Physical review letters_ , vol. 114, no. 2, p. 028103, 2015.
  * [12] E. Agliari, D. Migliozzi, and D. Tantari, “Non-convex multi-species hopfield models,” _Journal of Statistical Physics_ , vol. 172, no. 5, pp. 1247–1269, 2018.
  * [13] E. Agliari, A. Barra, A. Galluzzi, F. Guerra, D. Tantari, and F. Tavani, “Hierarchical neural networks perform both serial and parallel processing,” _Neural Networks_ , vol. 66, pp. 22–35, 2015.
  * [14] ——, “Metastable states in the hierarchical dyson model drive parallel processing in the hierarchical hopfield network,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 48, no. 1, p. 015001, 2014.
  * [15] ——, “Topological properties of hierarchical networks,” _Physical Review E_ , vol. 91, no. 6, p. 062807, 2015.
  * [16] A. Barra, G. Genovese, P. Sollich, and D. Tantari, “Phase transitions in restricted boltzmann machines with generic priors,” _Physical Review E_ , vol. 96, no. 4, p. 042156, 2017.
  * [17] ——, “Phase diagram of restricted boltzmann machines and generalized hopfield networks with arbitrary priors,” _Physical Review E_ , vol. 97, no. 2, p. 022310, 2018.
  * [18] A. Barra, P. Contucci, E. Mingione, and D. Tantari, “Multi-species mean field spin glasses. rigorous results,” in _Annales Henri Poincaré_ , vol. 16. Springer, 2015, pp. 691–708.
  * [19] E. Agliari, A. Barra, C. Longo, and D. Tantari, “Neural networks retrieving boolean patterns in a sea of gaussian ones,” _Journal of Statistical Physics_ , vol. 168, pp. 1085–1104, 2017.
  * [20] A. Barra, G. Genovese, F. Guerra, and D. Tantari, “How glassy are neural networks?” _Journal of Statistical Mechanics: Theory and Experiment_ , vol. 2012, no. 07, p. P07009, 2012.
  * [21] G. Genovese and D. Tantari, “Legendre equivalences of spherical boltzmann machines,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 53, no. 9, p. 094001, 2020.
  * [22] J. Rocchi, D. Saad, and D. Tantari, “High storage capacity in the hopfield model with auto-interactions—stability analysis,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 50, no. 46, p. 465001, 2017.
  * [23] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve _et al._ , “Hopfield networks is all you need,” _arXiv preprint arXiv:2008.02217_ , 2020.
  * [24] M. Widrich, B. Schäfl, M. Pavlović, H. Ramsauer, L. Gruber, M. Holzleitner, J. Brandstetter, G. K. Sandve, V. Greiff, S. Hochreiter _et al._ , “Modern hopfield networks and attention for immune repertoire classification,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 18 832–18 845, 2020.
  * [25] D. Krotov and J. Hopfield, “Large associative memory problem in neurobiology and machine learning,” _arXiv preprint arXiv:2008.06996_ , 2020.
  * [26] D. Krotov and J. J. Hopfield, “Dense associative memory for pattern recognition,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [27] H. Chen, Y. Lee, G. Sun, H. Lee, T. Maxwell, and C. L. Giles, “High order correlation model for associative memory,” in _AIP Conference Proceedings_ , vol. 151, no. 1. American Institute of Physics, 1986, pp. 86–99.
  * [28] D. Psaltis and C. H. Park, “Nonlinear discriminant functions and associative memories,” in _AIP conference Proceedings_ , vol. 151, no. 1. American Institute of Physics, 1986, pp. 370–375.
  * [29] P. Baldi and S. S. Venkatesh, “Number of stable points for spin-glasses and neural networks of higher orders,” _Physical Review Letters_ , vol. 58, no. 9, p. 913, 1987.
  * [30] E. Gardner, “Multiconnected neural network models,” _Journal of Physics A: Mathematical and General_ , vol. 20, no. 11, p. 3453, 1987.
  * [31] L. F. Abbott and Y. Arian, “Storage capacity of generalized networks,” _Physical review A_ , vol. 36, no. 10, p. 5091, 1987.
  * [32] D. Horn and M. Usher, “Capacities of multiconnected memory models,” _Journal de Physique_ , vol. 49, no. 3, pp. 389–395, 1988.
  * [33] L. Albanese, F. Alemanno, A. Alessandrelli, and A. Barra, “Replica symmetry breaking in dense hebbian neural networks,” _Journal of Statistical Physics_ , vol. 189, no. 2, p. 24, 2022.
  * [34] B. Hoover, Y. Liang, B. Pham, R. Panda, H. Strobelt, D. H. Chau, M. J. Zaki, and D. Krotov, “Energy transformer,” _arXiv preprint arXiv:2302.07253_ , 2023.
  * [35] B. Hoover, H. Strobelt, D. Krotov, J. Hoffman, Z. Kira, and D. H. Chau, “Memory in plain sight: A survey of the uncanny resemblances between diffusion models and associative memories,” _arXiv preprint arXiv:2309.16750_ , 2023.
  * [36] L. Ambrogioni, “In search of dispersed memories: Generative diffusion models are associative memory networks,” _arXiv preprint arXiv:2309.17290_ , 2023\.
  * [37] D. Krotov and J. Hopfield, “Dense associative memory is robust to adversarial inputs,” _Neural computation_ , vol. 30, no. 12, pp. 3151–3167, 2018.
  * [38] F. Alemanno, L. Camanzi, G. Manzan, and D. Tantari, “Hopfield model with planted patterns: A teacher-student self-supervised learning model,” _Applied Mathematics and Computation_ , vol. 458, p. 128253, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0096300323004228
  * [39] A. Decelle, S. Hwang, J. Rocchi, and D. Tantari, “Inverse problems for structured datasets using parallel tap equations and restricted boltzmann machines,” _Scientific Reports_ , vol. 11, no. 1, p. 19990, 2021.
  * [40] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against machine learning at test time,” in _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13_. Springer, 2013, pp. 387–402.
  * [41] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [42] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _stat_ , vol. 1050, p. 20, 2015.
  * [43] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations_ , 2018.
  * [44] A. Muhammad and S.-H. Bae, “A survey on efficient methods for adversarial robustness,” _IEEE Access_ , vol. 10, pp. 118 815–118 830, 2022.
  * [45] H. Nishimori, “Exact results and critical properties of the ising model with competing interactions,” _Journal of Physics C: Solid State Physics_ , vol. 13, no. 21, p. 4071, 1980.
  * [46] ——, _Statistical physics of spin glasses and information processing: an introduction_. Clarendon Press, 2001, no. 111.
  * [47] P. Contucci, C. Giardina, and H. Nishimori, “Spin glass identities and the nishimori line,” in _Spin Glasses: Statics and Dynamics: Summer School, Paris 2007_. Springer, 2009, pp. 103–121.
  * [48] Y. Iba, “The nishimori line and bayesian statistics,” _Journal of Physics A: Mathematical and General_ , vol. 32, no. 21, p. 3875, 1999.
  * [49] P. Charbonneau, “From the replica trick to the replica symmetry breaking technique,” _arXiv preprint arXiv:2211.01802_ , 2022.
  * [50] D. Sherrington and S. Kirkpatrick, “Solvable model of a spin-glass,” _Physical review letters_ , vol. 35, no. 26, p. 1792, 1975.
  * [51] G. G. Roussas, “Contiguity of probability measures,” _(No Title)_ , 1972.
  * [52] D. Achlioptas and A. Coja-Oghlan, “Algorithmic barriers from phase transitions,” in _2008 49th Annual IEEE Symposium on Foundations of Computer Science_. IEEE, 2008, pp. 793–802.
  * [53] F. Krzakala and L. Zdeborová, “Hiding quiet solutions in random constraint satisfaction problems,” _Physical review letters_ , vol. 102, no. 23, p. 238701, 2009.
  * [54] L. Zdeborová and F. Krzakala, “Quiet planting in the locked constraint satisfaction problems,” _SIAM Journal on Discrete Mathematics_ , vol. 25, no. 2, pp. 750–770, 2011.
  * [55] ——, “Statistical physics of inference: Thresholds and algorithms,” _Advances in Physics_ , vol. 65, no. 5, pp. 453–552, 2016.
  * [56] F. Antenucci, S. Franz, P. Urbani, and L. Zdeborová, “Glassy nature of the hard phase in inference problems,” _Physical Review X_ , vol. 9, no. 1, p. 011020, 2019.
  * [57] L. Zdeborová and F. Krzakala, “Phase transitions in the coloring of random graphs,” _Physical Review E_ , vol. 76, no. 3, p. 031131, 2007.
  * [58] E. Agliari, F. Alemanno, A. Barra, M. Centonze, and A. Fachechi, “Neural networks with a redundant representation: Detecting the undetectable,” _Physical review letters_ , vol. 124, no. 2, p. 028301, 2020.
  * [59] E. Agliari and G. De Marzo, “Tolerance versus synaptic noise in dense associative memories,” _The European Physical Journal Plus_ , vol. 135, no. 11, pp. 1–22, 2020.
  * [60] S. Gowal, C. Qin, J. Uesato, T. Mann, and P. Kohli, “Uncovering the limits of adversarial training against norm-bounded adversarial examples,” _arXiv preprint arXiv:2010.03593_ , 2020.
  * [61] H. Huang, Y. Wang, S. Erfani, Q. Gu, J. Bailey, and X. Ma, “Exploring architectural ingredients of adversarially robust deep neural networks,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 5545–5559, 2021.
  * [62] S. Bubeck, Y. Li, and D. M. Nagaraj, “A law of robustness for two-layers neural networks,” in _Proceedings of Thirty Fourth Conference on Learning Theory_ , ser. Proceedings of Machine Learning Research, M. Belkin and S. Kpotufe, Eds., vol. 134. PMLR, 15–19 Aug 2021, pp. 804–820. [Online]. Available: https://proceedings.mlr.press/v134/bubeck21a.html
  * [63] S. Bubeck and M. Sellke, “A universal law of robustness via isoperimetry,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 28 811–28 822, 2021.
  * [64] J. Puigcerver, R. Jenatton, C. Riquelme, P. Awasthi, and S. Bhojanapalli, “On the adversarial robustness of mixture of experts,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 9660–9671, 2022.
  * [65] A. H. Ribeiro and T. B. Schön, “Overparameterized linear regression under adversarial attacks,” _IEEE Transactions on Signal Processing_ , vol. 71, pp. 601–614, 2023.
  * [66] S. Jetley, N. Lord, and P. Torr, “With friends like these, who needs adversaries?” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [67] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [68] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in _International Conference on Learning Representations_ , no. 2019, 2019.
  * [69] M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and F. Vermet, “On a model of associative memory with huge storage capacity,” _Journal of Statistical Physics_ , vol. 168, pp. 288–299, 2017.
  * [70] C. Lucibello and M. Mézard, “The exponential capacity of dense associative memories,” _arXiv preprint arXiv:2304.14964_ , 2023.
  * [71] T. Hou, K. M. Wong, and H. Huang, “Minimal model of permutation symmetry in unsupervised learning,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 52, no. 41, p. 414001, 2019.
  * [72] N. E. Boukacem, A. Leary, R. Thériault, F. Gottlieb, M. Mani, and P. François, “A waddington landscape for prototype learning in generalized hopfield networks,” _arXiv preprint arXiv:2312.03012_ , 2023\.

##  Appendix A Gardner’s Hamiltonian vs Krotov’s Hamiltonian

Consider the generalized Hopfield Hamiltonian
$H\left[\sigma|\xi\right]=-\sum_{i_{1}<...<i_{p}=1}^{N}J_{i_{1}...i_{p}}\sigma_{i_{1}}...\sigma_{i_{p}}$
with $p$-body interactions
$J_{i_{1}...i_{p}}=\frac{p!}{N^{p-1}}\sum_{\mu=1}^{M}\xi_{i_{1}}^{\mu}...\xi_{i_{p}}^{\mu}$
described by Gardner [30], where $M$ indicates the number of patterns
$\xi^{\mu}$ used to construct $J$, and $N$ denotes the number of components of
each pattern $\xi^{\mu}$ and example $\sigma$. In this Section, we will omit
$\xi$ in the argument of $H\left[\sigma|\xi\right]$ and write
$H\left[\sigma\right]$ instead for notational simplicity. Unless indicated
otherwise, we will assume a large number number of components $N\gg 1$ and
patterns $M\sim\mathcal{O}\left(N^{p-1}\right)$. We will start by comparing it
to the dense Hopfield network Hamiltonian
$\mathcal{H}\left[\sigma\right]=-\frac{1}{N^{p-1}}\sum_{\mu}\left(\sum_{i}\xi_{i}^{\mu}\sigma_{i}\right)^{p}$
studied by Krotov [26].

For that purpose, we rewrite $H$ in the form
$H\left[\sigma\right]=-\frac{1}{p!}\sum_{i_{1}\neq...\neq
i_{p}}J_{i_{1}...i_{p}}\sigma_{i_{1}}...\sigma_{i_{p}}$ by summing over all
permutations of $\\{i_{1}...i_{p}\\}$ in place of the restricted set
$i_{1}<...<i_{p}$ and compensating for double counting with the prefactor
$\frac{1}{p!}$. This manipulation leads to

$\displaystyle H\left[\sigma\right]$
$\displaystyle=-\frac{1}{p!}\sum_{i_{1}\neq...\neq
i_{p}}J_{i_{1}...i_{p}}\sigma_{i_{1}}...\sigma_{i_{p}}$
$\displaystyle=-\frac{1}{N^{p-1}}\sum_{\mu}\sum_{i_{1}\neq...\neq
i_{p}}\xi_{i_{1}}^{\mu}...\xi_{i_{p}}^{\mu}\sigma_{i_{1}}...\sigma_{i_{p}}.$

On the other hand, Krotov’s Hamiltonian may be rewritten

$\displaystyle\mathcal{H}\left[\sigma\right]$
$\displaystyle=-\frac{1}{N^{p-1}}\sum_{\mu}\left(\sum_{i}\xi_{i}^{\mu}\sigma_{i}\right)^{p}$
$\displaystyle=-\frac{1}{N^{p-1}}\sum_{\mu}\left(\sum_{i_{1}}\xi_{i_{1}}^{\mu}\sigma_{i_{1}}\right)...\left(\sum_{i_{p}}\xi_{i_{p}}^{\mu}\sigma_{i_{p}}\right)$
$\displaystyle=-\frac{1}{N^{p-1}}\sum_{\mu}\sum_{i_{1}...i_{p}}\xi_{i_{1}}^{\mu}...\xi_{i_{p}}^{\mu}\sigma_{i_{1}}...\sigma_{i_{p}},$

where the sum over $i_{1}...i_{p}$ includes both the set of indices
$i_{1}\neq...\neq i_{p}$ found in $H\left[\sigma\right]$ and other
configurations where some indices are equal. For example, the configuration
$i_{1}\neq...\neq i_{p-1}=i_{p}$ contains the fewest equal indices after
$i_{1}\neq...\neq i_{p}$. In other words, $\mathcal{H}\left[\sigma\right]$ can
be expressed as an expansion around $H\left[\sigma\right]$, and the two
Hamiltonians are equivalent when the normalized residuals
$\frac{\mathcal{H}\left[\sigma\right]-H\left[\sigma\right]}{N}$ vanish in the
limit of large $N$. In this study, we will encounter two cases which will
bring different results.

  1. 1

The Hamiltonians $\mathcal{H}\left[\sigma\right]$ and $H\left[\sigma\right]$
are dominated by a few closely packed configurations $\xi^{\mu}$ that have
finite overlap
$\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma_{i}\sim\mathcal{O}\left(1\right)$ with
$\sigma$. We say that they are aligned with $\sigma$.

  2. 2

The Hamiltonians $\mathcal{H}\left[\sigma\right]$ and $H\left[\sigma\right]$
are dominated by many spread out configurations $\xi^{\mu}$ that have
microscopic overlap
$\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma_{i}\sim\mathcal{O}\left(N^{-1/2}\right)$
with $\sigma$. We say that they are misaligned with $\sigma$

We use the expansion of $\mathcal{H}\left[\sigma\right]$ to discuss both the
aligned case and the misaligned case. We start by writing the
$i_{1}\neq...\neq i_{p}$ and $i_{1}\neq...\neq i_{p-1}=i_{p}$ terms
explicitly, which leads to the form:

$\displaystyle\mathcal{H}\left[\sigma\right]$
$\displaystyle=-\frac{1}{N^{p-1}}\sum_{\mu}\sum_{i_{1}\neq...\neq
i_{p}}\xi_{i_{1}}^{\mu}...\xi_{i_{p}}^{\mu}\sigma_{i_{1}}...\sigma_{i_{p}}$
$\displaystyle\quad-\frac{1}{2}\frac{p(p-1)}{N^{p-1}}\sum_{\mu}\sum_{i_{1}\neq...\neq
i_{p-1}}\xi_{i_{1}}^{\mu}...\left(\xi_{i_{p-1}}^{\mu}\right)^{2}\sigma_{i_{1}}...\left(\sigma_{i_{p-1}}\right)^{2}+...$

because there are $\binom{p}{2}=\frac{p(p-1)}{2}$ ways for the indices
$i_{p-1}$ and $i_{p}$ to be equal. This expression can be summarized by
$\mathcal{H}\left[\sigma\right]=H\left[\sigma\right]+H^{\prime}\left[\sigma\right]+...$,
where $H^{\prime}\left[\sigma\right]$ simplifies to

$\displaystyle H^{\prime}\left[\sigma\right]$
$\displaystyle=-\frac{1}{2}\frac{p(p-1)}{N^{p-1}}\sum_{\mu}\sum_{i_{1}\neq...\neq
i_{p-1}}\xi_{i_{1}}^{\mu}...\left(\xi_{i_{p-1}}^{\mu}\right)^{2}\sigma_{i_{1}}...\left(\sigma_{i_{p-1}}\right)^{2}$
$\displaystyle=-\frac{1}{2}\frac{p(p-1)}{N^{p-2}}\sum_{\mu}\sum_{i_{1}\neq...\neq
i_{p-2}}\xi_{i_{1}}^{\mu}...\xi_{i_{p-2}}^{\mu}\sigma_{i_{1}}...\sigma_{i_{p-2}}$
$\displaystyle=-\frac{1}{2}\frac{p!}{N^{p-2}}\sum_{\mu}\sum_{i_{1}<...<i_{p-2}}\xi_{i_{1}}^{\mu}...\xi_{i_{p-2}}^{\mu}\sigma_{i_{1}}...\sigma_{i_{p-2}}.$

In the aligned case, $H^{\prime}\left[\sigma\right]$ is
$\mathcal{O}\left(1\right)$ in $N$ because the sum over $i_{1}<...<i_{p-2}$ is
$\mathcal{O}\left(N^{p-2}\right)$. The terms implied by the ellipsis are even
smaller because their sums are resctricted by more equality constraints.
Therefore, the residuals
$\frac{\mathcal{H}\left[\sigma\right]-H\left[\sigma\right]}{N}$ vanish in the
limit of large $N$, and the two Hamiltonians are equivalent. Conversely, we
find that $\mathcal{H}\left[\sigma\right]$ and $H\left[\sigma\right]$ differ
from each other in the misaligned case (see Appendix B for more details).
Therefore, although the phases of $H\left[\sigma\right]$ that we obtain in
this study are qualitatively similar to the ones observed by Krotov [26, 37],
the phase diagram of $H\left[\sigma\right]$ must be compared against a
simulation of $H\left[\sigma\right]$ rather than
$\mathcal{H}\left[\sigma\right]$ in order to test our theory quantitatively.

To understand how to sample $\sigma$ in both models, consider a Monte-Carlo
simulation used to find the statistical equilibrium of a spin ensemble
$\sigma$ with Hamiltonian $G\left[\sigma\right]$. To be more specific, suppose
$\sigma$ is updated to a randomly selected new state $\sigma^{\prime}$ with
acceptance probability
$P_{i}=\frac{1}{1+\exp\left[\beta\left(G\left[\sigma^{\prime}\right]-G\left[\sigma\right]\right)\right]}$
for a large number of time-steps. This approach works well for
$G\left[\sigma\right]=\mathcal{H}\left[\sigma\right]$. However, in the case of
$H\left[\sigma\right]$, we find that the simulation only converges when we use
the local field
$h_{i}=\frac{p!}{N^{p-1}}\sum_{\mu}\xi^{\mu}_{i}\sum_{i_{1}<...<i_{p-1}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p-1}}\sigma_{i_{1}}...\sigma_{i_{p-1}}$
mentioned by Gardner [30] to approximate
$\frac{H\left[\sigma^{\prime}\right]-H\left[\sigma\right]}{2\sigma_{i}}$ at
large $N$. In other words, we iteratively flip randomly chosen spins
$\sigma_{i}$ with acceptance probability $P_{i}=\frac{1}{1+\exp\left(2\beta
h_{i}\sigma_{i}\right)}$ for a large number of time steps. For arbitrary $p$,
it is not obvious how to compute $h_{i}$ quickly as a sub-routine of the
Monte-Carlo simulation. However, we find that both $p=3$ and $p=4$ have
closed-formed expressions that are easy to evaluate numerically in an
efficient way. To be more precise,

  * •

$p=3$ leads to
$h_{i}=3\sum_{\mu}\xi^{\mu}_{i}\left[\left(\frac{1}{N}\sum_{j}\xi^{\mu}_{j}\sigma_{j}\right)^{2}-\frac{1}{N}\right]$,

  * •

and $p=4$ leads to
$h_{i}=4\sum_{\mu}\xi^{\mu}_{i}\left(\frac{1}{N}\sum_{j}\xi^{\mu}_{j}\sigma_{j}\right)\left[\left(\frac{1}{N}\sum_{j}\xi^{\mu}_{j}\sigma_{j}\right)^{2}-\frac{3}{N}\right]$.

For this reason and also because the number
$M\sim\mathcal{O}\left(N^{p-1}\right)$ of patterns $\xi^{\mu}$ used in a
Monte-Carlo simulations increases exponentially with $p$, we choose to
simulate only $p=3$ and $p=4$.

The output of the neural network model that Krotov designed for classification
of data is
$c_{j}=\tanh\left[\frac{1}{2}\beta\left(\mathcal{H}\left[\sigma^{\prime}\right]-\mathcal{H}\left[\sigma\right]\right)\right]\approx\tanh\left[\beta
p\sum_{\mu}\xi^{\mu}_{j}\left(\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma_{i}\right)^{p-1}\right]$.
We omit the linear rectifier present in the original paper [26] because the
overlaps $\frac{1}{N}\sum_{i}\xi_{i}^{\mu}\sigma_{i}$ are almost always
positive (see for example the Supplement of [72]). The predicted class is then
$j^{\prime}=\operatorname*{argmax}_{j}\left\\{c_{i}\right\\}$. Using
$1-P_{j}=\frac{1}{1+\exp\left[\beta\left(\mathcal{H}\left[\sigma^{\prime}\right]-\mathcal{H}\left[\sigma\right]\right)\right]}$
instead of $c_{j}$ does not change $j^{\prime}$ because $1-P_{j}$ and $c_{j}$
are related by $1-P_{j}=\frac{1}{2}\left[c_{j}+1\right]$. When we evaluate
$P_{i}$ using $H$ instead of $\mathcal{H}$, this relation does not always hold
exactly. Rather, it should be considered an approximation.

##  Appendix B Direct model cumulant expansions

In the direct model, the average replicated partition function $\left\langle
Z^{L}\right\rangle$ takes the form:

$\displaystyle\left\langle Z^{L}\right\rangle$
$\displaystyle=\left\langle\sum_{\sigma}\exp\left(-\beta\sum_{\gamma=1}^{L}H\left[\sigma^{\gamma}|\xi\right]\right)\right\rangle$

with $\sigma=\begin{Bmatrix}\sigma^{1}\ \dots\ \sigma^{L}\end{Bmatrix}$.
Gardner simplifies it to:

$\displaystyle\left\langle Z^{L}\right\rangle$
$\displaystyle\approx\Bigg{\langle}\sum_{\sigma}\exp\Bigg{(}\beta
N\sum_{\gamma}\sum_{\mu\in\Gamma_{\gamma}}\left[\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma^{\gamma}_{i}\right]^{p}$
(9)
$\displaystyle\quad+\beta\sum_{\gamma}\sum_{\mu\in\bar{\Gamma}}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\Bigg{)}\Bigg{\rangle},$

where the sets $\Gamma_{\gamma}$ contain the patterns $\xi^{\mu}$ that have
macroscopic overlap with $\sigma_{\gamma}$, and their complement
$\bar{\Gamma}=\cap_{\gamma}\bar{\Gamma}_{\gamma}$ consists of the remaining
patterns. Two approximations are used to obtain this expression:

  * •

$\sum_{\mu\in\Gamma_{\gamma}}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\approx
N\sum_{\mu\in\Gamma_{\gamma}}\left[\frac{1}{N}\sum_{i}\xi^{\mu}_{i}\sigma^{\gamma}_{i}\right]^{p}$
because this part of $H\left[\sigma^{\gamma}|\xi\right]$ is aligned with
$\sigma$ (see Case 1 of Appendix A).

  * •

$\sum_{\mu\in\bar{\Gamma}_{\gamma}}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\approx\sum_{\mu\in\bar{\Gamma}}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}$
since $\bar{\Gamma}$ contains almost all of the elements in each
$\bar{\Gamma}_{\gamma}$ when $N$ is large.

Gardner evaluates the contribution of the $\mu\in\bar{\Gamma}$ terms via a
cumulant expansion, resulting in:

$\displaystyle\log\left\langle\exp\left(\beta\sum_{\gamma}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\right)\right\rangle$
$\displaystyle\approx\beta\left\langle\sum_{\gamma}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\right\rangle$
$\displaystyle\quad+\frac{1}{2}\beta^{2}\left\langle\left[\sum_{\gamma}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\right]^{2}\right\rangle$
$\displaystyle\approx\frac{1}{2}\beta^{2}\left\langle\left[\sum_{\gamma}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\right]\left[\sum_{\delta}\frac{p!}{N^{p-1}}\sum_{j_{1}<...<j_{p}}\xi^{\mu}_{j_{1}}...\xi^{\mu}_{j_{p}}\sigma^{\delta}_{j_{1}}...\sigma^{\delta}_{j_{p}}\right]\right\rangle$

because the product of independent spins
$\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}$ averages to $0$. The sums are then
regrouped to get

$\displaystyle\log\left\langle\exp\left(\beta\sum_{\gamma}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\right)\right\rangle$
$\displaystyle=\frac{1}{2}\beta^{2}\left[\frac{p!}{N^{p-1}}\right]^{2}\left\langle\sum_{\gamma}\sum_{\delta}\sum_{i_{1}<...<i_{p}}\sum_{j_{1}<...<j_{p}}\xi^{\mu}_{i_{1}}\xi^{\mu}_{j_{1}}...\xi^{\mu}_{i_{p}}\xi^{\mu}_{j_{p}}\
\sigma^{\gamma}_{i_{1}}\sigma^{\delta}_{j_{1}}...\sigma^{\gamma}_{i_{p}}\sigma^{\delta}_{j_{p}}\right\rangle.$

Consider $\xi^{\mu}_{i}\xi^{\mu}_{j}$ for an arbitrary pair of indices $i$ and
$j$. There are two cases.

  * •

If $i=j$, then $\xi^{\mu}_{i}\xi^{\mu}_{j}$ is deterministic and equal to $1$.

  * •

If $i\neq j$, then $\xi^{\mu}_{i}\xi^{\mu}_{j}$ can be either $+1$ and $-1$
with equal probabilities.

On the one hand, if $i_{n}=j_{n}$ for all $n$, then
$\left\langle\xi^{\mu}_{i_{1}}\xi^{\mu}_{j_{1}}...\xi^{\mu}_{i_{p}}\xi^{\mu}_{j_{p}}\right\rangle=1$.
On the other hand, if $i_{n}\neq j_{n}$ for some $n$, then
$\left\langle\xi^{\mu}_{i_{1}}\xi^{\mu}_{j_{1}}...\xi^{\mu}_{i_{p}}\xi^{\mu}_{j_{p}}\right\rangle=0$
because
$\xi^{\mu}_{i_{1}}\xi^{\mu}_{j_{1}}...\xi^{\mu}_{i_{p}}\xi^{\mu}_{j_{p}}$ is
still a product of independent random spins once the deterministic variables
are removed. These two cases can be summarized by
$\left\langle\xi^{\mu}_{i_{1}}\xi^{\mu}_{j_{1}}...\xi^{\mu}_{i_{p}}\xi^{\mu}_{j_{p}}\right\rangle=\delta_{i_{1}j_{1}}...\delta_{i_{p}j_{p}}$,
which then gives

$\displaystyle\log\left\langle\exp\left(\beta\sum_{\gamma}\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\right)\right\rangle$
$\displaystyle=\frac{1}{2}\beta^{2}\left[\frac{p!}{N^{p-1}}\right]^{2}\sum_{\gamma}\sum_{\delta}\sum_{i_{1}<...<i_{p}}\sum_{j_{1}<...<j_{p}}\delta_{i_{1}j_{1}}...\delta_{i_{p}j_{p}}\
\sigma^{\gamma}_{i_{1}}\sigma^{\delta}_{j_{1}}...\sigma^{\gamma}_{i_{p}}\sigma^{\delta}_{j_{p}}$
$\displaystyle=\frac{1}{2}\beta^{2}\left[\frac{p!}{N^{p-1}}\right]^{2}\sum_{\gamma}\sum_{\delta}\sum_{i_{1}<...<i_{p}}\sigma^{\gamma}_{i_{1}}\sigma^{\delta}_{i_{1}}...\sigma^{\gamma}_{i_{p}}\sigma^{\delta}_{i_{p}}$
$\displaystyle\approx\frac{1}{2}\beta^{2}\frac{p!}{N^{p-1}}\frac{1}{N^{p-1}}\sum_{\gamma\delta}\left[\sum_{i}\sigma^{\gamma}_{i}\sigma^{\delta}_{i}\right]^{p}$
$\displaystyle=\beta^{2}\frac{p!}{N^{p-1}}N\sum_{\gamma<\delta}\left[\frac{1}{N}\sum_{i}\sigma^{\gamma}_{i}\sigma^{\delta}_{i}\right]^{p}+\frac{1}{2}\beta^{2}\frac{p!}{N^{p-1}}LN.$

The order $n>2$ terms are subdominant in $N$ and can be neglected when $p\geq
3$ [30]. The RS free entropy is then obtained through a standard approach to
the replica method. Note that Gardner’s Hamiltonian is misaligned with
$\sigma$ when the free entropy is dominated by this cumulant expansion (see
Case 2 of Appendix A). In the case of Krotov’s Hamiltonian, we must also take
into account the correction
$H^{\prime}\left[\sigma\right]=\frac{1}{2}\frac{p!}{N^{p-2}}\sum_{\gamma}\sum_{i_{1}<...<i_{p-2}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p-2}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p-2}}$
introduced in appendix A by imposing $i_{p-1}=i_{p}$. In fact, a cumulant
expansion of this expression gives

$\displaystyle\log\left\langle\exp\left(\beta
p\sum_{\gamma}\frac{1}{2}\frac{p!}{N^{p-2}}\sum_{i_{1}<...<i_{p-2}}\xi^{\mu}_{i_{1}}...\xi^{\mu}_{i_{p-2}}\sigma^{\gamma}_{i_{1}}...\sigma^{\gamma}_{i_{p-2}}\right)\right\rangle$
$\displaystyle\approx\frac{1}{4}\beta^{2}\frac{p!}{N^{p-2}}\frac{p(p-1)}{N^{p-2}}\sum_{\gamma<\delta}\left[\sum_{i}\sigma^{\gamma}_{i}\sigma^{\delta}_{i}\right]^{p-2}+\frac{1}{8}\beta^{2}\frac{p!}{N^{p-2}}L$
$\displaystyle=\frac{1}{4}p(p-1)\beta^{2}\frac{p!}{N^{p-1}}N\sum_{\gamma<\delta}\left[\frac{1}{N}\sum_{i}\sigma^{\gamma}_{i}\sigma^{\delta}_{i}\right]^{p-2}+\frac{1}{8}\beta^{2}\frac{p!}{N^{p-1}}LN,$

which contributes to the free energy on the same order in $N$ as Gardner’s
Hamiltonian. Therefore, Krotov’s Hamiltonian is not equivalent to Gardner’s
Hamiltonian when the latter is misaligned with $\sigma$ (see Case 2). The
index configurations with more equality constraints also contribute to the
free entropy on the same order in $N$ because the factors of $N$ that are lost
to equality constraints are restored when the sums get squared in the cumulant
expansion.

$p=2$ is the only positive integer such that Gardner’s Hamiltonian and
Krotov’s Hamiltonian are equivalent [5, 30]. In the misaligned case with a
single stored pattern $\xi^{*}$ (see Case 2), the free entropy of $p=2$
simplifies to:

$\displaystyle\frac{\log\left(Z\right)}{N}$
$\displaystyle=\frac{1}{N}\log\left\langle\exp\left\\{\beta\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{\gamma}_{i_{1}}\sigma^{\gamma}_{i_{2}}\right\\}\right\rangle+\log
2$
$\displaystyle=\frac{1}{N}\log\left\langle\exp\left(-\beta\right)\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}+x\sqrt{\beta\frac{2}{N}}\sum_{i}\xi^{*}_{i}\sigma^{\gamma}_{i}\right\\}\right\rangle+\log
2$
$\displaystyle=\frac{1}{N}\log\left[\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\cosh^{N}\left(x\sqrt{\beta\frac{2}{N}}\right)\right]-\beta\frac{1}{N}+\log
2,$

by using the Hubbard-Stratonovich transformation. At large $N$, it
approximates to:

$\displaystyle\frac{\log\left(Z\right)}{N}$
$\displaystyle\approx\frac{1}{N}\log\left[\int
dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\left(1+\beta\frac{1}{N}x^{2}\right)^{N}\right]-\beta\frac{1}{N}+\log
2$
$\displaystyle\approx\frac{1}{N}\log\left[\int_{\mathbb{R}}dx\frac{1}{\sqrt{2\pi}}\exp\left\\{-\frac{1}{2}x^{2}\right\\}\exp\left(\beta
x^{2}\right)\right]-\beta\frac{1}{N}+\log 2$
$\displaystyle=\left(-\frac{1}{2}\log\left(1-2\beta\right)-\beta\right)\frac{1}{N}+\log
2,$

thanks to the well-known limit
$\lim_{N\rightarrow\infty}\left(1+\frac{1}{N}z\right)^{N}=\exp\left(z\right)$.
This free entropy is consistent with the one found in literature when
$\alpha=\frac{1}{N}$ [5].

##  Appendix C Teacher-student replicated partition function

Recall that the student samples its pattern from the posterior
$P\left(\xi|\sigma\right)=\frac{P\left(\xi\right)\prod_{a}P\left(\sigma^{a}|\xi\right)}{P\left(\sigma\right)}$
(see Section 3). Given $P\left(\xi\right)$ uniform, it can be rewritten as
$P\left(\xi|\sigma\right)=\frac{\prod_{a}P\left(\sigma^{a}|\xi\right)}{\sum_{\xi}\prod_{a}P\left(\sigma^{a}|\xi\right)}$,
where $P\left(\sigma^{a}|\xi\right)$ is the distribution of the direct model
with a single pattern $\xi$. To simplify $P\left(\xi|\sigma\right)$ further,
we need to manipulate the partition function
$Z=\sum_{\sigma^{a}}\exp\left(-\beta H\left[\sigma^{a}|\xi\right]\right)$ of
$P\left(\sigma^{a}|\xi\right)$ (see Appendix A for the definition of
$H\left[\sigma|\xi\right]$). Under the gauge transformation
$\sigma^{a}_{i}\rightarrow\xi_{i}\sigma^{a}_{i}$, we may write

$\displaystyle
Z=\sum_{\sigma^{a}}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)$

without changing the configurations of $\sigma^{a}$ that we are summing over.
Therefore, $Z$ does not depend on $\xi$, and we can factor it out of the sum
$\sum_{\xi}$, which yields

$\displaystyle P\left(\xi|\sigma\right)$
$\displaystyle=\frac{\prod_{a}\frac{1}{Z}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi_{i_{1}}...\xi_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)}{\sum_{\xi}\prod_{a}\frac{1}{Z}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{i_{1}<...<i_{p}}\xi_{i_{1}}...\xi_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)}$
$\displaystyle=\frac{\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{a}\sum_{i_{1}<...<i_{p}}\xi_{i_{1}}...\xi_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)}{\sum_{\xi}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{a}\sum_{i_{1}<...<i_{p}}\xi_{i_{1}}...\xi_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)}.$

From this expression, we define the partition function of the inverse model to
be $\mathcal{Z}=\sum_{\xi}\exp\left(-\beta H\left[\xi|\sigma\right]\right)$
(again, see Appendix A for the definition of $H\left[\xi|\sigma\right]$). The
$L^{\text{th}}$ power of $\mathcal{Z}$ and its average then take the form

$\displaystyle\mathcal{Z}^{L}$
$\displaystyle=\sum_{\xi}\prod_{b}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{a}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right)$
$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle=\sum_{\sigma}P\left(\sigma\right)\sum_{\xi}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{ab}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right),$

where $b\in\begin{Bmatrix}1\ \dots\ L\end{Bmatrix}$ label replicas in the set
of patterns $\xi=\begin{Bmatrix}\xi^{1}\ \dots\ \xi^{L}\end{Bmatrix}$ inferred
by the student. Using the definition of conditional probability, we rewrite
$P\left(\sigma\right)$ as

$\displaystyle P\left(\sigma\right)$
$\displaystyle=\sum_{\xi^{*}}P\left(\sigma\big{|}\xi^{*}\right)P\left(\xi^{*}\right)$
$\displaystyle=\frac{1}{2^{N}}\sum_{\xi^{*}}P\left(\sigma\big{|}\xi^{*}\right)$
$\displaystyle=\frac{1}{2^{N}}\sum_{\xi^{*}}\prod_{a}P\left(\sigma^{a}\big{|}\xi^{*}\right),$

where $P\left(\sigma|\xi^{*}\right)$ has the same functional form as
$P\left(\sigma|\xi^{b}\right)$, but has hyperparameters $p^{*}$ and
$\beta^{*}$ in place of $p$ and $\beta$. As we did for $Z$, we factor the
partition function $Z^{*}$ of $P\left(\sigma^{a}|\xi^{*}\right)$ out of the
sum, which yields

$\displaystyle P\left(\sigma\right)$
$\displaystyle=\frac{1}{2^{N}}\frac{1}{\left[Z^{*}\right]^{M}}\sum_{\xi^{*}}\prod_{a}\exp\left(\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}\right)$
$\displaystyle=\frac{1}{2^{N}}\frac{\mathcal{Z}^{*}}{\left[Z^{*}\right]^{M}}=\frac{1}{2^{MN}}\frac{\mathcal{Z}^{*}}{\left[2^{N/M-N}Z^{*}\right]^{M}},$

where
$\mathcal{Z}^{*}=\sum_{\xi^{*}}\exp\left(-\beta^{*}H\left[\xi^{*}|\sigma\right]\right)$
is the partition function of the inverse model with interaction order $p^{*}$.
Using $\sum_{\sigma}P\left(\sigma\right)=1$, we immediately deduce that
$\left[2^{N/M-N}Z^{*}\right]^{M}=\left\langle\mathcal{Z}^{*}\right\rangle$.
Plugging
$P\left(\sigma\right)=\frac{1}{2^{MN}}\frac{\mathcal{Z}^{*}}{\left\langle\mathcal{Z}^{*}\right\rangle}$
back in $\left\langle\mathcal{Z}^{L}\right\rangle$ then gives

$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle=\frac{1}{2^{MN}}\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}}\sum_{\sigma}\exp\left(\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}\right)$
$\displaystyle\quad\sum_{\xi}\exp\left(\beta\frac{p!}{N^{p-1}}\sum_{ab}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right).$

We simplify this expression to:

$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle=\frac{1}{2^{MN}}\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}}\sum_{\sigma}\exp\Bigg{(}\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a\in\Gamma_{*}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}$
$\displaystyle\quad+\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a\in\bar{\Gamma}_{*}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}\Bigg{)}$
$\displaystyle\quad\sum_{\xi}\exp\Bigg{(}\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{a\in\Gamma_{b}}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}$
$\displaystyle\quad+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{a\in\bar{\Gamma}_{b}}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{)}$
$\displaystyle\approx\frac{1}{2^{MN}}\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}}\sum_{\sigma}\exp\Bigg{(}\beta^{*}N\sum_{a\in\Gamma_{*}}\left[\frac{1}{N}\sum_{i}\xi^{*}_{i}\sigma^{a}_{i}\right]^{p^{*}}$
$\displaystyle\quad+\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}\Bigg{)}$
$\displaystyle\quad\sum_{\xi}\exp\Bigg{(}\beta
N\sum_{b}\sum_{a\in\Gamma_{b}}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}\right]^{p}$
$\displaystyle\quad+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{)},$

where $\Gamma_{b}$ represents the set of inputs $\sigma^{a}$ which have
macroscopic overlap with the pattern $\xi^{b}$, and
$\bar{\Gamma}=\left[\cap_{b}\bar{\Gamma}_{b}\right]\cap\bar{\Gamma}_{*}$
contains almost all of the elements in each $\bar{\Gamma}_{b}$ and
$\bar{\Gamma}_{*}$ for $N\rightarrow\infty$. The reasoning used to build the
sets $\Gamma_{*}$, $\Gamma_{b}$ and $\bar{\Gamma}$ is the same as outlined at
the start of appendix B.

##  Appendix D Teacher-student free entropy

Assuming that the teacher is misaligned with $\sigma$ (see Case 2 of Appendix
A), the form of $\left\langle\mathcal{Z}^{L}\right\rangle$ obtained in
appendix C simplifies to

$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle\approx\frac{1}{2^{MN}}\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}\xi}\sum_{\sigma}\exp\Bigg{(}\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}\Bigg{)}$
$\displaystyle\quad\exp\Bigg{(}\beta
N\sum_{b}\sum_{a\in\Gamma_{b}}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}\right]^{p}+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{)}.$

In order to evaluate
$\left\langle\mathcal{Z}^{*}\right\rangle=\left[2^{N/M-N}Z^{*}\right]^{M}$, we
recall that the teacher is a special case of the direct model with a single
memory (see Section 3). Since the teacher is in the misaligned case, its free
entropy is

$\displaystyle\frac{\log\left(Z^{*}\right)}{N}$
$\displaystyle=\begin{cases}\left(-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}\right)\frac{1}{N}+\log
2&\quad p^{*}=2\\\
\frac{1}{2}\left[\beta^{*}\right]^{2}\frac{p^{*}!}{N^{p^{*}-1}}+\log
2+\mathcal{O}\left(\frac{1}{N^{3p^{*}/2-2)}}\right)&\quad p^{*}\geq
3,\end{cases}$

as derived in Appendix B. Given $\alpha^{*}=\frac{Mp^{*}!}{N^{p^{*}-1}}$, we
use it to simplify $\frac{\log\left\langle\mathcal{Z}^{*}\right\rangle}{N}$ to

$\displaystyle\frac{\log\left\langle\mathcal{Z}^{*}\right\rangle}{N}$
$\displaystyle=\frac{M\log\left[2^{N/M-N}Z^{*}\right]}{N}$
$\displaystyle=\begin{cases}\frac{1}{2}\left(-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}\right)\alpha^{*}+\log
2&\quad p^{*}=2\\\ \frac{1}{2}\left[\beta^{*}\right]^{2}\alpha^{*}+\log
2+\mathcal{O}\left(\frac{1}{N^{p^{*}/2-1}}\right)&\quad p^{*}\geq
3,\end{cases}$

which is the paramagnetic free entropy of a $p^{*}$-body Hopfield network [5,
30]. Coming back to $\left\langle\mathcal{Z}^{L}\right\rangle$, we fix order
parameters $q^{*b}$, $q^{bc}$ and $m^{b}_{a}$ using the delta functions
$\delta\left(Nq^{*b}-\sum_{i}\xi^{*}_{i}\xi^{b}_{i}\right)$,
$\delta\left(Nq^{bc}-\sum_{i}\xi^{b}_{i}\xi^{c}_{i}\right)$ and
$\delta\left(Nm^{b}_{a}-\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}\right)$, which
results in

$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle=\frac{1}{2^{MN}}\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}\xi}\sum_{\sigma}\int_{\mathbb{R}}\prod_{b}dq^{*b}\prod_{b<c}dq^{bc}\prod_{b}\prod_{a\in\Gamma_{b}}dm^{b}_{a}$
$\displaystyle\quad\delta\left(Nq^{*b}-\sum_{i}\xi^{*}_{i}\xi^{b}_{i}\right)\delta\left(Nq^{bc}-\sum_{i}\xi^{b}_{i}\xi^{c}_{i}\right)\delta\left(Nm^{b}_{a}-\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}\right)$
$\displaystyle\quad\exp\Bigg{(}\beta
N\sum_{b}\sum_{a\in\Gamma_{b}}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}\right]^{p}$
$\displaystyle\quad+\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}$
$\displaystyle\quad+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{)}.$

In Fourier space, this expression takes the form

$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle=\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}\xi}\Bigg{\langle}\int\prod_{b}dq^{*b}dr^{*b}\prod_{b<c}dq^{bc}dr^{bc}\prod_{b}\prod_{a\in\Gamma_{b}}dm^{b}_{a}dk^{b}_{a}$
$\displaystyle\quad\exp\left\\{\beta^{*}\beta\alpha\sum_{b}\left(\sum_{i}\xi^{*}_{i}\xi^{b}_{i}-Nq^{*b}\right)r^{*b}+\beta^{2}\alpha\sum_{b<c}\left(\sum_{i}\xi^{b}_{i}\xi^{c}_{i}-Nq^{bc}\right)r^{bc}\right\\}$
$\displaystyle\quad\exp\Bigg{\\{}\beta\sum_{b}\sum_{a\in\Gamma_{b}}\left(\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}-Nm^{b}_{a}\right)k^{b}_{a}+\beta
N\sum_{b}\sum_{a\in\Gamma_{b}}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}\right]^{p}$
$\displaystyle\quad+\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}$
$\displaystyle\quad+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{a\in\bar{\Gamma}}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{\\}}\Bigg{\rangle}_{\sigma},$

where the sum over $\sigma$ with a pre-factor of $\frac{1}{2^{MN}}$ was
replaced by the uniform average $\langle\rangle_{\sigma}$. Following the same
reasoning as in appendix B, a second order cumulant expansion of the last two
terms for any $a\in\bar{\Gamma}$ yields

$\displaystyle\log\Bigg{\langle}\exp\Bigg{\\{}\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}$
$\displaystyle\quad+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{\\}}\Bigg{\rangle}$
$\displaystyle\approx\frac{1}{2}\beta^{2}\left[\frac{p!}{N^{p-1}}\right]^{2}\sum_{b\neq
c}\sum_{i_{1}<...<i_{p}}\sum_{j_{1}<...<j_{p}}\xi^{b}_{i_{1}}\xi^{c}_{j_{1}}...\xi^{b}_{i_{p}}\xi^{c}_{j_{p}}\left\langle\sigma^{a}_{i_{1}}\sigma^{a}_{j_{1}}...\sigma^{a}_{i_{p}}\sigma^{a}_{j_{p}}\right\rangle$
$\displaystyle\quad+\beta^{*}\beta\frac{p^{*}!}{N^{p^{*}-1}}\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p^{*}}}\sum_{j_{1}<...<j_{p}}\left\langle\xi^{*}_{i_{1}}\sigma^{a}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{p^{*}}}\xi^{b}_{j_{1}}\sigma^{a}_{j_{1}}...\xi^{b}_{j_{p}}\sigma^{a}_{j_{p}}\right\rangle$
$\displaystyle\quad+\frac{1}{2}\beta^{2}\frac{p!}{N^{p-1}}LN+\frac{1}{2}\left[\beta^{*}\right]^{2}\frac{p^{*}!}{N^{p^{*}-1}}N.$

When $p^{*}=p$, it reduces to

$\displaystyle\log\Bigg{\langle}\exp\Bigg{\\{}\beta^{*}\frac{p^{*}!}{N^{p^{*}-1}}\sum_{i_{1}<...<i_{p^{*}}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p^{*}}}$
$\displaystyle\quad+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{\\}}\Bigg{\rangle}$
$\displaystyle=\beta^{2}\frac{p!}{N^{p-1}}N\sum_{b<c}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\xi^{c}_{i}\right]^{p}+\beta^{*}\beta\frac{p!}{N^{p-1}}N\sum_{b}\left[\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi^{b}_{i}\right]^{p}$
$\displaystyle\quad+\frac{1}{2}\beta^{2}\frac{p!}{N^{p-1}}LN+\frac{1}{2}\left[\beta^{*}\right]^{2}\frac{p!}{N^{p-1}}N,$

because
$\left\langle\sigma^{a}_{i_{n}}\sigma^{a}_{j_{n}}\right\rangle=\delta_{i_{n}j_{n}}$
(see Appendix B for more details). On the contrary, the second order
expectation
$\left\langle\xi^{*}_{i_{1}}\sigma^{a}_{i_{1}}...\xi^{*}_{i_{p^{*}}}\sigma^{a}_{i_{p^{*}}}\xi^{b}_{j_{1}}\sigma^{a}_{j_{1}}...\xi^{b}_{j_{p}}\sigma^{a}_{j_{p}}\right\rangle$
vanishes when $p^{*}\neq p$. In fact, spins come in pairs
$\left\langle\sigma^{a}_{i_{n}}\sigma^{a}_{j_{n}}\right\rangle=\delta_{i_{n}j_{n}}$
only up to $n\leq\min\left\\{p^{*},p\right\\}$, and the remaining single-spin
averages $\left\langle\sigma^{a}_{i_{n}}\right\rangle=0$ make the second order
expectation vanish.

We need to go beyond second order to treat $p^{*}\neq p$. We will focus on
$p^{*}=2$ and $p\geq 3$ to investigate the consequences of using a $p$-body
model to learn examples generated by the original $2$-body Hopfield model. For
simplicity, we take $p$ even so that the spins of both terms can be grouped in
pairs at order $\frac{p}{2}+1$, when the teacher term
$\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}$
is raised to the power of $\frac{p}{2}$ and the student term is raised to the
power of $1$. This restriction will simplify some of the incoming
calculations. To leading order in $N$, the cumulant generating function
reduces to

$\displaystyle\log\Bigg{\langle}\exp\Bigg{\\{}\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{\\}}\Bigg{\rangle}$
$\displaystyle\approx\log\Bigg{[}\left\langle\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle\quad\left\langle\exp\left\\{\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right\\}\right\rangle$
$\displaystyle\quad+\left\langle\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{j_{1}<...<j_{p}}\xi^{b}_{j_{1}}...\xi^{b}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{p}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle\Bigg{]},$

where the last term encompasses the teacher-student coupling that allows
retrieval to take place. The teacher term

$\displaystyle\log\left\langle\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle\approx-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}$

and the student term

$\displaystyle\log\left\langle\exp\left\\{\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\right\\}\right\rangle$
$\displaystyle\approx\beta^{2}\frac{p!}{N^{p-1}}N\sum_{b<c}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\xi^{c}_{i}\right]^{p}+\frac{1}{2}\beta^{2}\frac{p!}{N^{p-1}}LN$

are both known from Appendix B. Later on, we will use $\log\left(z^{*}\right)$
and $z^{*}$ as shorthands for
$-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}$ and
$\exp\left(-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}\right)$,
respectively. The coupling between the teacher and the student can be
rewritten as

$\displaystyle\left\langle\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{j_{1}<...<j_{p}}\xi^{b}_{j_{1}}...\xi^{b}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{p}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle=\left\langle\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{j_{1}<...<j_{p}}\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\xi^{b}_{j_{1}}...\xi^{b}_{j_{p}}\
\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{p}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle=\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{j_{1}<...<j_{p}}\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\xi^{b}_{j_{1}}...\xi^{b}_{j_{p}}\left\langle\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{p}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$

because $\left[\xi^{*}_{j_{n}}\right]^{2}=1$ for every index $j_{n}$. All
interacting spin tuples of the form
$\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{i_{p}}$ are
statistically equivalent as long as $j_{1}<...<j_{p}$, so the teacher-student
coupling simplifies to

$\displaystyle\left\langle\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{j_{1}<...<j_{p}}\xi^{b}_{j_{1}}...\xi^{b}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{p}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle=\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}$
$\displaystyle\quad\left\langle\frac{p!}{N^{p}}\sum_{j_{1}<...<j_{p}}\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{i_{p}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle=V\left(\beta^{*},p\right)\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{*}_{i_{1}}...\xi^{*}_{i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}},$

where
$V\left(\beta^{*},p\right)=\left\langle\frac{p!}{N^{p}}\sum_{j_{1}<...<j_{p}}\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{i_{p}}\exp\left(\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right)\right\rangle$
does not depend on the microscopic details of the system. In fact, it can be
expressed as a combination of the moments of $z^{*}$, which can all be derived
from $\log\left(z^{*}\right)$. To leading order in $N$, the cumulant
generating function expands to

$\displaystyle\log\Bigg{\langle}\exp\Bigg{\\{}\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}+\beta\frac{p!}{N^{p-1}}\sum_{b}\sum_{i_{1}<...<i_{p}}\xi^{b}_{i_{1}}...\xi^{b}_{i_{p}}\sigma^{a}_{i_{1}}...\sigma^{a}_{i_{p}}\Bigg{\\}}\Bigg{\rangle}$
$\displaystyle\approx-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}+\beta^{2}\frac{p!}{N^{p-1}}N\sum_{b<c}\left[\frac{1}{N}\sum_{i}\xi^{b}_{i}\xi^{c}_{i}\right]^{p}+\frac{1}{2}\beta^{2}\frac{p!}{N^{p-1}}LN$
$\displaystyle\quad+\left[1-2\beta^{*}\right]^{1/2}\exp\left(\beta^{*}\right)V\left(\beta^{*},p\right)\beta
N\sum_{b}\left[\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi^{b}_{i}\right]^{p}.$

At this stage, we only need to find $V\left(\beta^{*},p\right)$ in order to
solve the system. We focus on two different scalings of $M$ and $\beta^{*}$
that make the teacher-student coupling leading order in $N$:

  1. 1

$M\sim\mathcal{O}\left(N^{p-1}\right)$ and
$\beta^{*}\sim\mathcal{O}\left(N^{2/p-1}\right)$ will be called the large-
noise scaling.

  2. 2

$M\sim\mathcal{O}\left(N^{p/2}\right)$ and
$\beta^{*}\sim\mathcal{O}\left(1\right)$ will be called the finite-noise
scaling.

The student term vanishes in the first scenario but is leading order in the
second one. The case of the teacher-student coupling is more subtle. When
$\beta^{*}$ is small, we may keep only the first non-vanishing order of the
exponential function present in the definition of $V\left(\beta^{*},p\right)$.
Since $p$ is even, it leads to

$\displaystyle V\left(\beta^{*},p\right)$
$\displaystyle\approx\frac{1}{(p/2)!}\left\langle\frac{p!}{N^{p}}\sum_{j_{1}<...<j_{p}}\xi^{*}_{j_{1}}...\xi^{*}_{j_{p}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{p}}\left(\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right)^{p/2}\right\rangle$
(10)
$\displaystyle=\frac{\left[\beta^{*}\right]^{p/2}}{(p/2)!}\frac{2^{p/2}}{N^{p/2}}\frac{p!}{2^{p/2}}$
$\displaystyle=\frac{\left[\beta^{*}\right]^{p/2}}{(p/2)!}\frac{p!}{N^{p/2}}$

because there are $\prod^{p/2}_{n=1}\binom{2n}{2}=\frac{p!}{2^{p/2}}$ spin
pairings with non-zero expectation that satisfy the inequality constraints. In
the large-noise scaling, we set

$\displaystyle\lambda$
$\displaystyle=\frac{\left[\beta^{*}\right]^{p/2}}{(p/2)!}N^{p/2-1}\sim\mathcal{O}\left(1\right)$

to get the asymptotically exact expression
$V\left(\left[(p/2)!\right]^{2/p}N^{1-2/p},p\right)=\lambda\frac{p!}{N^{p-1}}$.
In the finite-noise scaling, this expansion is only an order of magnitude
approximation. However, it still indicates that $V\left(\beta^{*},p\right)$ is
$\mathcal{O}\left(N^{-p/2}\right)$ when $\beta^{*}$ is
$\mathcal{O}\left(1\right)$ in $N$. In other words, it shows that there is an
$\mathcal{O}\left(1\right)$ parameter $\eta$ such that
$V\left(\beta^{*}\left(\eta,p\right),p\right)=\eta\frac{\left(p/2+1\right)!}{N^{p/2}}$.
We will now use the cumulants
$\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*}}$ and
$\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*2}}$ of $z^{*}$ to
derive the value of $\eta$ corresponding to $p=4$. First of all, note that
$\frac{4!}{N^{4}}\sum_{j_{1}<...<j_{4}}\xi^{*}_{j_{1}}...\xi^{*}_{j_{4}}\sigma^{a}_{j_{1}}...\sigma^{a}_{j_{4}}$
can be expressed as:

$\displaystyle\frac{24}{N^{4}}\sum_{j_{1}<j_{2}<j_{3}<j_{4}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\xi^{*}_{j_{3}}\xi^{*}_{j_{4}}\sigma^{a}_{j_{1}}\sigma^{a}_{j_{2}}\sigma^{a}_{j_{3}}\sigma^{a}_{j_{4}}$
$\displaystyle=\frac{1}{N^{4}}\sum_{j_{1}\neq j_{2}\neq j_{3}\neq
j_{4}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\xi^{*}_{j_{3}}\xi^{*}_{j_{4}}\sigma^{a}_{j_{1}}\sigma^{a}_{j_{2}}\sigma^{a}_{j_{3}}\sigma^{a}_{j_{4}}$
$\displaystyle=\frac{1}{N^{4}}\left[\sum_{j_{1}\neq
j_{2}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\sigma^{a}_{j_{1}}\sigma^{a}_{j_{2}}\right]\left[\sum_{j_{3}\neq
j_{4}}\xi^{*}_{j_{3}}\xi^{*}_{j_{4}}\sigma^{a}_{j_{3}}\sigma^{a}_{j_{4}}\right]-\frac{4}{N^{3}}\left[\sum_{j_{1}\neq
j_{2}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\sigma^{a}_{j_{1}}\sigma^{a}_{j_{2}}\right]-\frac{2}{N^{2}}$
$\displaystyle=\frac{1}{N^{2}}\left[\frac{2}{N}\sum_{j_{1}<j_{2}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\sigma^{a}_{j_{1}}\sigma^{a}_{j_{2}}\right]^{2}-\frac{4}{N^{2}}\left[\frac{2}{N}\sum_{j_{1}<j_{2}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\sigma^{a}_{j_{1}}\sigma^{a}_{j_{2}}\right]-\frac{2}{N^{2}}$

by subtracting the diagonals where pairs of indices are equal. Therefore,
$\frac{1}{z^{*}}V\left(\beta^{*},p\right)$ reduces to

$\displaystyle\frac{1}{z^{*}}V\left(\beta^{*},p\right)$
$\displaystyle=\left\langle\frac{24}{N^{4}}\sum_{j_{1}<j_{2}<j_{3}<j_{4}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\xi^{*}_{j_{3}}\xi^{*}_{j_{4}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\sigma^{a}_{i_{3}}\sigma^{a}_{i_{4}}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle=\frac{1}{z^{*}}\frac{1}{N^{2}}\Bigg{[}\left\langle\left[\frac{2}{N}\sum_{j_{1}<j_{2}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right]^{2}\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle\quad-4\left\langle\left[\frac{2}{N}\sum_{j_{1}<j_{2}}\xi^{*}_{j_{1}}\xi^{*}_{j_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right]\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle$
$\displaystyle\quad-2\left\langle\exp\left\\{\beta^{*}\frac{2}{N}\sum_{i_{1}<i_{2}}\xi^{*}_{i_{1}}\xi^{*}_{i_{2}}\sigma^{a}_{i_{1}}\sigma^{a}_{i_{2}}\right\\}\right\rangle\Bigg{]}$
$\displaystyle=\frac{1}{N^{2}}\left[\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*2}}+\left[\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*}}\right]^{2}-4\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*}}-2\right].$

The cumulants evaluate to

$\displaystyle\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*}}$
$\displaystyle=\frac{\partial}{\partial\beta^{*}}\left[-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}\right]=\frac{2\beta^{*}}{1-2\beta^{*}}$
$\displaystyle\frac{\partial\log\left(z^{*}\right)}{\partial\beta^{*2}}$
$\displaystyle=\frac{\partial}{\partial\beta^{*2}}\left[-\frac{1}{2}\log\left(1-2\beta^{*}\right)-\beta^{*}\right]=\frac{2}{\left(1-2\beta^{*}\right)^{2}},$

so we obtain

$\displaystyle\frac{1}{z^{*}}V\left(\beta^{*},p\right)$
$\displaystyle=\frac{1}{N^{2}}\left[\frac{2}{\left(1-2\beta^{*}\right)^{2}}+\frac{4\left[\beta^{*}\right]^{2}}{\left(1-2\beta^{*}\right)^{2}}-\frac{8\beta^{*}}{1-2\beta^{*}}-2\right]$
$\displaystyle=\frac{6}{N^{2}}\frac{2\left[\beta^{*}\right]^{2}}{\left(1-2\beta^{*}\right)^{2}}.$

In other terms, we find
$\eta=\frac{2\left[\beta^{*}\right]^{2}}{\left(1-2\beta^{*}\right)^{2}}$ when
$p=4$. In summary, depending on the scaling, the teacher student coupling
either simplifies to

  1. 1

$\beta\lambda\alpha\frac{N}{M}\sum_{b}\left[\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi^{b}_{i}\right]^{p}$
where $\alpha=\frac{Mp!}{N^{p-1}}$ and
$\lambda=\frac{\left[\beta^{*}\right]^{p/2}}{(p/2)!}N^{p/2-1}$ are finite,

  2. 2

or
$\beta\eta\alpha\frac{N}{M}\sum_{b}\left[\frac{1}{N}\sum_{i}\xi^{*}_{i}\xi^{b}_{i}\right]^{p}$
where $\alpha=\frac{M\left(p/2+1\right)!}{N^{p/2}}$ and $\eta$ are finite.

In either case, the result is similar to $p^{*}=p$ except for its pre-factor.
We describe the rest of the derivation only for $p^{*}=p$ because the
$p^{*}=2$ and $p\geq 3$ calculations are almost identical. Putting the result
of the $p^{*}=p$ cumulant expansion back in
$\left\langle\mathcal{Z}^{L}\right\rangle$, we get:

$\displaystyle\left\langle\mathcal{Z}^{L}\right\rangle$
$\displaystyle\approx\frac{1}{\left\langle\mathcal{Z}^{*}\right\rangle}\sum_{\xi^{*}\xi}\Bigg{\langle}\int\prod_{b}dq^{*b}dr^{*b}\prod_{b<c}dq^{bc}dr^{bc}\prod_{b}\prod_{a\in\Gamma_{b}}dm^{b}_{a}\
dk^{b}_{a}$
$\displaystyle\quad\exp\left\\{\beta^{*}\beta\alpha\sum_{b}\left(\sum_{i}\xi^{*}_{i}\xi^{b}_{i}-Nq^{*b}\right)r^{*b}+\beta^{2}\alpha\sum_{b<c}\left(\sum_{i}\xi^{b}_{i}\xi^{c}_{i}-Nq^{bc}\right)r^{bc}\right\\}$
$\displaystyle\quad\exp\left\\{\beta\sum_{b}\sum_{a\in\Gamma_{b}}\left(\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}-Nm^{b}_{a}\right)k^{b}_{a}+\beta
N\sum_{b}\sum_{a\in\Gamma_{b}}\left[m^{b}_{a}\right]^{p}\right\\}$
$\displaystyle\quad\exp\left\\{\beta^{*}\beta\alpha
N\sum_{b}\left[q^{*b}\right]^{p}+\beta^{2}\alpha
N\sum_{b<c}\left[q^{bc}\right]^{p}+\frac{1}{2}\beta^{2}\alpha
LN+\frac{1}{2}\left[\beta^{*}\right]^{2}\alpha N\right\\}\Bigg{\rangle},$

where $\alpha=\frac{Mp!}{N^{p-1}}$. The saddle point of
$\left\langle\mathcal{Z}^{L}\right\rangle$ then evaluates to

$\displaystyle\frac{\log\left\langle\mathcal{Z}^{L}\right\rangle}{N}$
$\displaystyle\approx\operatorname*{Extr}_{m,k,q,r,q^{*},r^{*}}\Bigg{[}\beta^{*}\beta\alpha\sum_{b}\left[q^{*b}\right]^{p}+\beta^{2}\alpha\sum_{b<c}\left[q^{bc}\right]^{p}+\beta\sum_{b}\sum_{a\in\Gamma_{b}}\left[m^{b}_{a}\right]^{p}$
$\displaystyle\quad-\beta^{*}\beta\alpha\sum_{b}r^{*b}q^{*b}-\beta^{2}\alpha\sum_{b<c}r^{bc}q^{bc}-\beta\sum_{b}\sum_{a\in\Gamma_{b}}m^{b}_{a}k^{b}_{a}$
$\displaystyle\quad+\frac{1}{2}\beta^{2}\alpha
L+\frac{1}{2}\left[\beta^{*}\right]^{2}\alpha-\frac{\log\left\langle\mathcal{Z}\right\rangle}{N}+\log
2$
$\displaystyle\quad+\frac{1}{N}\log\Bigg{\langle}\sum_{\xi}\exp\Bigg{\\{}\beta\sum_{b}\sum_{a\in\Gamma_{b}}k^{b}_{a}\sum_{i}\xi^{b}_{i}\sigma^{a}_{i}$
$\displaystyle\quad+\beta^{*}\beta\alpha\sum_{b}r^{*b}\sum_{i}\xi^{*}_{i}\xi^{b}_{i}+\beta^{2}\alpha\sum_{b<c}r^{bc}\sum_{i}\xi^{b}_{i}\xi^{c}_{i}\Bigg{\\}}\Bigg{\rangle}_{\xi^{*}\sigma}\Bigg{]}$
$\displaystyle=\operatorname*{Extr}\Bigg{[}\beta^{*}\beta\alpha\sum_{b}\left[q^{*b}\right]^{p}+\beta^{2}\alpha\sum_{b<c}\left[q^{bc}\right]^{p}+\beta\sum_{b}\sum_{a\in\Gamma_{b}}\left[m^{b}_{a}\right]^{p}$
$\displaystyle\quad-\beta^{*}\beta\alpha\sum_{b}r^{*b}q^{*b}-\beta^{2}\alpha\sum_{b<c}r^{bc}q^{bc}-\beta\sum_{b}\sum_{a\in\Gamma_{b}}m^{b}_{a}k^{b}_{a}$
$\displaystyle\quad+\frac{1}{2}\beta^{2}\alpha
L+\frac{1}{2}\left[\beta^{*}\right]^{2}\alpha-\frac{\log\left\langle\mathcal{Z}\right\rangle}{N}+\log
2$
$\displaystyle\quad+\frac{1}{N}\sum_{i}\log\Bigg{\langle}\sum_{\xi_{i}}\exp\Bigg{\\{}\beta\sum_{b}\sum_{a\in\Gamma_{b}}k^{b}_{a}\xi^{b}_{i}\sigma^{a}_{i}$
$\displaystyle\quad+\beta^{*}\beta\alpha\sum_{b}r^{*b}\xi^{*}_{i}\xi^{b}_{i}+\beta^{2}\alpha\sum_{b<c}r^{bc}\xi^{b}_{i}\xi^{c}_{i}\Bigg{\\}}\Bigg{\rangle}_{\xi^{*}_{i}\sigma_{i}}\Bigg{]},$

where the average over $\xi^{*}$ and $\sigma$ is uniform. We use
$\frac{\log\left\langle\mathcal{Z}^{*}\right\rangle}{N}=\frac{1}{2}\left[\beta^{*}\right]^{2}\alpha+\log
2$ to simplify $\frac{\log\left\langle\mathcal{Z}^{L}\right\rangle}{N}$ to

$\displaystyle\frac{\log\left\langle\mathcal{Z}^{L}\right\rangle}{N}$
$\displaystyle\approx\operatorname*{Extr}\Bigg{[}\beta^{*}\beta\alpha\sum_{b}\left[q^{*b}\right]^{p}+\beta^{2}\alpha\sum_{b<c}\left[q^{bc}\right]^{p}+\beta\sum_{b}\sum_{a\in\Gamma_{b}}\left[m^{b}_{a}\right]^{p}$
$\displaystyle\quad-\beta^{*}\beta\alpha\sum_{b}r^{*b}q^{*b}-\beta^{2}\alpha\sum_{b<c}r^{bc}q^{bc}-\beta\sum_{b}\sum_{a\in\Gamma_{b}}m^{b}_{a}k^{b}_{a}$
$\displaystyle\quad+\frac{1}{2}\beta^{2}\alpha
L+\frac{1}{N}\sum_{i}\log\Bigg{\langle}\sum_{\xi_{i}}\exp\Bigg{\\{}\beta\sum_{b}\sum_{a\in\Gamma_{b}}k^{b}_{a}\xi^{b}_{i}\sigma^{a}_{i}$
$\displaystyle\quad+\beta^{*}\beta\alpha\sum_{b}r^{*b}\xi^{*}_{i}\xi^{b}_{i}+\beta^{2}\alpha\sum_{b<c}r^{bc}\xi^{b}_{i}\xi^{c}_{i}\Bigg{\\}}\Bigg{\rangle}_{\xi^{*}_{i}\sigma_{i}}\Bigg{]}.$

Assuming each $\xi^{b}$ has macroscopic overlap with at most one pattern
$\sigma^{a}$ and using the replica-symmetric ansatz $q^{*b}=q^{*}$,
$q^{bc}=q$, $r^{*b}=r^{*}$, $r^{bc}=r$, $m^{b}_{a}=m$, $k^{b}_{a}=k$, the free
entropy approximates to

$\displaystyle f$ $\displaystyle=\lim_{N\rightarrow\infty,L\rightarrow
0}\left(\frac{\partial}{\partial L}\left[\frac{1}{N}\log\left\langle
Z^{L}\right\rangle\right]\right)$
$\displaystyle\approx\operatorname*{Extr}\Bigg{[}\beta^{*}\beta\alpha\left[q^{*}\right]^{p}-\frac{1}{2}\beta^{2}\alpha
q^{p}+\beta m^{p}-\beta^{*}\beta\alpha r^{*}q^{*}+\frac{1}{2}\beta^{2}\alpha
rq$ $\displaystyle\quad-\beta mk+\frac{1}{2}\beta^{2}\alpha+\lim_{L\rightarrow
0}\Bigg{(}\frac{\partial}{\partial
L}\Bigg{[}\log\Bigg{\langle}\sum_{\xi_{i}}\exp\Bigg{\\{}\beta
k\sum_{b}\xi^{b}_{i}\sigma^{a}_{i}$ $\displaystyle\quad+\beta^{*}\beta\alpha
r^{*}\sum_{b}\xi^{*}_{i}\xi^{b}_{i}+\beta^{2}\alpha
r\sum_{b<c}\xi^{b}_{i}\xi^{c}_{i}\Bigg{\\}}\Bigg{\rangle}\Bigg{]}\Bigg{)}\Bigg{]}.$

Furthermore, the Hubbard-Stratonovich transformation gives

$\displaystyle\exp\left\\{\beta^{2}\alpha
r\sum_{b<c}\xi^{b}_{i}\xi^{c}_{i}\right\\}\propto\exp\left\\{-\frac{1}{2}\beta^{2}\alpha
rL\right\\}\int_{\mathbb{R}}dx\exp\left\\{-\frac{1}{2}x^{2}+x\beta\sqrt{\alpha
r}\sum_{b}\xi^{b}_{i}\right\\}.$

We can then use the factorization

$\displaystyle\sum_{\xi_{i}}\exp\Bigg{\\{}\beta
k\sum_{b}\xi^{b}_{i}\sigma^{a}_{i}+\beta^{*}\beta\alpha
r^{*}\sum_{b}\xi^{*}_{i}\xi^{b}_{i}+x\beta\sqrt{\alpha