11institutetext:  Física Teórica, Universidad de Sevilla, Apartado de Correos
1065, E-41080 Sevilla, Spain
Departamento de Física Aplicada I, E.P.S., Universidad de Sevilla, Virgen de
África 7, E-41011 Sevilla, Spain

# Non-equilibrium memory effects: granular fluids and beyond

A. Patrón 11 B. Sánchez-Rey 22 C. A. Plata 11 and A. Prados 111122

###### Abstract

In this perspective paper, we look into memory effects in out-of-equilibrium
systems. To be concrete, we exemplify memory effects with the paradigmatic
case of granular fluids, although extensions to other contexts such as
molecular fluids with non-linear drag are also considered. The focus is put on
two archetypal memory effects: the Kovacs and Mpemba effects. In brief, the
first is related to imperfectly reaching a steady state—either equilibrium or
non-equilibrium, whereas the second is related to reaching a steady state
faster despite starting further. Connections to optimal control theory thus
naturally emerge and are briefly discussed.

##  1 Introduction

Under quite general conditions, many physical systems tend in the long time
limit to a state in which all trace of initial conditions is lost. This state
is often stationary, either an equilibrium state or a non-equilibrium steady
state (NESS), but it also may be a time-dependent “hydrodynamic” state—in
which a reduced description in terms of a few “thermodynamic” or
“macrosocopic” variables accounts for the complete characterisation of the
time evolution of the system.

Memory effects are intimately related to aging [1, 2, 3]. A system displays
aging when its relaxation or time correlations are not invariant under time
translation after being aged for a long waiting time; instead, they explicitly
depend on such a time. A memory effect emerges in a physical system when its
time evolution depends on the previous history, i.e. on its initial
preparation that, in turn, depends on how it has been previously “aged”.

A classic example of memory effect is the so-called Kovacs hump, first
reported by Kovacs for the volume relaxation of polymeric glasses [4, 5].
Analogous behaviours have been repeatedly observed in different contexts [6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 30, 31]. Let us consider a quantity $P$ of a physical system in
contact with a thermal bath. Its equilibrium value is denoted by
$P_{\text{eq}}(T)$, which is assumed to be a monotonic function. The Kovacs
hump is the non-monotonic response of the system to the two-jump protocol
described below.

Figure 1: Qualitative picture of the Kovacs hump. The time evolution of a
physical quantity $P$ is depicted on the top panel, when the system is
submitted to the two-jump protocol in the temperature shown on the bottom
panel. The relaxation from $T_{i}$ to $T_{1}$ (dashed line) is interrupted at
$t=t_{w}$, when the quantity $P$ has its equilibrium value at the final
temperature $T_{f}$, $P(t_{w})=P_{\text{eq}}(T_{f})$. Nevertheless, $P(t)$
deviates from $P_{\text{eq}}(T_{f})$ and passes through a maximum before
returning thereto—thus showing the need of additional physical quantities to
completely characterise the state of the system.

Figure 1 shows a sketch of the Kovacs protocol and the associated Kovacs
response. The system is initially equilibrated at temperature $T_{i}$, and
therefrom aged at a lower temperature $T_{1}<T_{i}$ in the time interval
$0<t<t_{w}$. At $t=t_{w}$, the instantaneous value of $P$ is $P(t_{w})$, and
thereat the temperature of the bath is abruptly changed to $T_{f}$, with
$P(t_{w})=P_{\text{eq}}(T_{f})$—thus, $T_{i}>T_{f}>T_{1}$. The system displays
the Kovacs effect when, for $t>t_{w}$, $P$ departs from its equilibrium value,
which $P$ has as a consequence of the choice of $T_{f}$, and presents a non-
monotonic behaviour. The existence of this Kovacs hump entails that the pair
$(T,P)$ does not suffice to completely characterise the state of the system:
additional state variables are necessary.

Another example of memory effect is the Mpemba effect [32]. Originally, the
Mpemba effect refers to “hot” water freezing faster than “cold” water [32,
33], in contradiction with the usual Newton’s law of cooling [34]. In this
context, the very existence of the Mpemba effect is still controversial [35,
36]. Recently, the Mpemba effect has attracted the attention of the non-
equilibrium physics community, understanding it in a generalised way as
follows. The relaxation of two samples of the same system to a common final
steady state is considered. Under certain conditions, the sample initially
further from the steady state relaxes thereto faster than that initially
closer.

The Mpemba effect is qualitatively depicted in fig. 2. Both the Mpemba—the
hotter cools sooner—and the inverse Mpemba—the colder heats sooner—effects
have been observed in many different systems [37, 38, 39, 40, 41, 42, 43, 24,
44, 45, 46, 47, 48, 49, 50, 51, 30, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61,
62, 63, 64, 65]. In the theoretical studies, two main approaches have been
used: stochastic (entropic) [37] and kinetic (thermal) [38], which we describe
later in detail. In the former, distance to equilibrium is defined in
probability space, e.g. with the Kullback-Leibler divergence. In the latter,
distance to equilibrium is monitored through the kinetic temperature, which is
proportional to the average kinetic energy.111For equilibrium systems, due to
the equipartition theorem, the kinetic temperature equals the thermodynamic
temperature. This is no longer the case for out-of-equilibrium states.

Figure 2: Qualitative picture of the Mpemba memory effect. The “hot” sample
$A$, with initial kinetic temperature $T_{i,A}$, is further from equilibrium
at the common bath temperature $T_{s}$ than the “cold” sample $B$, with
initial temperature $T_{i,B}<T_{i,A}$. In the kinetic approach, the thermal
Mpemba effect emerges when the time evolution of the initially hotter sample
(red solid line) overtakes that of the initially colder one (blue dashed). In
the stochastic processes approach, the evolution of the distance in
probability space—e.g. the Kullback-Leibler divergence—is monitored instead of
the kinetic temperature, and the entropic Mpemba effect arises when a similar
crossing of the relaxation towards equilibrium of samples $A$ and $B$ is
observed.

In the Mpemba effect, the system that is further from the steady state somehow
takes a shortcut and thus relaxes thereto faster than the closer one. Then,
there appears a natural connection with the general field of shortcuts or,
employing the terminology introduced in ref. [66], swift state to state
transformations. In particular, a related problem is the optimisation of the
relaxation route to equilibrium—or to a NESS. For given initial and final
states, the minimisation of the connection time between them by engineering
the time dependence of some physical quantities, like the temperature or the
potential, is a well-defined mathematical problem in optimal control theory
[67]. This is the classic brachistochrone problem, which very recently has
been addressed for both quantum and non-equilibrium systems [68, 69, 70, 71,
72, 73, 66, 74, 75].

##  2 Kovacs effect

For systems with a master equation dynamics, there are general results for the
shape of the Kovacs hump in linear response. These results hold under quite
general conditions, basically (i) a canonical form of the equilibrium
probability distribution function (pdf), proportional to $\exp(-\beta H)$,
with $\beta=(k_{B}T)^{-1}$ and $H$ being the system’s Hamiltonian, and (ii)
detailed balance in the dynamics [14]. With these assumptions, the form of the
Kovacs hump for the energy $E(t)=\expectationvalue{H}(t)$ is directly related
to the form of its “direct” relaxation function $\phi_{E}(t)$ from $T_{i}$ to
$T_{f}$, with only one jump.

From the explicit expression of the Kovacs hump in linear response, eq. (43)
of ref. [14], one deduces that: (i) the Kovacs hump is always positive, i.e.
$E(t)\geq E_{\text{eq}}(T_{f})$, (ii) there is only one maximum of $E(t)$.
Interestingly, the explicit expression of the Kovacs hump derived in ref. [14]
resembles the phenomenological expression written by Kovacs [5]. Although the
majority of studies are done in the non-linear regime, i.e. with large values
of the temperature jumps, the behaviour described by the linear response
theory, i.e. (i) and (ii) above, a positive hump with only one maximum, is the
one found in glassy and other complex systems [4, 5, 6, 7, 8, 9, 11, 10, 12,
13, 14, 15, 16, 17, 18, 25, 26, 28, 30, 31]; thus the term “normal Kovacs
hump” has been coined to describe it. The normal hump stems from the structure
of the direct relaxation function in linear response, which is a sum of
exponentially decreasing modes with positive coefficients.

In glassy systems, the emergence of the Kovacs effect is often explained as a
consequence of the complex energy landscape typical thereof. Still, the Kovacs
effect has also been observed in systems with a much simpler energy landscape.
A paradigmatic case is that of granular gases, which are intrinsically non-
equilibrium systems: energy is purely kinetic but it is continuously
dissipated in collisions. Therefore, an external mechanism is needed to drive
the system to a stationary state, which is always a NESS with a non-Maxwellian
velocity distribution function (vdf) [76]. The simplest one is that of the
uniformly heated granular gas, in which independent white noise forces with
variance $\chi$ act on all particles. The kinetic temperature—here also called
granular temperature—at the NESS is then a certain function of $\chi$ [77,
78].

Despite the simple energy landscape, the Kovacs effect neatly appears when a
system of smooth inelastic hard particles222In smooth collisions, the
tangential component of the relative velocity is conserved, whereas the normal
component is reversed and shrunk with the restitution coefficient $\alpha$;
the energy loss is thus proportional to $1-\alpha^{2}$ and $\alpha=1$
corresponds to the elastic case. is submitted to the two-jump Kovacs protocol,
with the intensity of the driving playing the role of the bath temperature,
$\chi_{i}\to\chi_{1}\to\chi_{f}$ [19, 20]. This entails that the instantaneous
value of the kinetic temperature $T(t)$ does not suffice to completely
describe granular fluids. It is the non-Gaussianities that are responsible for
the emergence of the Kovacs effect, and it is thus essential to incorporate
them to the physical picture. It suffices to do so in the simplest way by
including only the excess kurtosis $a_{2}(t)$—the so-called first Sonine
approximation.

More interestingly, the sign of the Kovacs hump depends on the inelasticity.
Specifically, it depends on the sign of the excess kurtosis at the steady
state, $a_{2}^{\text{s}}$, which is negative (positive) for small (large)
inelasticity. The key point is the cooling rate being an increasing function
of $a_{2}$. By aging the system with a very low value of the driving
$\chi_{1}$, the system falls onto the homogeneous cooling state (HCS) [79], in
which the granular fluid freely cools following Haff’s law [80], $T(t)\propto
t^{-2}$, and the excess kurtosis becomes constant and equals
$a_{2}^{\text{HCS}}$. One always has
$\text{sgn}(a_{2}^{\text{HCS}})=\text{sgn}(a_{2}^{\text{s}})$ and
$|a_{2}^{\text{HCS}}|>|a_{2}^{\text{s}}|$—the white noise forcing diminishes
the non-Gaussian character of the vdf, thus decreasing
$\absolutevalue{a_{2}}$. Then, just after the second jump
$\chi_{1}\to\chi_{f}$ at $t=t_{w}$, despite having the “correct” kinetic
temperature $T_{f}$, the system is cooling slower (faster) than at the steady
state when $a_{2}^{\text{HCS}}-a_{2}^{\text{s}}$, or simply
$a_{2}^{\text{s}}$, is negative (positive), i.e. for small (large)
inelasticity.

For $t>t_{w}$, the discussion above entails that the kinetic temperature
$T(t)$ initially increases (decreases) and passes through a maximum (minimum)
before going back to $T_{f}$ when $a_{2}^{\text{s}}<0$ ($a_{2}^{\text{s}}>0$),
i.e. for small (large) inelasticity. Therefore, the Kovacs hump is normal,
similar to that of molecular fluids—positive and with only one maximum—for
small inelasticity, whereas the Kovacs hump turns out to be anomalous, using
the term introduced in ref. [19], for large inelasticity: negative with one
minimum. Figure 3 shows two examples of the aforementioned behaviours.

Figure 3: Kovacs memory effect for the uniformly heated granular gas. The
theoretical curves in the first Sonine approximation for both the normal
positive hump (top panel) and the anomalous negative hump (bottom) are shown
as a function of a dimensionless time $\tau$. See refs. [19, 20] for more
details and the comparison with numerical simulations.

In the granular gas, both the normal and the anomalous Kovacs effect persist
in the linear response regime [29].

The Kovacs effect has also been investigated in a granular fluid of rough
particles. In addition to inelastic, collisions have a certain degree of
roughness, i.e. the tangential component of the relative velocity is not
conserved in collisions. This induces a coupling between the translational and
rotational degrees of freedom. More complex Kovacs responses emerge, which may
involve several extrema [81].

The linear response theory for molecular systems [14] has been generalised to
athermal systems [23, 22]. Specifically, the relation between the Kovacs hump
and the direct relaxation function remains valid, but the latter is not
necessarily a sum of positive modes. It is this fact that makes it possible
the emergence of the anomalous Kovacs effect, at least in linear response
[29].

Finally, it is interesting to note that the Kovacs effect has also been
recently investigated in a variety of systems, such as active matter [23],
disordered mechanical systems [82], frictional interfaces [83], a levitated
colloidal nanoparticle [27], or fluids with non-linear drag [30].

##  3 Mpemba effect

To start with, we discuss the entropic (stochastic) Mpemba effect, triggered
by the seminal Lu and Raz’s work [37]. A mesoscopic system is considered and
its time evolution is analysed in terms of the pdf of the relevant variables,
which obeys a Markovian evolution equation (master equation, Fokker-Planck
equation, etc.) with detailed balance. Distance to equilibrium is defined in
terms of a functional of the pdf, e.g. the Kullback-Leibler divergence or
other norms like the $\mathcal{L}^{1}$ or $\mathcal{L}^{2}$ norms. By
expanding the solution of the evolution equation in the eigenfunctions of the
relevant operator, the entropic Mpemba effect is found when, under appropriate
conditions, the amplitude of the slowest relaxation mode presents a non-
monotonic dependence with the temperature [37, 39, 84, 49, 46, 58, 59, 62].
Also, a strong Mpemba effect has been reported, which arises when, by
adequately choosing the system parameters, the coefficient of the slowest
relaxation mode vanishes and the relaxation to equilibrium becomes
exponentially faster [37, 39, 84, 59].

In the thermal (kinetic) approach, started with Lasanta et al.’s analysis of a
granular gas [38], its time evolution is analysed in terms of the one-particle
vdf, which evolves following a kinetic equation—Boltzmann-Fokker-Planck,
typically. The relaxation to the steady state is monitored by the kinetic
temperature. The kinetic approach has been employed for both granular fluids
[38, 40, 43, 51, 48, 52], in which collisions between particles are inelastic,
and molecular fluids with elastic collisions but with a non-linear drag force
[44, 30, 57, 52]. The former relaxes to a NESS that is characterised by the
intensity of the driving applied to balance, in average, the energy dissipated
in collisions; the latter relaxes to a true equilibrium state with a
Maxwellian vdf.

The Mpemba effect has also been studied in spin glasses within the thermal
approach; it is the internal energy that displays the crossing therein [41].
The Mpemba effect is present only in the spin-glass phase and stems from the
aging dynamics of the internal energy, which is controlled by the non-
equilibrium coherence length. Interestingly, this suggests that the Mpemba
effect can be considered as an effective probe for the existence of a glass
transition.

There are some key differences between the stochastic and kinetic approaches.
On the one hand, the monitored quantity in the kinetic approach, the kinetic
temperature (or the energy), is much closer to an experimentally measurable
quantity than the abstract distance between distributions employed in the
stochastic approach.333For granular gases, the velocity fluctuations of an
immersed rotating blade may be used as a proxy of the kinetic temperature
[85]. In addition, the thermal Mpemba effect typically takes place for short
times, far away from the final state—which makes it easier to be observed, in
principle. On the other hand, the initial conditions in the kinetic approach
must be non-stationary and thus, essentially, non-trivial to
implement—although for non-linear fluids it has been discussed the aging
procedure to obtain these initial conditions, which correspond to a long-
lived, metastable, non-equilibrium state [30]; whereas the initial conditions
for the entropic Mpemba effect in the stochastic approach are equilibrium
states.444Initial stationary conditions have also been considered in the
granular case, but an unrealistic asymmetric driving mechanism has to be
introduced to trigger the Mpemba effect[48, 55].

Now we focus on the kinetic approach. Following Prados and Trizac’s study of
the Kovacs effect [19, 20], the first Sonine approximation was employed to
analyse the emergence of the Mpemba effect in a granular fluid by
incorporating non-Gaussianities to the picture [38]. Indeed, it is the non-
Gaussian vdf that makes the Mpemba effect possible: if it were Gaussian, the
kinetic temperature would obey a closed first-order differential equation,
without additional variables, and neither the Mpemba effect nor any other
memory effect would emerge.

Figure 4: Mpemba memory effect for the uniformly heated granular gas. The
dimensionless temperature $\theta\equiv T/T_{\text{s}}$ is plotted as a
function of a dimensionless time $\tau$. Both the Mpemba and the inverse
Mpemba effects are shown—as in fig. 3, only the theoretical curves in the
first Sonine approximation. See ref. [38] for the comparison with numerical
simulations.

In fig. 4, specific examples of both the Mpemba effect, for
$T_{i,A}>T_{i,B}>T_{s}$, and the inverse Mpemba effect, for
$T_{s}>T_{i,A}>T_{i,B}$, are shown. The hot sample A is prepared in an initial
state with kinetic temperature $T_{i,A}$ and excess kurtosis $a_{2,i}^{A}$,
and cools down to a NESS corresponding to a certain value of the driving
$\chi_{\text{s}}$ following the dynamical curve $T_{A}(t)$ (red solid line).
The cold sample is prepared in an initial state with kinetic temperature
$T_{i,B}<T_{i,A}$ and excess kurtosis $a_{2,i}^{B}$, and also cools down to
the same NESS following the dynamical curve $T_{B}(t)$ (blue dashed). Again,
the key point is the cooling rate increasing with $a_{2}$: if
$a_{2,i}^{A}>a_{2,i}^{B}$, the difference of the initial cooling rates may
become large enough to facilitate the crossing of the corresponding time
evolutions $T_{A}(t)$ and $T_{B}(t)$—at least for small enough kinetic
temperature difference $\Delta T_{i}\equiv T_{i,A}-T_{i,B}$. As the initial
states are not stationary states, the initial values of the kurtosis $a_{2,i}$
can be tuned to bring the Mpemba effect about.

As the difference of the initial kurtosis $\Delta a_{2,i}\equiv
a_{2,i}^{A}>a_{2,i}^{B}$ increases, the range of initial temperatures $\Delta
T_{i}\equiv T_{i,A}-T_{i,B}$ for which the Mpemba effect is observed
increases. Since the cooling rate depends on the inelasticity $\alpha$, the
range of temperatures for which the Mpemba effect emerges depends on the
inelasticity as well; decreasing with it and vanishing in the elastic limit
$\alpha\to 1$.

The thermal Mpemba effect has also been investigated for a gas of inelastic
rough hard spheres [40]. Therein, the Mpemba effect is giant, much larger than
in the smooth granular gas. The initially hotter sample may cool sooner, even
when the initial temperatures differ by more than one order of magnitude. The
largeness of the memory effect stems from the coupling between the
translational and rotational temperatures, which are of the same order—in the
smooth case, the Mpemba effect stemmed from the coupling with the (quite
small) non-Gaussianities.

It is interesting to note that the Mpemba effect has also recently been found
in a molecular fluid, in which the collisions between particles are elastic,
with non-linear drag $\zeta(v)=\zeta_{0}(1+\gamma\,mv^{2}/2k_{B}T_{s})$ [44,
30, 57]. The non-linearity is measured by a dimensionless parameter $\gamma$,
and the relevance of collisions by a dimensionless collision rate $\xi^{-1}$
($\xi=\infty$ thus corresponds to the collisionless case.) The kinetic
temperature is not constant due to the interactions with the thermal
bath—modelled as a background fluid of particles with comparable mass [86, 87,
88]. The non-linearity of the drag implies that the evolution equation for the
temperature is coupled to higher-order cumulants of the vdf, bringing about
the possible emergence of memory effects.

One key question, unanswered in previous studies of granular fluids [38, 40],
is the aging procedure that gives rise to the specific initial non-equilibrium
conditions that maximise the Mpemba effect. Remarkably, it is possible to give
an answer for the non-linear fluid: the hot sample must be prepared by heating
it from a much lower temperature, whereas the cold sample must be prepared by
cooling it from a much higher temperature. The high-temperature quench of the
cold sample makes it fall in a long-lived far-from-equilibrium state [89],
over which the kinetic temperature follows a very slow decay to equilibrium,
which (i) increases the magnitude of the Mpemba effect and (ii) makes it
universal—in the sense that the curves corresponding to different initial
temperatures, non-linearity $\gamma$, and collision rate $\xi$ collapse onto a
unique master curve upon a suitable rescaling, see fig. 5 [30].

Figure 5: Universal Mpemba memory effect for the molecular fluid with non-
linear drag. The dimensionless temperature is again $\theta=T/T_{s}$. Several
cold samples B (filled blue symbols) for different values of
$(\theta_{i,B},\gamma,\xi)$ are shown together with the corresponding hot
samples A (empty red), for a fixed ratio of initial temperatures
$\theta_{i,A}/\theta_{i,B}=1.1$. When plotted as a function of a dimensionless
time $s_{B}=\gamma\theta_{i,B}\zeta_{0}t$, all curves collapse onto a
universal behaviour.

##  4 Optimal control

What is the fastest relaxation route between two given states, either
equilibrium, NESSs, or arbitrary ones? In general, this is the problem of the
brachistochrone, which has recently been addressed in different physical
contexts [68, 69, 70, 71, 72, 73, 66, 74, 75]. It is tempting to relate this
problem with the Mpemba effect, since the relaxation from the initially
further from equilibrium state overtaking that of the initially closer may be
interpreted as the former finding a shortcut to the common final state.

The thermal brachistochrone has been recently investigated in uniformly driven
granular fluids [71, 72]. It refers to the minimum time connection by
controlling the intensity of the stochastic forcing $\chi$. The protocols
minimising the connection time between the initial and final NESSs
corresponding to initial and final kinetic temperatures $T_{i}$ and $T_{f}$
are of bang-bang type, i.e. they comprise different time intervals in which
the thermostat alternates between its maximum and minimum available values
[67].

In the granular fluid, the time over the brachistochrone $t_{f}$ typically
beats the experimental relaxation time $t_{R}$ by at least one order of
magnitude—see fig. 6. Remarkably, in the usual relaxation experiment with a
sudden step at $t=0$, the relaxation is never complete in a finite time—the
empirical relaxation time is defined by estimating that the system is close
“enough” to the final state. On the contrary, over the brachistochrone, the
system reaches exactly the final state in a finite time.

A similar situation, with the thermal brachistochrone given by bang-bang
protocols is found in Fokker-Planck systems [71, 73]. The case of coupled
harmonic oscillators that are driven from an initial equilibrium state at
temperature $T_{i}$ to a final equilibrium state at temperature $T_{f}$ has
been analysed in detail, and an unexpected discontinuity of the minimum
connection time with increasing dimension has been unveiled [73].

Figure 6: Acceleration factor over the brachistochrone for the granular gas.
We plot the ratio $t_{R}/t_{f}$ of the experimental relaxation time to the
brachistochrone time as a function of the ratio $T_{f}/T_{i}$ of the final to
the initial temperature, for $\alpha=0.3$ (solid line) and $\alpha=0.8$
(dashed)—see ref. [71] for details.

##  5 Discussion

We have reviewed the emergence of non-equilibrium memory effects, mainly in
granular fluids and fluids with non-linear drag. Despite being quite different
from a fundamental point of view—collisions in granular fluids are inelastic,
so they are intrinsically out-of-equilibrium systems with non-Gaussian vdfs
even in the stationary state, both types of systems display the Kovacs and the
Mpemba memory effects. Still, one key difference between granular and non-
linear fluids is the emergence of the anomalous Kovacs effect in the former.
Even when a non-linear drag is present, the Kovacs effect is always normal
when the stationary state corresponds to equilibrium and the dynamics verify
detailed balance.

The existence of these memory effects in the relaxation of the kinetic
temperature, proportional to the average kinetic energy, of granular and non-
linear fluids stems from its evolution being coupled to additional variables,
higher-order cumulants of the velocity that measure the deviation of the vdf
from the Gaussian shape. In other words, the kinetic temperature does not
suffice to univocally determine the macroscopic state of the system. Hence, it
is essential in general to keep track of the non-Gaussianities to understand
the non-equilibrium behaviour.

The thermal and entropic approaches to the Mpemba effect have been scarcely
compared [90, 62]. In ref. [90], it was shown that the thermal Mpemba effect
may appear without its entropic counterpart—or vice versa—in a molecular fluid
with non-linear drag. Therein, some situations appear in which the kinetic
temperature overshoots the stationary value, which makes it necessary to
revise the usual definition of the thermal Mpemba effect in this scenario. The
authors of ref. [90] propose a separation of the Kullback-Leibler divergence
into a “kinetic” contribution plus a “local-equilibrium” distribution that
allows for defining a non-equilibrium temperature, not necessarily associated
with the average kinetic temperature, for any system relaxing to equilibrium.
It seems worth exploring if this line of thought could lead to a unique
framework for the thermal and entropic Mpemba effects.

Much progress has been made in the understanding of these memory effects.
However, there is still room for further work in this appealing line of
research. One perspective is related to their optimal control, e.g. for
maximising the “positive” consequences of a (tailored) preparation of the
initial state—as in the Mpemba effect. Therein, it seems also worth
investigating possible connections between the Mpemba effect and the
optimisation of the relaxation route to equilibrium, which has attracted a lot
of attention recently from different perspectives: e.g. the impact of a
precooling strategy [91] or the possible asymmetry between heating and cooling
[92, 93, 94].

###### Acknowledgements.

We acknowledge financial support from Grant PID2021-122588NB-I00 funded by
MCIN/AEI/10.13039/ 501100011033/ and by “ERDF A way of making Europe”, and
also from Grant ProyExcel_00796 funded by Junta de Andalucía’s PAIDI 2020
programme. A. Patrón acknowledges support from the FPU programme through Grant
FPU2019-4110. C. A. Plata acknowledges the funding received from EU Horizon
Europe–Marie Skłodowska-Curie 2021 programme through the Postdoctoral
Fellowship with ref. 101065902 (ORION). We are indebted with all the people
with whom we have collaborated in this exciting field of memory effects.

## References

  * [1] Cugliandolo L. F., Kurchan J. Ritort F. Physical Review B4919946331.
  * [2] Keim N. C., Paulsen J. D., Zeravcic Z., Sastry S. Nagel S. R. Reviews of Modern Physics912019035002.
  * [3] Jaeger T. Simmons D. Journal of Chemical Physics1562022114504.
  * [4] Kovacs A. J. Fortschritte Der Hochpolymeren-Forschung31963394.
  * [5] Kovacs A. J., Aklonis J. J., Hutchinson J. M. Ramos A. R. Journal of Polymer Science: Polymer Physics Edition1719791097.
  * [6] Chow T. S. The Journal of Chemical Physics7919834602.
  * [7] Berthier L. Holdsworth P. C. EPL58200235.
  * [8] Buhot A. Journal of Physics A: Mathematical and General36200312367.
  * [9] Bertin E. M., Bouchaud J. P., Drouffe J. M. Godrèche C. Journal of Physics A: Mathematical and General36200310701.
  * [10] Arenzon J. J. Sellitto M. The European Physical Journal B-Condensed Matter and Complex Systems422004543.
  * [11] Cugliandolo L. F., Lozano G. Lozza H. The European Physical Journal B-Condensed Matter and Complex Systems41200487.
  * [12] Mossa S. Sciortino F. Physical Review Letters922004045504.
  * [13] Aquino G., Leuzzi L. Nieuwenhuizen T. M. Physical Review B732006094205.
  * [14] Prados A. Brey J. J. Journal of Statistical Mechanics: Theory and Experiment2010P02009.
  * [15] Bouchbinder E. Langer J. S. Soft Matter620103065.
  * [16] Diezemann G. Heuer A. Physical Review E832011031505.
  * [17] Chang Y.-C., Wang C., Yin S., Hoffman R. C. Mott A. G. Optics Express21201317760.
  * [18] Ruiz-García M. Prados A. Physical Review E892014012140.
  * [19] Prados A. Trizac E. Physical Review Letters1122014198001.
  * [20] Trizac E. Prados A. Physical Review E902014012204.
  * [21] Brey J., De Soria M., Maynar P. Buzón V. Physical Review E902014.
  * [22] Plata C. A. Prados A. Entropy192017539.
  * [23] Kürsten R., Sushkov V. Ihle T. Physical Review Letters1192017188001.
  * [24] Mompó E., López-Castaño M. A., Lasanta A., Vega Reyes F. Torrente A. Physics of Fluids332021062005.
  * [25] Peyrard M. Garden J.-L. Physical Review E1022020052122.
  * [26] Mandal R., Tapias D. Sollich P. Physical Review Research32021043153.
  * [27] Militaru A. et al. Physical Review Letters1272021130603.
  * [28] Lulli M., Lee C.-S., Zhang L.-H., Deng H.-Y. Lam C.-H. Journal of Statistical Mechanics: Theory and Experiment2021093303.
  * [29] Sánchez-Rey B. Prados A. Physical Review E1042021024903.
  * [30] Patrón A., Sánchez-Rey B. Prados A. Physical Review E1042021064127.
  * [31] Godrèche C. Luck J.-M. Journal of Physics A: Mathematical and Theoretical552022495001.
  * [32] Mpemba E. B. Osborne D. G. Physics Education41969172.
  * [33] Jin J. Goddard W. A. The Journal of Physical Chemistry C11920152622.
  * [34] Maruyama S. Moriya S. International Journal of Heat and Mass Transfer1642021120544.
  * [35] Burridge H. C. Linden P. F. Scientific Reports6201637665.
  * [36] Burridge H. Hallstadius O. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences476202020190829.
  * [37] Lu Z. Raz O. Proceedings of the National Academy of Sciences11420175083.
  * [38] Lasanta A., Vega Reyes F., Prados A. Santos A. Physical Review Letters1192017148001.
  * [39] Klich I., Raz O., Hirschberg O. Vucelja M. Physical Review X92019021060.
  * [40] Torrente A. et al. Physical Review E992019060901(R).
  * [41] Baity-Jesi M. et al. Proceedings of the National Academy of Sciences116201915350.
  * [42] Yang Z.-Y. Hou J.-X. Physical Review E1012020052106.
  * [43] Biswas A., Prasad V. V., Raz O. Rajesh R. Physical Review E1022020012906.
  * [44] Santos A. Prados A. Physics of Fluids322020072010.
  * [45] Carollo F., Lasanta A. Lesanovsky I. Physical Review Letters1272021060401.
  * [46] Chétrite R., Kumar A. Bechhoefer J. Frontiers in Physics92021654271.
  * [47] Uskoković E., Uskoković T., Wu V. Uskoković V. Substantia4202159.
  * [48] Biswas A., Prasad V. V. Rajesh R. Europhysics Letters136202146001.
  * [49] Busiello D. M., Gupta D. Maritan A. New Journal of Physics232021103012.
  * [50] Takada S., Hayakawa H. Santos A. Physical Review E1032021032901.
  * [51] Gómez González R., Khalil N. Garzó V. Physics of Fluids332021053301.
  * [52] Megías A. Santos A. Frontiers in Physics102022971671.
  * [53] Zhang S. Hou J.-X. Physical Review E1062022034131.
  * [54] Degünther J. Seifert U. EPL139202241002.
  * [55] Biswas A., Prasad V. Rajesh R. Journal of Statistical Physics186202245.
  * [56] Lin J., Li K., He J., Ren J. Wang J. Physical Review E1052022014104.
  * [57] Megías A., Santos A. Prados A. Physical Review E1052022054140.
  * [58] Holtzman R. Raz O. Communications Physics52022280.
  * [59] Kumar A., Chétrite R. Bechhoefer J. Proceedings of the National Academy of Sciences1192022e2118484119.
  * [60] Yang Z.-Y. Hou J.-X. Physical Review E1052022014119.
  * [61] Chorazewski M., Wasiak M., Sychev A., Korotkovskii V. Postnikov E. Journal of Solution Chemistry2023 doi: 10.1007/s10953-023-01268-1.
  * [62] Biswas A., Rajesh R. Pal A. The Journal of Chemical Physics1592023044120.
  * [63] Sun C., Huang Y., Zhang X., Ma Z. Wang B. Physics Reports99820231.
  * [64] Chatterjee A. K., Takada S. Hayakawa H. Physical Review Letters1312023080402.
  * [65] Teza G., Yaacoby R. Raz O. Physical Review Letters1312023017101.
  * [66] Guéry-Odelin D., Jarzynski C., Plata C. A., Prados A. Trizac E. Reports on Progress in Physics862023035902.
  * [67] Liberzon D. Calculus of Variations and Optimal Control Theory: A Concise Introduction (Princeton University Press) 2012.
  * [68] Deffner S. Campbell S. Journal of Physics A: Mathematical and Theoretical502017453001.
  * [69] Plata C. A., Guéry-Odelin D., Trizac E. Prados A. Physical Review E1012020032129.
  * [70] Lam M. R. et al. Physical Review X112021011035.
  * [71] Prados A. Physical Review Research32021023128.
  * [72] Ruiz-Pino N. Prados A. Entropy242022131.
  * [73] Patrón A., Prados A. Plata C. A. The European Physical Journal Plus13720221011.
  * [74] Aghion E. Green J. R. Journal of Physics A: Mathematical and Theoretical56202305LT01.
  * [75] Pires L. B. et al. Phys. Rev. Lett.1312023097101.
  * [76] Pöschel T. Luding S. (Editors) Granular Gases Lecture Notes in Physics 564 (Springer-Verlag Berlin Heidelberg) 2001.
  * [77] Van Noije T. P. C. Ernst M. H. Granul. Matter1199857.
  * [78] Montanero J. M. Santos A. Granular Matter2200053.
  * [79] Brey J. J., Ruiz-Montero M. J. Cubero D. Physical Review E5419963664.
  * [80] Haff P. K. Journal of Fluid Mechanics1341983401.
  * [81] Lasanta A., Reyes F. V., Prados A. Santos A. New Journal of Physics212019033042.
  * [82] Lahini Y., Gottesman O., Amir A. Rubinstein S. M. Physical Review Letters1182017085501.
  * [83] Dillavou S. Rubinstein S. M. Physical Review Letters1202018224101.
  * [84] Kumar A. Bechhoefer J. Nature584202064.
  * [85] Scalliet C., Gnoli A., Puglisi A. Vulpiani A. Physical Review Letters1142015198001.
  * [86] Ferrari L. Chemical Physics336200727.
  * [87] Ferrari L. Chemical Physics4282014144.
  * [88] Hohmann M., Kindermann F., Lausch T., Mayer D., Schmidt F., Lutz E. Widera A. Physical Review Letters1182017263401.
  * [89] Patrón A., Sánchez-Rey B., Trizac E. Prados A. arXiv: 2307.04728 (2023).
  * [90] Megías A. Santos A. Entropy2420221436.
  * [91] Gal A. Raz O. Physical Review Letters1242020060602.
  * [92] Lapolla A. Godec A. Physical Review Letters1252020110602.
  * [93] Van Vu T. Hasegawa Y. Physical Review Research32021043160.
  * [94] Ibáñez M., Dieball C., Lasanta A., Godec A. Rica R. A. arXiv:2302.09061 (2023).


# Constructing a NFT Price Index and Applications

Hugo Schnoering  Napoleon Group
<EMAIL_ADDRESS>Hugo Inzirillo  Napoleon Group
CREST - Institut Polytechnique de Paris
<EMAIL_ADDRESS>
###### Abstract

We are witnessing the emergence of a new digital art market, the art market
3.0. Blockchain technology has taken on a new sector which is still not well
known, Non-Fungible tokens (NFT). In this paper we propose a new methodology
to build a NFT Price Index that represents this new market on the whole. In
addition, this index will allow us to have a look on the dynamics and
performances of NFT markets, and to diagnose them.

##  I Introduction

Last year has been the year of the democratization of NFTs. The number of
owners (figure 1) and the number of transactions (figure 2) on Ethereum have
been exponentially increasing since January 2021. Moreover, the increasing
interest of famous brand names for them also suggests that their adoption
keeps growing. Online marketplaces like Opensea (https://opensea.io) or
LooksRare (https://looksrare.org) have become focal points for trading these
new assets.

Figure 1: Cumulative number of wallet that have ever owned an ETH ERC-721 or
ERC-1155 NFT [1].

NFTs or Non-Fungible Tokens are transferable assets secured by a blockchain. A
blockchain is an ordered list of blocks that are linked together using
cryptography. Blocks contain data about transactions, and are validated and
added to the chain through a consensus protocol. Blockchain are used as public
transaction ledgers of most cryptocurrencies. For instance bitcoin or ethereum
respectively with the network Bitcoin [2] or Ethereum [3]. The need for
consensus of blockchain solves the double-spending problem using a
cryptographic proof instead of a trusted central authority.

Designed as a medium of exchange, cryptocurrencies are transferable assets as
well. Because they are defined by their value, cryptocurrencies are perfectly
fungible assets, for example a coin can be substituted to another. In
opposition to cryptocurrencies, NFTs are uniquely identified by an id and a
set of properties, and cannot be interchangeable or divisible. A NFT is thus
the perfect way to represent anything unique on blockchains. They can be used
for several purposes, but are often considered as digital art due to their
properties. As art, most collectors acquire NFTs for aesthetic reasons,
acquiring a social status or as a mean of investment [4].

Figure 2: Cumulative number of NFT traded volumes in USD [1].

NFTs are created or minted by the execution of a smart contract setting in
stone the creation of tokens. Several NFTs can be minted from the same
contract, a collection is defined as a set of tokens minted from the same
contract. Tokens minted from the same contract collect remain however
perfectly identifiable and may differ from each others by their id and their
properties or traits. To illustrate this, we represent figures 4 and 4
respectively the tokens CryptoPunk #1463 and CryptoPunk #1466 of the Ethereum
collection CryptoPunk.

Figure 3: CryptoPunk #1463. Properties and corresponding frequencies in the
collection: male (60%), cigarette (10%), earring (25%), Mohawk thin (4%).

Figure 4: CryptoPunk #1466. Properties and corresponding frequencies in the
collection: male (60%), stringy hair (5%), clown eyes green (4%).

Besides exceptional trade volumes, sales of NFTs have reached record prices,
sometimes up to several tens of millions of dollars [5, 6]. Record breaking
sales thus suggest a price explosion, but the dynamics of NFT markets are more
complex. Indeed, if some top collections are steadily becoming more and more
popular and expensive, e.g. the Ethereum CyberPunks and Bored Ape Yacht Club,
prices of most collections vary widely. For this reason, it is difficult to
have a global vision of the NFT market. In this work, we undertake the hedonic
model to build a price index from thousands of NFT transaction records.
Subsequent applications presented in this study include:

  * •

the detection of price explosion periods through statistical tests

  * •

the computation of the correlation matrix between the NFT returns with the
return of the cryptocurrencies market

  * •

the detection of undervalued and overvalued assets

The hedonic regression framework is a well known model in Economics. Hedonic
models decompose an item into its core characteristics and study the
contribution of each of them to its value. These models are commonly used in
Real Estate Economics [7, 8]. A hedonic index is an index computed from a
fitted hedonic regression model, Triplett reviews in [9] the different
methodologies that can be used. Hedonic indices are often used as proxies of
price indices, such as consumer price indices (CPI). A price index represents
the aggregate price of a basket of items, and tracks how the prices of these
items, taken as a whole, change over time. The hedonic methodology is
particularly appraised because it can be applied to illiquid markets.
Moreover, it can even handle the removal, replacement and addition of items
over time. For all these reasons, several previous studies have applied it to
diagnose art markets [10, 4, 11]. Some works also used the hedonic model to
price NFTs, but focused on a single collection: CryptoKitties [12],
CryptoPunks [13] or Decentraland [14]. To the best of our knowledge, our
article is the first proposal of a global NFT price index.

##  II Methods

We will use the following notations:

  * •

$\mathcal{C}$ is the set of collections

  * •

$\mathcal{A}$ is the set of assets

  * •

$\\{0,1,...,T\\}$ is the set of sale dates

  * •

$I(a,b)\triangleq\\{a,a+1,...,b-1,b\\}$

  * •

$I(b)\triangleq I(0,b)$

###  II-A Definitions

An asset $a$ is defined by a collection $C^{a}$, a token id $i^{a}$ and a
dictionary of traits $\mathcal{P}^{a}$. $\mathcal{P}^{a}$ is a set of (trait,
value) characterizing $a$ within its collection. $\mathcal{P}^{a}$ is usually
used to quantify the scarcity of $a$ w.r.t. other assets of $C^{a}$, and,
then, to price $a$. Intuitively the scarcer an asset is, the more expensive it
should be. It is therefore straightforward to deal with frequencies of traits
within a collection when it comes to build a pricing model. Given an asset
$a$, a couple $(p,v)\in\mathcal{P}^{a}$, the frequency of the value $v$ for
the trait $p$ is defined as follows:

$f^{a}_{(p,v)}\triangleq\frac{1}{|C^{a}|}\sum_{a^{\prime}\in
C^{a}}\sum_{(p^{\prime},v^{\prime})\in\mathcal{P}^{a^{\prime}}}\mathrm{1}_{p^{\prime}=p}\mathrm{1}_{v^{\prime}=v}$
(1)

Since the number of traits may differ from an asset to another, even within
the same collection, we construct three aggregate quantities: the minimum
frequency $f^{a}_{\mathrm{min}}$ (equation 2), the average frequency
$f^{a}_{\mathrm{avg}}$ (equation 3), and the maximum frequency
$f^{a}_{\mathrm{max}}$ (equation 4). All these quantities are set equal to 1
if $\mathcal{P}^{a}$ is empty.

$f^{a}_{\min}\triangleq\min_{(p,v)\in\mathcal{T}^{a}}f^{a}_{(p,v)}$ (2)

$f^{a}_{\mathrm{avg}}\triangleq\frac{1}{|\mathcal{P}^{a}|}\sum_{(p,v)\in\mathcal{P}^{a}}f^{a}_{(p,v)}$
(3)

$f^{a}_{\max}\triangleq\max_{(p,v)\in\mathcal{P}^{a}}f^{a}_{(p,v)}$ (4)

A sale is defined by an asset $a$, a date $t$ and a price $P^{a}_{t}$ in USD
equivalent.

###  II-B Pricing model

The pricing model that we develop in this section is the result of three
empiric observations:

  * •

as mentioned in the introduction, the price of an asset is impacted by the
popularity and hype around its collection,

  * •

assets within the same collection may differ from each others by their traits,
the scarcer the traits of an asset are, the more appreciated it is in its
collection,

  * •

the NFT market seems to experience periods during which prices of NFT are
globally impacted positively or negatively.

We finally come up with the multiplicative pricing model of equation (5).

$P^{a}_{t}=P\times f(C^{a})\times g(a)\times h(t)\times\epsilon(a,t)$ (5)

where:

  * •

$P\in\mathbb{R}^{+*}$ defines a scale price,

  * •

$f:\mathcal{C}\rightarrow\mathbb{R}^{+*}$ impacts the price of an asset
according to its collection,

  * •

$g:\mathcal{A}\rightarrow\mathbb{R}^{+*}$ impacts the price of an asset
according to the scarcity of its traits within its collection,

  * •

$h:\\{0,1,...,T\\}\rightarrow\mathbb{R}^{+*}$ impacts the prices according to
the global state of the NFT market,

  * •

$\epsilon:\\{0,1,...,T\\}\times\mathcal{A}\rightarrow\mathbb{R}^{+*}$ is a
noise term explaining price fluctuations.

We model $f$, $g$, $h$ and $\epsilon$ as follows:

$f:C\rightarrow\exp\sum_{C^{{}^{\prime}}}\alpha_{C^{{}^{\prime}}}\mathrm{1}_{C^{{}^{\prime}}=C}$
(6)

$g:a\rightarrow\exp\left(\beta_{\min}f^{a}_{\min}+\beta_{\mathrm{avg}}f^{a}_{\mathrm{avg}}+\beta_{\max}f^{a}_{\max}\right)$
(7)

$h:t\rightarrow\exp\sum_{t^{{}^{\prime}}=0}^{T}\gamma_{t^{{}^{\prime}}}\mathrm{1}_{t^{{}^{\prime}}=t}$
(8)

$\epsilon:(a,t)\rightarrow\exp\chi(a,t)$ (9)

where $\chi(a,t)\overset{i.i.d.}{\sim}\mathrm{Normal}(0,\sigma^{2})$.

Equivalently,

$\begin{split}\log P^{a}_{t}&=\log
P+\sum_{C^{{}^{\prime}}}\alpha_{C^{{}^{\prime}}}\mathrm{1}_{C^{{}^{\prime}}=C}\\\
&+\beta_{\min}f^{a}_{\min}+\beta_{\mathrm{avg}}f^{a}_{\mathrm{avg}}+\beta_{\max}f^{a}_{\max}\\\
&+\sum_{t^{{}^{\prime}}=0}^{T}\gamma_{t^{{}^{\prime}}}\mathrm{1}_{t^{{}^{\prime}}=t}+\chi(t,a)\end{split}$
(10)

The model of equation (10) is also known as the time dummy variable version of
the hedonic regression model [9]. Hedonic coefficients $P$,
$\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are inferred only once using
all dates $\\{0,1,...,T\\}$, this approach is called the multi-period pooled
regression [9].

###  II-C Price Index Construction

Since we used the time dummy variable method to train our pricing model, the
hedonic index $I$ is defined as follows [9]:

$\frac{I_{t+1}}{I_{t}}=\frac{\exp\gamma_{t+1}}{\exp\gamma_{t}}$ (11)

As a result,

$\small\begin{split}I_{t}&=A\prod_{t^{{}^{\prime}}=0}^{t-1}\left(1+\left(\frac{I_{t^{\prime}+1}}{I_{t^{\prime}}}-1\right)\right)\\\
&=A\prod_{t^{{}^{\prime}}=0}^{t-1}\left(1+\left(\frac{\exp\gamma_{t^{{}^{\prime}}+1}}{\exp\gamma_{t^{{}^{\prime}}}}-1\right)\right)\\\
&=A\prod_{t^{{}^{\prime}}=0}^{t-1}\frac{\exp\gamma_{t^{{}^{\prime}}+1}}{\exp\gamma_{t^{{}^{\prime}}}}\\\
&=A\exp(\gamma_{t}-\gamma_{0})\end{split}$ (12)

Thus, the return of $I$ between $t$ and $t+1$ is
$\frac{\exp\gamma_{t^{{}^{\prime}}+1}}{\exp\gamma_{t^{{}^{\prime}}}}-1$.

The construction of $I$ can be justified as follows, if $I$ is proportional to
the geometric mean of prices over all assets of $\mathcal{A}$, then:

$\small\begin{split}\frac{I_{t+1}}{I_{t}}&=\frac{\prod_{a\in\mathcal{A}}(P^{a}_{t+1})^{\frac{1}{|\mathcal{A}|}}}{\prod_{a\in\mathcal{A}}(P^{a}_{t})^{\frac{1}{|\mathcal{A}|}}}\\\
&=\prod_{a\in\mathcal{A}}\left(\frac{P^{a}_{t+1}}{P^{a}_{t}}\right)^{\frac{1}{|\mathcal{A}|}}\\\
&=\frac{\exp\gamma_{t+1}}{\exp\gamma_{t}}\prod_{a\in\mathcal{A}}\exp\left(\chi(a,t+1)-\chi(a,t)\right)^{\frac{1}{|\mathcal{A}|}}\\\
&=\frac{\exp\gamma_{t+1}}{\exp\gamma_{t}}\exp\left(\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}}\chi(a,t+1)-\chi(a,t)\right)\end{split}$
(13)

Since
$\chi(a,t+1)-\chi(a,t)\overset{i.i.d.}{\sim}\mathrm{Normal}(0,2\sigma^{2})$,
according to the strong law of large numbers:

$\small\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}}\chi(a,t+1)-\chi(a,t)\underset{|\mathcal{A}|\rightarrow+\infty}{\rightarrow}0,\
\mathrm{a.s.}$ (14)

Thus,

$\small\exp\left(\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}}\chi(a,t+1)-\chi(a,t)\right)\underset{|\mathcal{A}|\rightarrow+\infty}{\rightarrow}1,\
\mathrm{a.s.}$ (15)

because exp is continuous. Finally, if $\mathcal{A}$ is large enough:

$\small\frac{I_{t+1}}{I_{t}}\approx\frac{\exp\gamma_{t+1}}{\exp\gamma_{t}}$
(16)

###  II-D Bubble Detection

A bubble is often defined as a period of explosive or mildly explosive
behavior in the price dynamic [15]. In particular, autoregressive dynamics can
be observed during speculative bubbles. For this reason, unit root tests are
useful for making a market diagnosis, e.g. for detecting market excesses or
mispricing. Augmented Dickey-Fuller (ADF) tests are usually used to determine
whether a series $y$ is stationary or not, but they can also be used to detect
explosive behaviors. Under the null hypothesis $H_{0}$, the process is
autoregressive and has an unit root. Under the alternative hypothesis $H_{1}$,
the process has an explosive root. Hypothesis $H_{0}$ is wide and must be
specified in order to derive an asymptotic distribution and the critical
values useful for the ADF test. To this end, Phillips et al. assume that the
process $y$ is a random walk with an unit root and an asymptotically
negligible drift [16], i.e.:

$\small y(t+1)=d(T+1)^{-\eta}+\theta y(t)+\epsilon(t)$ (17)

where $\theta=1$, $\eta>1/2$ and
$\epsilon(t)\overset{i.i.d.}{\sim}\mathrm{Normal}(0,\sigma^{2})$.

The ADF testing procedure is applied to the regression model of equation (18).

$\small\begin{split}\Delta y(t)=\mu+\nu y(t-1)+\sum_{i=1}^{k}\psi^{i}\Delta
y(t-i)+\epsilon(t)\end{split}$ (18)

where $\Delta y(t)=y(t)-y(t-1)$, $k\in\mathbb{N}$ is a lag order and
$\epsilon\overset{i.i.d.}{\sim}\mathrm{Normal}(0,\sigma^{2})$. The ADF
statistic is defined as the t-statistic of the coefficient $\nu$ of the
regression model. Under the null hypothesis (equation 17), the process $y$ has
an unit root, thus $\nu=0$. Under the alternative hypothesis of an explosive
root, $\nu>0$. For this reason, we use the right-sided version of the ADF
test, i.e. if ADF exceeds the critical value : $H_{0}$ is rejected and $H_{1}$
accepted, else : $H_{0}$ is accepted and $H_{1}$ rejected

In order to detect local bubbles and not to be fooled by pseudo-stationary
behaviors, the ADF test can be performed on continuous sub-sample of $y$. We
denote by $ADF_{t_{1}\rightarrow t_{2}}$ the ADF statistic computed on
$\\{y_{t_{1}},y_{t_{1}+1},...,y_{t_{2}}\\}$. Phillips et al. develop in [16] a
methodology to detect multiple bubbles. For this purpose, they derive the
BSADF statistic from the ADF statistic (equation 19).

$\small BSDAF(t)=\sup_{t^{{}^{\prime}}\in
I(0,t-w)}ADF_{t^{{}^{\prime}}\rightarrow t}$ (19)

where $w$ is a hyper-parameter defining the minimum window size on which ADF
tests can be performed. The beginning and the end of a bubble, denoted
respectively by $t_{b}$ and $t_{e}$, are detected as follows:

$t_{b}\triangleq\inf\\{t\in I(w,T),BSDAF(t)>v^{c}_{w}(t)\\}$ (20)
$t_{e}\triangleq\inf\\{t\in I(t_{b}+\delta,T),BSDAF(t)<v^{c}_{w}(t)\\}$ (21)

where $\delta$ is the minimum duration of a bubble and $v^{c}_{w}(t)$ is a
critical value defined as the $(1-c)$ quantile of the distribution of the
statistic $\sup_{t^{{}^{\prime}}\in\\{w,...,t\\}}ADF_{0\rightarrow
t^{{}^{\prime}}}$. Critical values $v^{c}_{w}(t)$ can be estimated using
Monte-Carlo simulations.

##  III Data

Several blockchains support NFTs including Ethereum, Solana, Flow, Tezos or
Polygon. However, most of the NFT trade volume is concentrated on Ethereum
(figure 5). For this reason, we decide to focus our work on Ethereum NFT
collections, more precisely on NFTs satisfying the standard ERC-721.

Figure 5: Daily sale volumes in dollar on four blockchains (data from [17]).

We have selected 59 Ethereum collections among the most traded collections on
OpenSea during the second semester of 2021. The list of collection names and
corresponding Ethereum smart contracts can be found in the appendix (section
VII-B).

We use the official Opensea API (https://docs.opensea.io) to download the
properties of all tokens in the selected collections, as well as all sale
transactions concerning these collections from the 1st June 2021 to the 15th
January 2022. Our dataset totals up 597198 transactions.

##  IV Materials and Applications

In order to train the model of equation (18) we use the Huber Regressor, a
linear regression model that is robust to outliers. We train two models: a
model with all collected collections in order to build the NFT price index,
and another model with only metaverse-related collections to build the
metaverse price index. Collections used for both indices can be found in the
appendix (section VII-B). The first value of our indices, i.e. the hyper-
parameter $A$, is set equal to 100. We report on table I the inferred
parameters $\boldsymbol{\beta}$ of $g$, the asset rarity impact function, for
the NFT price index.

Coefficient | Value
---|---
$\beta_{\min}$ | -0.176
$\beta_{\mathrm{avg}}$ | -0.009
$\beta_{\max}$ | -0.025
TABLE I: Asset-related coefficients of the model.

We plot in figures 7 and 7 the 7-day moving averages of the NFT and Metaverse
Index respectively. Raw indices can be found in the appendix (section VII-A).
We also plot in figures 7 and 7 the Google Trends (GT) signals for the queries
”nft” and ”metaverse” between the 1st June 2021 and the 16th January 2022 [18,
19]. Since the GT signals for $q$ is proportional to the Google search volume
for $q$, both GT signals will be used to chart the fad for NFTs and the
metaverse.

Figure 6: 7-day-MA NFT Index.

Figure 7: 7-day-MA Metaverse Index.

###  IV-A Correlation with cryptocurrency markets

In table II, we report the correlation coefficients between the daily returns
of our indices, ETH, BTC and SOL.

| NFT | Metaverse | ETH | SOL | BTC
---|---|---|---|---|---
NFT |  | 0.09 | 0.16 | 0.18 | 0.10
Metaverse |  |  | 0.30 | 0.17 | 0.20
ETH |  |  |  | 0.59 | 0.82
SOL |  |  |  |  | 0.48
TABLE II: Correlation matrix of daily returns. Correlation coefficients are
computed using the data from the 1st June 2021 to the 15th January 2022.

We report in table III the realized return of our indices, ETH, BTC, and SOL
between the 1st June 2021 and the 15st January 2022.

| Return
---|---
NFT | 7011%
Metaverse | 1064%
ETH | 26%
BTC | 6%
SOl | 372%
TABLE III: Realized return computed from prices in USDT between the 1st June
2021 and the 15st January 2022.

###  IV-B Bubble Detection

The minimum length $w$ to compute an ADF test is fixed equal to $40$. We use
the methodology of Phillips et al. presented in section II-D to detect the
start and the end of bubbles. $c$-critical values $v^{c}_{w}(t)$ of
$\sup_{t^{{}^{\prime}}\in I(w,t)}ADF_{0\rightarrow t^{{}^{\prime}}}$ are
estimated using Monte-Carlo simulations of $N_{MC}=5000$ random walks with an
asymptotically negligible drift (equation 17) where we set $\eta=\sigma=d=1$.
$v^{c}_{w}(t)$ are estimated for $t\in I(w,T)$. We report in particular the
$90\%$, $95\%$ and $99\%$ critical values when $t=T=230$ in table IV. We plot
on figures 9 and 9 the $t\rightarrow BSADF(t)$ signals (equation (19))
computed for both indices. We have also plotted the $95\%$ and $99\%$ critical
value signal for the statistics $t\rightarrow\sup_{t^{{}^{\prime}}\in
I(w,t)}ADF_{0\rightarrow t^{{}^{\prime}}}$. We recall that an explosive
behavior is detected in the price dynamic if the BSADF statistic exceeds the
$99\%$ critical value signal.

$c$ | $v^{c}_{w,t}$
---|---
$90\%$ | 1.08
$95\%$ | 1.36
$99\%$ | 1.97
TABLE IV: Critical values $v^{c}_{w}(t)$ of SADF with $t=T=230$ and $w=40$.

Figure 8: 7-day-MA NFT Index (above). BSADF statistic signal for the NFT Index
(below). $95\%$ and $99\%$ critical values (below).

Figure 9: 7-day-MA Metaverse Index (above). BSADF statistic signal for the
Metaverse Index (below). $95\%$ and $99\%$ critical values (below).

###  IV-C Undervalued and Overvalued Assets

Once the parameters $P$, $\boldsymbol{\alpha}$, $\boldsymbol{\beta}$,
$\boldsymbol{\gamma}$ and $\sigma$ of equation (10) are inferred after a
training phase, any asset $a$ can be priced at time $t\in\\{0,...,T\\}$ if at
least one asset of the same collection has been used during the training
phase. According to the model of equation (10), the log-price of $a$ at time
$t$ follows the distribution $\mathrm{Normal}(\log\bar{P}^{a}_{t},\sigma^{2})$
where $\log\bar{P}^{a}_{t}$ is computed as follows:

$\small\begin{split}\log\bar{P}^{a}_{t}&=\log
P+\alpha_{C^{a}}+\beta_{\min}f^{a}_{\min}+\beta_{\mathrm{avg}}f^{a}_{\mathrm{avg}}+\beta_{\max}f^{a}_{\max}+\gamma_{t}\end{split}$
(22)

We define the probabilities of being undersold and oversold in equations (23)
and (24), respectively.

$p_{\mathrm{under}}(P^{a}_{t})\triangleq\frac{1}{2}\left(1-\mathrm{erf}\left(\frac{\log(P^{a}_{t})-\log(\bar{P}^{a}_{t})}{\sigma\sqrt{2}}\right)\right)$
(23)

$p_{\mathrm{over}}(P^{a}_{t})\triangleq\frac{1}{2}\left(1+\mathrm{erf}\left(\frac{\log(P^{a}_{t})-\log(\bar{P}^{a}_{t})}{\sigma\sqrt{2}}\right)\right)$
(24)

where erf is the Gauss error function. According to the definitions (23) and
(24), it is straightforward to derive the following properties:

  * •

$p_{\mathrm{over}}+p_{\mathrm{under}}=1$,

  * •

at $P^{a}_{t}=\bar{P}^{a}_{t}$,
$p_{\mathrm{over}}(P^{a}_{t})=p_{\mathrm{under}}(P^{a}_{t})=\frac{1}{2}$,

  * •

$p_{\mathrm{under}}$ is strictly increasing,
$\lim_{0^{+}}p_{\mathrm{under}}=1$ and $\lim_{\infty}p_{\mathrm{under}}=0$.

The simplest investment strategy could consist in investing at time $t$ in an
asset $a$ for which the listing price $P^{a}_{t}$ gives a high undersold
probability $p_{\mathrm{under}}(P^{a}_{t})$ below $1/2$, and listing it at
$\bar{P}^{a}_{t}$. One problem, however, is that equation (22) assumes to have
estimated $\gamma_{t}$, i.e. the market impact coefficient of today. To deal
with it, $\gamma_{t}$ can be estimated using earlier in the day transactions,
or it can be approximated by a moving average of the market impact
coefficients over recent previous days.

##  V Discussion

From figures 7 and 7, we detect a strong upward trend for both price indices
indicating a global price rise. This increase in prices appears to follow the
rising interest for both themes. The NFT price index seems to have experienced
several brief explosive periods, while the metaverse price index experienced
an unique explosive period beginning in November 2021. This comes a short time
after the renaming of Facebook’s parent company to Meta reveiling their
interest for the metaverse.

As shown in table I, the parameters of $g$ are all non positive. As a
consequence, $g$, and then the price, is strictly decreasing with the
quantities $f_{\min}$, $f_{\mathrm{avg}}$ and $f_{\max}$. It statistically
confirms our intuition that scarcer assets are more expensive.

According to table II, returns of both indices are positively correlated with
the returns of the top-tier cryptocurrencies BTC, ETH and SOL, but correlation
coefficients remain relatively small compared to the ones between BTC, ETH and
SOL. From table III, we observe that the selected collections have, on the
whole, dramatically outperformed these cryptocurrencies. Putting these results
together suggests that blue ship NFTs have, taken as a whole, offered both-
crypto-and-NFT investors high returns while diversifying their risk.

According to figure 9, only a brief bubble has been detected by the presented
methodology. Thus, the NFT price index has globally never experienced any
explosive price dynamic, but it doesn’t mean that some collections have not.
Indeed, as shown in figure 9, a bubble has been detected in the metaverse
index between the 2. November 2021 and the 2. December 2021 following the
renaming of Facebook.

Concerning our methodology, two weaknesses have been identified. First, the
selection of collections is highly subjective and includes a look-ahead bias.
Secondly, hedonic coefficients $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$
are hold fixed over time, thus, our model can not take into account the time
varying popularity of collections to explain price dynamics. Both of these
problems could be solved by implementing a quantitative rule to update the
collection set on a regular basis, and, by using the adjacent dummy variable
method [9], an alternative training procedure for the time dummy variable
hedonic model. It could also be interesting in a future work to consider the
temporal dependence of the trends according to the collections, or to detect
NFT market regimes. Finally, it would also be appropriate to test and compare
other bubble detection models.

##  VI Acknowledgement

We would like to thank Opensea for providing us an API key to collect all the
necessary data to conduct this study.

##  VII Conclusion

In this work, we have proposed a methodology to construct price indices for
the NFT markets. In contrary to previous works, our indices aggregate
transactions from varied collections, this allows us to better represent this
new art market 3.0. These indices have allowed us to analyse the dynamics and
performances of NFT markets, and to perform diagnostic tests. In particular,
we have used statistical tests to detect speculative bubbles. Finally, we have
demonstrated that simple intuitive investment strategies could be derived from
the pricing model used for the contruction of our indices.

## References

  * [1] D. Analytics and @thomas_m. Nft market overview. [Online]. Available: https://dune.xyz/thomas_m/NFT-stats
  * Nakamoto [2008] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” _Decentralized Business Review_ , p. 21260, 2008.
  * Buterin et al. [2013] V. Buterin _et al._ , “Ethereum white paper,” _GitHub repository_ , vol. 1, pp. 22–23, 2013.
  * Kräussl et al. [2016] R. Kräussl, T. Lehnert, and N. Martelin, “Is there a bubble in the art market?” _Journal of Empirical Finance_ , vol. 35, pp. 99–109, 2016.
  * Opensea [a] Opensea. Cryptopunk #9998. [Online]. Available: https://opensea.io/assets/0xb47e3cd837ddf8e4c57f05d70ab865de6e193bbb/9998
  * Opensea [b] ——. Cryptopunk #3100. [Online]. Available: https://opensea.io/assets/0xb47e3cd837ddf8e4c57f05d70ab865de6e193bbb/3100
  * Herath and Maier [2010] S. Herath and G. Maier, “The hedonic price method in real estate and housing market research: a review of the literature,” 2010.
  * Selim [2011] S. Selim, “Determinants of house prices in turkey: A hedonic regression model,” _Doğuş Üniversitesi Dergisi_ , vol. 9, no. 1, pp. 65–76, 2011.
  * Triplett [2004] J. Triplett, “Handbook on hedonic indexes and quality adjustments in price indexes: Special application to information technology products,” 2004.
  * Kräussl and Elsland [2008] R. Kräussl and N. v. Elsland, “Constructing the true art market index: A novel 2-step hedonic approach and its application to the german art market,” CFS working paper, Tech. Rep., 2008.
  * Witkowska [2014] D. Witkowska, “An application of hedonic regression to evaluate prices of polish paintings,” _International Advances in Economic Research_ , vol. 20, no. 3, pp. 281–293, 2014.
  * Kireyev and Lin [2021] P. Kireyev and R. Lin, “Infinite but rare: Valuation and pricing in marketplaces for blockchain-based nonfungible tokens,” 2021.
  * Kong and Lin [2021] D.-R. Kong and T.-C. Lin, “Alternative investments in the fintech era: The risk and return of non-fungible token (nft),” _Available at SSRN 3914085_ , 2021.
  * Goldberg et al. [2021] M. Goldberg, P. Kugler, and F. Schär, “The economics of blockchain-based virtual worlds: A hedonic regression model for virtual land,” _Available at SSRN 3932189_ , 2021.
  * Phillips and Magdalinos [2007] P. C. Phillips and T. Magdalinos, “Limit theory for moderate deviations from a unit root,” _Journal of Econometrics_ , vol. 136, no. 1, pp. 115–130, 2007\.
  * Phillips et al. [2012] P. C. Phillips, S. Shi, and J. Yu, “Testing for multiple bubbles,” 2012.
  * [17] Cryptoslam. [Online]. Available: https://cryptoslam.io
  * goo [a] Google trends. [Online]. Available: https://trends.google.com/trends/explore?date=2021-01-23%202022-01-23&q=metaverse
  * goo [b] Google trends. [Online]. Available: https://trends.google.com/trends/explore?date=2021-01-23%202022-01-23&q=nft

## Appendix

###  VII-A Raw signals

Figure 10: NFT Index. Figure 11: Metaverse Index.

###  VII-B Smart contracts

Index | Collection | Ethereum Contract
---|---|---
NFT | CryptoPunks | 0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB
| CloneX | 0x49cF6f5d44E70224e2E23fDcdd2C053F30aDA28B
| MutantApeYachtClub | 0x60E4d786628Fea6478F785A6d7e704777c86a7c6
| BoredApeYachtClub | 0xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D
| Sandbox | 0x50f5474724e0Ee42D9a4e711ccFB275809Fd6d4a
| VOX Series 2 | 0xf76179bb0924BA7da8E7b7Fc2779495D7A7939d8
| Nanopass | 0xf54cC94f1F2f5De012B6Aa51F1E7eBdc43Ef5afC
| Shiba Social Club | 0xD692cEd124A474f051f9744a301C26D1017B3D54
| Psychedelics Anonymous Genesis | 0x75E95ba5997Eb235F40eCF8347cDb11F18ff640B
| Art Blocks | 0xa7d8d9ef8D8Ce8992Df33D8b8CF4Aebabd5bD270
| CoolmansUniverse | 0xa5C0Bd78D1667c13BFB403E2a3336871396713c5
| My Pet Hooligan | 0x09233d553058c2F42ba751C87816a8E9FaE7Ef10
| Decentraland | 0xF87E31492Faf9A91B02Ee0dEAAd50d51d56D5d4d
| Terraforms | 0x4E1f41613c9084FdB9E34E11fAE9412427480e56
| Doodles | 0x8a90CAb2b38dba80c64b7734e58Ee1dB38B8992e
| Neo Tokyo | 0x698FbAACA64944376e2CDC4CAD86eaa91362cF54
| Creature World | 0xc92cedDfb8dd984A89fb494c376f9A48b999aAFc
| More Than Gamers | 0x49907029e80dE1cBB3A46fD44247BF8BA8B5f12F
| VOX Series 1 | 0xad9Fd7cB4fC7A0fBCE08d64068f60CbDE22Ed34C
| DinoBabies | 0xf9d53E156fE880889E777392585FEb46D8D840f6
| Party Bears | 0x35471f47c3C0BC5FC75025b97A19ECDDe00F78f8
| DEGENERATE/REGENERATE | 0x7828c811636CCf051993C1EC3157b0B732e55B23
| ALIENFRENS | 0x123b30E25973FeCd8354dd5f41Cc45A3065eF88C
| NFT Worlds | 0xBD4455dA5929D5639EE098ABFaa3241e9ae111Af
| CyberKongz | 0x57a204AA1042f6E66DD7730813f4024114d74f37
| Slotie | 0x5fdB2B0C56Afa73B8ca2228e6aB92Be90325961d
| RumbleKongLeague | 0xEf0182dc0574cd5874494a120750FD222FdB909a
| Cool Cats | 0x1A92f7381B9F03921564a437210bB9396471050C
| Punks Comic 2 | 0x128675d4FddbC4a0D3f8aA777D8EE0fb8B427C2F
| Wolf Game | 0x7F36182DeE28c45dE6072a34D29855BaE76DBe2f
| Little Lemon Friends | 0x0B22fE0a2995C5389AC093400e52471DCa8BB48a
| VeeFriends | 0xa3AEe8BcE55BEeA1951EF834b99f3Ac60d1ABeeB
| Crypto Bull Society | 0x469823c7B84264D1BAfBcD6010e9cdf1cac305a3
| House of Legends | 0x8C714199d2eA08CC1f1F39A60f5cD02aD260A1e3
| Meta-Legends | 0xF9c362CDD6EeBa080dd87845E88512AA0A18c615
| Gambling Apes | 0x90cA8a3eb2574F937F514749ce619fDCCa187d45
| BoredApeKennelClub | 0xba30E5F9Bb24caa003E9f2f0497Ad287FDF95623
| Cryptoadz | 0x1CB1A5e65610AEFF2551A50f76a87a7d3fB649C6
| apekidsclub | 0x9Bf252f97891b907F002F2887EfF9246e3054080
| ExpansionPunks | 0x0D0167A823C6619D430B1a96aD85B888bcF97C37
| Elderly Ape Retirement Club | 0x9ee36cD3E78bAdcAF0cBED71c824bD8C5Cb65a8C
| Divine Wolves | 0xb4e9123bd3Ef4Df17f8cc6EF7C2Be66428CF4931
| Superlative Apes | 0x1e87eE9249Cc647Af9EDEecB73D6b76AF14d8C27
| Desperate ApeWives | 0xF1268733C6FB05EF6bE9cF23d24436Dcd6E0B35E
| Solarbots | 0x8009250878eD378050eF5D2a48c70E24EB2edE7E
| merge. | 0xc3f8a0F5841aBFf777d3eefA5047e8D413a1C9AB
| Neo Tokyo Part 3 | 0x0938E3F7AC6D7f674FeD551c93f363109bda3AF9
| Art blocks | 0x059EDD72Cd353dF5106D2B9cC5ab83a52287aC3a
| Champions | 0x97a923ed35351a1382E6bcbB5239fc8d93360085
| Pepsi Mic Drop | 0xa67D63E68715DCF9b65e45e5118b5fcD1e554b5f
| CROAKZ | 0x7caE7B9b9a235D1D94102598E1f23310A0618914
| Meebits | 0x7Bd29408f11D2bFC23c34f18275bBf23bB716Bc7
| Cyber Gorillas | 0x1E1b4E127A510cafa6d0eAec024a4319a5E18821
| Lost Poets | 0x4b3406a41399c7FD2BA65cbC93697Ad9E7eA61e5
| SuperRare | 0xb932a70A57673d89f4acfFBE830E8ed7f75Fb9e0
| Lazy Lions | 0x8943C7bAC1914C9A7ABa750Bf2B6B09Fd21037E0
| Neo Tokyo Part 2 | 0xab0b0dD7e4EaB0F9e31a539074a03f1C1Be80879
| Loot | 0xFF9C1b15B16263C61d017ee9F65C50e4AE0113D7
| Cryptovoxels | 0x79986aF15539de2db9A5086382daEdA917A9CF0C
Metaverse | Decentraland | 0xF87E31492Faf9A91B02Ee0dEAAd50d51d56D5d4d
| Sandbox | 0x50f5474724e0Ee42D9a4e711ccFB275809Fd6d4a
| Cryptovoxels | 0x79986aF15539de2db9A5086382daEdA917A9CF0C
| NFT Worlds | 0xBD4455dA5929D5639EE098ABFaa3241e9ae111Af
TABLE V: Composition of the indices.


# Integrated Node Encoder for Labelled Textual Networks

Ye Ma1, Lu Zong2
{ye.ma<EMAIL_ADDRESS>
###### Abstract

Voluminous works have been implemented to exploit content-enhanced network
embedding models, with little focus on the labelled information of nodes.
Although TriDNR (?) leverages node labels by treating them as node attributes,
it fails to enrich unlabelled node vectors with the labelled information,
which leads to the weaker classification result on the test set in comparison
to existing unsupervised textual network embedding models. In this study, we
design an integrated node encoder (INE) for textual networks which is jointly
trained on the structure-based and label-based objectives. As a result, the
node encoder preserves the integrated knowledge of not only the network text
and structure, but also the labelled information. Furthermore, INE allows the
creation of label-enhanced vectors for unlabelled nodes by entering their node
contents. Our node embedding achieves state-of-the-art performances in the
classification task on two public citation networks, namely Cora and DBLP,
pushing benchmarks up by 10.0% and 12.1%, respectively, with the 70% training
ratio. Additionally, a feasible solution that generalizes our model from
textual networks to a broader range of networks is proposed.

## Introduction

Content-enhanced network embedding, aims at learning continuous vectors for
nodes with rich node contents such as texts. These node representations can be
directly used in downstream tasks including classification, link prediction,
recommendation, etc. To complete or improve these tasks, artificial labels are
often added to the node. However, most of the time only a small number of
labels could be accessed due to the costly human resources. Therefore, the
necessity of making full use of precious human knowledge to automatically
annotate unlabelled nodes and enrich their representations emerges.

Most textual network embedding models (?; ?; ?; ?; ?; ?) focus on learning
unsupervised global node embeddings before classifying nodes in a supervised
manner, which actually, is deemed as a waste of the valuable labelled
information. TriDNR (?) learns vectors of node labels which are used to
enhance representations of nodes with known labels. However, this approach
fails to improve the representations of unlabelled nodes, which explains its
weaker performance in the classification task in comparison to other
unsupervised node embedding approaches, such as CENE (?). In addition, there
are two common disadvantages of network embeddings according to ? (?). First,
graph embedding often initializes a vector to each node without shared
parameters among different nodes in the embedding matrix, which requires a
great level of computing resources in the training as the number of nodes
grows. Second, it lacks a flexible way to generate representations of new
nodes, and often requires the re-construction of a new network to re-train the
node vectors.

To address these problems, this study first uses a shared node/text encoder,
whose parameters for different nodes are shared via same words of the input
text, to embed all nodes into continuous vectors. Subsequently, we train these
node embeddings based on the structural information of the network and node
labels. Finally, the trained node encoder, which not only preserves the
structural and labelled information, but also extracts the semantic and
syntactic features of texts. The node encoder could be used later to generate
representations of nodes, including new nodes, by entering their node
contents.

Our embedding achieves state-of-the-art classification results in different
training ratios on two the public textual networks. With the $10\%$
training/labelled ratio, we push the benchmarks up by $2.7\%$ and $4.8\%$
respectively, indicating that even a low percentage of labels improves node
representations. Additionally, a feasible solution that generalizes our model
from textual networks to a broader range of networks is proposed.

## Related Work

Structure-based network embedding: DeepWalk (?) employs random walks to sample
node sequences whilst skip-gram (?) is adopted to learn node representations
by regarding nodes as words. LINE (?) learns node embedding by taking both the
first- and second-order proximity into account. Node2vec (?) improves uniform
random walks of DeepWalk by adopting biased random walks.

Textual network embedding: TADW (?) incorporates text information in the
network embedding based on the DeepWalk-derived matrix factorization. TriDNR
(?) trains node representations on three parties, namely node-node, node-word
and label-word, so that the trained node vectors simultaneously represent the
network-structural, textual and labelled information. CENE (?) constructs a
heterogeneous network by treating documents as nodes and learns node
embeddings by second-order proximity. According to CANE (?), each node vector
is a concatenation of the context-aware text embedding and the structural-
based embedding. Context-aware means the text is assigned with different
vectors while being connected to different nodes. To improve CANE, WANE (?),
DMTE (?) and GANE (?) are developed to adopt the fine-grained alignment,
truncated diffusion maps and global attention via optimal transport,
respectively.

## Problem Definition

We introduce basic notations and definitions used in this paper.

Labelled textual network: Let $G=(V,E,T,L)$ denote a network, where $V$ is the
set of nodes, $E\in(V\times V)$ is the set of edges between nodes, and $T$ and
$L$ represent the textual information and the labelled information of nodes,
respectively. Each node $v$ denotes a document that is composed of a word
sequence $(w_{1},w_{2},\cdots,w_{n})$ with a positional encoding sequence
$(p_{1},p_{2},\cdots,p_{n})$, where $w_{i},p_{i}\in\mathbb{R}^{d}$ and $d$ is
the dimension of the node embedding. In addition, $N(v)$ is the network
neighborhoods of $v$, and $L(v)$ is the labels of $v$, such as the categorical
label. Note that not all nodes have labels.

Integrated node encoder: The input of the encoder is the content $T(v)$ of the
node $v$ whilst the output is a fixed node vector $\Phi(v)\in\mathbb{R}^{d}$.
The node encoding model simultaneously preserves network structural
information and labelled information so that node embedding represents the
integrated knowledge.

## Algorithm

### Overall Framework

An overview of our node encoding model is presented in Figure 1. Random walks
are employed to sample neighbour nodes. The objective of our model is to train
an integrated node encoder $\Phi$.

Figure 1: Overview of our node encoding model

With such encoder, each node whose content is a document is represented as a
continuous vector that preserves both the structural and labelled information
of the network. To achieve this goal, the objective is divided into two parts,
namely the structure-based objective:

$O_{s}=\sum_{v\in V}logPr(N(v)|\Phi(v)),$ (1)

and the label-based objective:

$O_{l}=\sum_{v\in V}\alpha_{v}\ logPr(L(v)|\Phi(v)),$ (2)
$\alpha_{v}=\left\\{\begin{array}[]{ll}1,&v\ is\ labelled;\\\ 0,&v\ is\
unlabelled.\end{array}\right.$ (3)

Considering the practical situation that not all nodes have labels, we apply a
masking mechanism $\alpha_{v}$ to our objective function. In practice, we use
a special token to denote the label of unlabelled node. No matter what the
token is, there is no gradient descent as its gradient is zero after
converting the loss to zero. At the same time, this masking mechanism is the
key to a fair comparison of our node embedding with other textual network
embedding models.

The final objective is assigned to maximize the sum of the above two
objectives:

$O=O_{s}+\beta\ O_{l}$ (4)

where $\beta$ is a constant coefficient.

In the remaining of this section, a more explicit explanation is given in
terms of the architecture of the node encoder, the structure-based & label-
based objectives, as well as their joint training.

### Node Encoder

The node encoder converts the document to a fixed vector. As shown in Figure
2, the encoding process is divided into two steps. The first step is to
generate the contextual word embedding $e^{\prime}_{i}\in\mathbb{R}^{d}$,
whilst the second is to assign weights to each contextual word embedding by an
attention layer thus to obtain the node representation, which is the weighted
average of the contextual word embeddings.

Figure 2: Node encoder

Contextual word embedding: Aims at creating different representations for the
same word according to its context. Inspired by the encoding model of
Transformer (?), we first add a positional encoding from sine and cosine
functions to each word embedding in the right order. To extract the context
information of each word, we then compute the intra-attention using the Multi-
Head Attention (mha) model which consists of Scaled Dot-Product Attentions.
Multi-headers are designed to extract a variety of semantic and syntactic
features of a document. For more details on the positional encoding, mha and
Scaled Dot-Product Attentions please refer to ? (?).

Node/text embedding: The easiest way of computing the text embedding with
contextual word embeddings is to use either the add or the average pooling.
However, it is practically difficult to mask the padding values in the matrix
calculation using these two approaches. Moreover, it is believed that weighted
words make more sense than equally important words. Therefore, we adopt
another attention mechanism to create node embeddings. The basic architecture
of the attention layer is as follows:

$h_{v,i}=tanh(W_{e}e_{v,i}^{\prime}+b_{e})$ (5)
$k_{v,i}=\frac{exp(\frac{h_{v,i}^{T}h_{w}}{\sqrt{d}})}{\sum_{j=1}^{n}{exp(\frac{h_{v,i}^{T}h_{w}}{\sqrt{d}})}}$
(6) $\Phi(v)=\sum_{i=1}^{n}{k_{v,i}e_{v,i}^{\prime}}$ (7)

where $W_{e}\in\mathbb{R}^{d\times d},b_{e}$ and $h_{w}\in\mathbb{R}^{d}$ are
the trainable parameters. Similar to the Scaled Dot-Product Attention, the
same scaling mechanism (i.e. divide each by $\sqrt{d}$) is adopted for more
stable gradient. Meanwhile, a mask mechanism is employed on $h_{v,i}^{T}h_{w}$
by setting each value in the padding position to $-\infty$. Additionally, a
dropout (?) is included for $e_{i}$ and $e^{\prime}_{i}$.

### Structure-based Objective

Our structure-based objective is developed in the framework of network
embedding based on random walks (?). With the assumption of conditional
independence, the probability in $O_{s}$ is factorized as:

$logPr(N(v)|\Phi(v))=\sum_{u\in N(v)}logPr(u|\Phi(v)).$ (8)

Network neighborhoods $N(v)$ are first sampled by random walks and later
sampled again by the context windows after being concatenated into node
sequences. To reduce the computational cost, negative sampling (?) is adopted
to approximate the conditional probability.

$\begin{split}logPr(u|\Phi(v))\approx logPr(1|\Phi(v),u)+\\\
\sum_{i=1}^{m}\mathbb{E}_{u^{{}^{\prime}}_{i}\sim
P(v)}[logPr(0|\Phi(v),u^{{}^{\prime}}_{i})]\end{split}$ (9)

where $u^{\prime}$ ($u^{\prime}\neq u$) are sampled $m$ times from the noise
distribution $P(v)$. Probabilities of binary classifications are computed by
the logistic function.

### Label-based Objective

Suppose that each node is assigned with several labels of different types in a
given network. In this study, we assume that the node labels are independent
of each other, which leads to the conditional probability in $O_{l}$ written
as:

$logPr(L(v)|\Phi(v))=\sum_{l\in L(v)}logPr(l|\Phi(v))$ (10)

Since the data sets in the experiment are given with only one type of label,
i.e. the node category ($L(v)=c_{v}$), the conditional probability is
simplified as:

$Pr(c_{v}|\Phi(v))=\frac{exp(\sigma_{c_{v}}^{T}\Phi(v))}{\sum_{c\in
C}exp(\sigma_{c}^{T}\Phi(v))}$ (11)

where $\sigma_{c}$ is trainable and $C$ denotes the set of node classes.

### Joint Training

As far as the joint training is concerned, the key problem is that the
structure-based objective has more targets than the label-based objective, as
each element in the node sequence has several context nodes but only one label
and one masking factor. To address this matter, we copy the labels and masking
factors so that they have the same number of targets as the structure-based
objective does. Algorithm 1 shows the framework of the joint training in our
node encoder.

Input: $S_{n}$, the node sequence $[v_{1},v_{2},\cdots,v_{n}]$ generated by
random walks

Output: $\Phi$, the trained integrated node encoder

1 Initialize $\Phi$ ;

2 Map node contents into $S_{n}$, $S_{t}=[T(v_{1}),T(v_{2}),\cdots,T(v_{n})]$;

3 Map node categories into $S_{n}$,
$S_{c}=[c_{v_{1}},c_{v_{2}},\cdots,c_{v_{n}}]$;

4 Map masking factors into $Sn$,
$S_{\alpha}=[\alpha_{v_{1}},\alpha_{v_{2}},\cdots,\alpha_{v_{n}}]$;

5 for  _$t,c,a$ in Zip($S_{t},S_{c},S_{\alpha}$)_ do

6 for  _$u$ in $Context(t)$_ do

7 $NegativeSampling\ u^{\prime}$ from $S_{t}/\\{u\\}$;

8 $AdamOptimizer$ to minimize $-[logPr(u|\Phi(v))+\beta\ a\ logPr(c|\Phi(v))]$
;

9

10

Return $\Phi$

Algorithm 1 Joint Training

From Algorithm 1, we can see the loss function is based on Eq. (4) where
$\Phi(v)$ is the output of the node encoder $\Phi$ with the input $t$, and
Adam (?) is used to optimize the objective function. Note that in order to
give a clearer demonstration of our training framework, we set the batch size
to be $1$ in Algorithm 1, whereas the batch size is $64$ by default in the
actual operation aiming to speed up the training.

## Node Classification

### Experimental setup

All datasets and codes are open source on Github (the link is omitted due to
the blind review policy).

Datasets: ($i$) Cora111 https://people.cs.umass.edu/ ̵̃mccallum/data.ht, a
paper citation network with each node denoting the bibliography data of a
paper (?). We use the dataset pruned by ? (?), which keeps only the abstracts.
($ii$) DBLP222http://arnetminer.org/citation (V4 version is used), a paper
citation network similar to Cora. A simplified version built by ? (?) is hired
in this study. Table 1 summarizes the two datasets used in our experiment.
Note that out of the $60744$ nodes in total, DBLP has only $17725$ effective
nodes that have edges.

| Cora | DBLP
---|---|---
#Effective Nodes | 2277 | 17725
#Edges | 5214 | 52914
#Node Contents | Abstract | Title
#Maximum Lengths | 410 | 28
#Classes | 7 | 4
Table 1: Summary of datasets

Parameter setup: For both Cora and DBLP, we set the coefficient of $O_{l}$:
$\beta=10$ (After investigating the coefficient, we find that $\beta\geq 3$
outperform smaller $\beta^{\prime}s$ significantly). The dimension of the node
embedding is $100$, the dropout rate is $0.5$, the number of attention headers
is $2$, the context size is $5$, the walk length is $80$, the batch size is
$64$, the epoch is $1$, and negative samples are $5$. Moreover, the number of
walks is set to be $3$ for Cora, and $6$ for DBLP.

The classifier and evaluation criteria:  To train the node encoder, the node
set is first shuffled and split into the training and test sets based on the
training/labelled ratio. Only labels of the training set are used in the joint
training whereas the rest of labels are masked by forcing their losses to
zero. The trained node encoder is then used to compute embeddings for the
nodes. The support vector machine (SVM) from scikit-learn (?) is adopted to
make classifications whilst the Macro-f1 score is used to evaluate the
classification results. Prior to the SVM, we re-sample the training set to
balance the data. The experiment of each train ratio is repeated for at least
$5$ times, and the final results are computed as the average Macro-f1 scores.

Baselines: We introduce one unsupervised text embedding model Doc2vec (?) and
nine network embedding models including DeepWalk, LINE, TADW, TriDNR, CENE,
CANE, WANE, DMTE, GANE (please refer to Related Work for their briefs) as
baselines.

The majority of the baselines included use the SVM for classification based on
their unsupervised global node embeddings. Due to the supervised nature of
SVM, labels are inevitably needed to train the classifier but neglected in the
node embedding process, which leads to the loss of the important labelled
information. As far as fairness is concerned, our node encoder only makes full
use of those labels, which baselines use to train the classifier, at the same
time keeps the same proportion of labels as the baselines.

### Results

The classification results are shown in Table 2 and Table 3. Results of the
baselines are partially obtained from ? $\dagger$ (?), ? $\ddagger$ (?) and ?
$\S$ (?). We further adjust the hyper-parameters of Doc2vec and DeepWalk for
better performance in the classification. In addition, to roundly compare the
INE embeddings to the baselines, three versions are included, namely
INE-$O_{s}$ (structural-based: unsupervised version), INE-$O_{l}$ (label-
based), INE-joint (joint training).

Training ratio | 10% | 30% | 50% | 70%
---|---|---|---|---
LINE$\dagger$ | 53.9% | 56.7% | 58.8% | 60.1%
Doc2vec | 71.0% | 74.5% | 76.5% | 76.0%
DeepWalk | 74.9% | 76.6% | 78.7% | 79.4%
TADW$\dagger$ | 71.0% | 71.4% | 75.9% | 77.2%
CANE$\dagger$ | 81.6% | 82.8% | 85.2% | 86.3%
DMTE$\dagger$ | 81.8% | 83.9% | 86.3% | 87.9%
WANE$\dagger$ | 81.9% | 83.9% | 86.4% | 88.1%
GANE$\dagger$ | 82.3% | 84.2% | 86.7% | 88.5%
INE-$O_{s}$ | 76.2% | 82.3% | 83.3% | 84.5%
INE-$O_{l}$ | 49.7% | 80.8% | 94.1% | 97.0%
INE-joint | 85.0% | 95.2% | 97.6% | 98.5%
Table 2: Macro f1 scores on Cora Training ratio | 10% | 30% | 50% | 70%
---|---|---|---|---
LINE$\ddagger$ | 42.7% | 43.8% | 43.8% | 43.9%
Doc2vec | 65.3% | 66.7% | 67.4% | 67.4%
DeepWalk | 73.3% | 73.4% | 75.0% | 74.8%
TADW$\ddagger$ | 67.6% | 68.9% | 69.2% | 69.5%
TriDNR$\ddagger$ | 68.7% | 72.7% | 73.8% | 74.4%
CENE$\S$ | 72.8% | 73.7% | 75.0% | 76.3%
INE-$O_{s}$ | 74.3% | 76.0% | 76.3% | 76.9%
INE-$O_{l}$ | 21.1% | 21.9% | 42.4% | 41.7%
INE-joint | 78.1% | 85.0% | 87.0% | 88.4%
Table 3: Macro f1 scores on DBLP

Node vectors generated by the jointly-trained node encoder achieve dominant
results in the classification task. Observations are presented as follows:

Figure 3: Cora visualization Figure 4: DBLP visualization

  * •

INE-joint significantly outperforms state-of-the-art methods on both Cora and
DBLP, indicating labelled information greatly improves the quality of node
representations. It should be mentioned that the classification result of CENE
on Cora is between CANE and WANE (refer to Figure 3 in ? (?) and Figure 3(c)
in ? (?)).

  * •

INE-$O_{s}$ outperforms DeepWalk, showing textual information improves pure
structural-based embeddings. At the same time, we find that the improvement on
Cora is higher than that on DBLP, due to the lower quality of node contents of
the latter. This finding is further supported by the poorer performance of
Doc2vec in DBLP than that in Cora, in comparison to DeepWalk.

  * •

INE-$O_{l}$ performs better as the labelled ratio increases on Cora,
indicating that the model can produce good classification results by simply
relying on the text as long as the training set is adequately large. On the
other hand, the poor performance of INE-$O_{l}$ on DBLP indicates the
difficulty in distinguishing the node type based on the node contents of DBLP.
In this case, the citation information (i.e. structural information of
citation network) plays a major role in the node classification.

### Investigate labelled ratios

We investigate the classification results of different labelled ratios on the
same training and test set. Labelled nodes are only considered in the training
set. As an example, the result of $30\%$ labelled data on a training ratio
$10\%$ cannot be computed since the number of labelled data exceeds the number
of training data.

Training ratio | 10% | 30% | 50% | 70%
---|---|---|---|---
10% Labelled | 85.0% | 90.7% | 92.8% | 93.3%
30% Labelled | - | 95.2% | 97.1% | 97.4%
50% Labelled | - | - |  97.6% | 98.2%
70% Labelled | - | - | - |  98.5%
Table 4: Macro f1 scores of different labelled ratios on Cora training ratio | 10% | 30% | 50% | 70%
---|---|---|---|---
10% labelled | 78.1% | 78.9% | 78.9% | 79.2%
30% labelled | - |  85.0% | 84.6% | 84.8%
50% labelled | - | - |  87.0% | 87.3%
70% labelled | - | - | - | 88.4%
Table 5: Macro f1 scores of different labelled ratios on DBLP

As Table 4 (Cora) and Table 5 (DBLP) suggest, it is found that given the same
training ratio, data with more labels has higher scores. This means that the
integrated node encoder preserves more node information and produce higher
quality node vectors as the labelled information increases. Moreover, the rate
of Macro f1 improvement slows down as the labelled ratio increases.

## Network Visualization

In order to gain a more intuitive understanding on the advantages of our node
representations, we reduce the dimension of the Cora and DBLP embeddings by
t-SNE (?) (i.e. t-distributed stochastic neighbor embedding). For both
datasets, we choose the embeddings produced by DeepWalk, INE-$O_{s}$
(unsupervised), 10% labelled INE-joint and 70% labelled INE-joint for the
purpose of comparison. Dimension reduction results are shown in Figure 3 and
Figure 4 with the same color indicating the same type of node.

Two cases, namely micro clustering and macro clustering, are defined and
discussed for better interpretation of the dimension reduction plots.
Specifically, micro clustering means that nodes of the same type (or color)
gather in multiple small clusters that may scatter in the graph. A typical
example of micro clustering is given by the DeepWalk embeddings. Intuitively,
micro clustering indicates that node embeddings captures the network structure
information as DeepWalk is solely based on network structure. On the other
hand, macro clustering refers to the situation that nodes or small clusters of
the same color gather in a certain location close to each other. As the
textual and labelled information is included, the phenomenon of macro
clustering becomes clearer. In Plots (d) of Figure 3, 4, nodes of the same
color cluster all together, indicating that our embedding captures the
labelled information in a great manner. Nevertheless, Plots (c) and (d) show
that macro clustering maintains the characteristics of micro clustering. In
particular, the macro clusters consist of micro clusters of the same color,
which suggests that our embedding integrates both the structural and labelled
information.

## Generalized INE

For textual networks or other networks with node contents, the same framework
of the INE could be used by only re-designing the node encoder according to
the inputs (i.e. node contents). However, for a broader range of networks
without node contents, the input could only be a vector that represents solely
the node itself, leading to the two drawbacks according to ? (?) (refer to
Introduction). To address networks without node contents, we propose to
extract node attributes or side information as node contents. The advantages
of this technique is twofold. First, the two drawbacks of classic network
embedding models (?) are solved. Despite that there could be an infinite
number of distinct nodes, a limited number of node attributes that are shared
by different nodes could be extracted. Second, this method enriches node
vectors with attributes or side information by treating them as encoder
inputs. Suppose that a social APP collects its users’ data to constructs a
social network of users based on their interactions, node contents could be
then obtained by analysing and recording their behaviors and characteristics.
In this way, node attributes are similar to unordered words that could be
inputted into the node encoder. Nonetheless, our model provides a flexible way
to incorporate human knowledge (i.e. labels that need to be manually annotated
such as customer levels) into the node representations.

## Conclusions

This paper proposes a novel idea to embed networks with node contents. For the
textual network, we first design an advanced text encoder to effectively
extract semantic and syntactic features. In order to preserve both the
structural and labelled information, the node encoder is jointly trained based
on structure- and label-based objectives. By making the most of node labels,
our experiments show that even a small proportion of node labels improve node
representations significantly in the classification task. Further, A shared
node encoder, whose inputs are node contents, not only highly saves the
computational source in the learning of representations, but also enables the
embedding model to generate the embeddings of new nodes. Finally, we discuss
the solution to extent our model from textual networks to a wider range of
networks. With the generalized framework, network embedding models are given
with the capacity to incorporate manually annotated information and node
attributes into node representations, at the same time are able to infer
representations for new nodes.

## References

  * [Chen et al. 2019] Chen, L.; Wang, G.; Tao, C.; Shen, D.; Cheng, P.; Zhang, X.; Wang, W.; Zhang, Y.; and Carin, L.  2019\.  Improving textual network embedding with global attention via optimal transport.  arXiv preprint arXiv:1906.01840.
  * [Grover and Leskovec 2016] Grover, A., and Leskovec, J.  2016\.  node2vec.  Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16.
  * [Kingma and Ba 2014] Kingma, D. P., and Ba, J.  2014\.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * [Le and Mikolov 2014] Le, Q., and Mikolov, T.  2014\.  Distributed representations of sentences and documents.  In International conference on machine learning, 1188–1196.
  * [Maaten and Hinton 2008] Maaten, L. v. d., and Hinton, G.  2008\.  Visualizing data using t-sne.  Journal of machine learning research 9(Nov):2579–2605.
  * [McCallum et al. 2000] McCallum, A. K.; Nigam, K.; Rennie, J.; and Seymore, K.  2000\.  Automating the construction of internet portals with machine learning.  Information Retrieval 3(2):127–163.
  * [Mikolov et al. 2013a] Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J.  2013a.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * [Mikolov et al. 2013b] Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.; and Dean, J.  2013b.  Distributed representations of words and phrases and their compositionality.  CoRR abs/1310.4546.
  * [Pan et al. 2016] Pan, S.; Wu, J.; Zhu, X.; Zhang, C.; and Wang, Y.  2016\.  Tri-party deep network representation.  In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, 1895–1901.  AAAI Press.
  * [Pedregosa et al. 2011] Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E.  2011\.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 12:2825–2830.
  * [Perozzi, Al-Rfou, and Skiena 2014] Perozzi, B.; Al-Rfou, R.; and Skiena, S.  2014\.  Deepwalk: Online learning of social representations.  In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, 701–710.  New York, NY, USA: ACM.
  * [Shen et al. 2018] Shen, D.; Zhang, X.; Henao, R.; and Carin, L.  2018\.  Improved semantic-aware network embedding with fine-grained word alignment.  Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
  * [Srivastava et al. 2014] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R.  2014\.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research 15:1929–1958.
  * [Sun et al. 2016] Sun, X.; Guo, J.; Ding, X.; and Liu, T.  2016\.  A general framework for content-enhanced network representation learning.  ArXiv abs/1610.02906.
  * [Tang et al. 2015] Tang, J.; Qu, M.; Wang, M.; Zhang, M.; Yan, J.; and Mei, Q.  2015\.  Line.  Proceedings of the 24th International Conference on World Wide Web - WWW ’15.
  * [Tu et al. 2017] Tu, C.; Liu, H.; Liu, Z.; and Sun, M.  2017\.  CANE: Context-aware network embedding for relation modeling.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1722–1731.  Vancouver, Canada: Association for Computational Linguistics.
  * [Vaswani et al. 2017] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I.  2017\.  Attention is all you need.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems 30. Curran Associates, Inc.  5998–6008.
  * [Yang et al. 2015] Yang, C.; Liu, Z.; Zhao, D.; Sun, M.; and Chang, E. Y.  2015\.  Network representation learning with rich text information.  In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15, 2111–2117.  AAAI Press.
  * [Zhang et al. 2018] Zhang, X.; Li, Y.; Shen, D.; and Carin, L.  2018\.  Diffusion maps for textual network embedding.  In Advances in Neural Information Processing Systems, 7587–7597.
  * [Zhou et al. 2018] Zhou, J.; Cui, G.; Zhang, Z.; Yang, C.; Liu, Z.; and Sun, M.  2018\.  Graph neural networks: A review of methods and applications.  arXiv preprint arXiv:1812.08434.