# A Generative Model for Realistic Galaxy Cluster X-ray Morphologies

Maya Benyas Department of Physics, Stanford University, 382 Via Pueblo Mall,
Stanford, CA 94305, USA  Jordan Pfeifer Department of Physics, Brown
University, Providence, RI 02912-1843, USA  Adam B. Mantz Kavli Institute for
Particle Astrophysics and Cosmology, Stanford University, 452 Lomita Mall,
Stanford, CA 94305, USA  Steven W. Allen Department of Physics, Stanford
University, 382 Via Pueblo Mall, Stanford, CA 94305, USA  Kavli Institute for
Particle Astrophysics and Cosmology, Stanford University, 452 Lomita Mall,
Stanford, CA 94305, USA  SLAC National Accelerator Laboratory, 2575 Sand Hill
Road, Menlo Park, CA 94025, USA  Elise Darragh-Ford Department of Physics,
Stanford University, 382 Via Pueblo Mall, Stanford, CA 94305, USA  Kavli
Institute for Particle Astrophysics and Cosmology, Stanford University, 452
Lomita Mall, Stanford, CA 94305, USA
mbenyas@stanford.edu,jordan_pfeifer<EMAIL_ADDRESS>
###### Abstract

The X-ray morphologies of clusters of galaxies display significant variations,
reflecting their dynamical histories and the nonlinear dependence of X-ray
emissivity on the density of the intracluster gas. Qualitative and
quantitative assessments of X-ray morphology have long been considered a proxy
for determining whether clusters are dynamically active or “relaxed.”
Conversely, the use of circularly or elliptically symmetric models for cluster
emission can be complicated by the variety of complex features realized in
nature, spanning scales from Mpc down to the resolution limit of current X-ray
observatories. In this work, we use mock X-ray images from simulated clusters
from The Three Hundred project to define a basis set of cluster image
features. We take advantage of clusters’ approximate self similarity to
minimize the differences between images before encoding the remaining
diversity through a distribution of high order polynomial coefficients.
Principal component analysis then provides an orthogonal basis for this
distribution, corresponding to natural perturbations from an average model.
This representation allows novel, realistically complex X-ray cluster images
to be easily generated, and we provide code to do so. The approach provides a
simple way to generate training data for cluster image analysis algorithms,
and could be straightforwardly adapted to generate clusters displaying
specific types of features, or selected by physical characteristics available
in the original simulations.

††software: scikit-learn (Pedregosa et al., 2011), zernike (Antonello &
Verhaegen, 2015)

##  1 Introduction

Clusters of galaxies are filled with hot plasma, the intracluster medium,
which comprises most of the baryonic matter within them. The X-ray emission
from this gas, primarily from bremsstrahlung and line emission, carries a
wealth of information about the dynamical histories of clusters and their
internal astrophysical processes. Since the advent of high-spatial-resolution
X-ray imaging spectroscopy with the Chandra and XMM-Newton observatories, it
has been possible to study in detail features such as shocks and cold fronts
(Markevitch & Vikhlinin, 2007), gas sloshing (e.g. Ascasibar & Markevitch
2006; Roediger et al. 2011; ZuHone et al. 2011; Johnson et al. 2012;
Simionescu et al. 2012; Paterno-Mahler et al. 2013), cavities associated with
feedback from active galactic nuclei (McNamara et al., 2000; Fabian, 2012),
and the cool, dense cores found in some clusters (Peterson & Fabian, 2006).
More recently, such data have been used in investigations of turbulence
(Churazov et al., 2012; Zhuravleva et al., 2015, 2016, 2018, 2019; Arévalo et
al., 2016) and gas clumping in cluster outskirts (Eckert et al., 2015;
Mirakhor & Walker, 2021; de Vries et al., 2023), the latter having been
spearheaded by lower-resolution but also lower-background Suzaku observations
of relatively nearby systems (Simionescu et al., 2011; Urban et al., 2011,
2014; Walker et al., 2012a, b).

The high sensitivity of X-ray emission to the dynamical processes playing out
within clusters has led to features apparent in their X-ray surface brightness
– their X-ray morphology – being used as a proxy for the underlying dynamics.
Quantitative estimates of overall symmetry (Mohr et al., 1993; Buote & Tsai,
1995; Jeltema et al., 2005; Nurgaliev et al., 2013; Rasia et al., 2013) as
well as the presence of a compact, bright core (Vikhlinin et al., 2007; Santos
et al., 2008; Böhringer et al., 2010), or both, have been employed to identify
the most dynamically “relaxed” clusters (e.g. Mantz et al. 2015). Other sub-
populations presenting distinct morphologies that may be of special interest
include major plane-of-the-sky mergers (e.g. Massey et al. 2011), feedback-
induced cavities, and large-scale sloshing, to name a few.

While a small number of simple estimators can be useful and effective for
specific purposes, in general the rich detail and complexity of modern X-ray
images of clusters far exceed what can be encoded in a few numbers.
Conversely, the presence of complex, unmodeled structure can in principle
result in pathological cases where simple estimators misbehave. This has led
to the increasing adoption of “deep learning” methods that implicitly account
for morphological characteristics while estimating quantities of interest,
most often mass (e.g. Ntampaka et al. 2019; Yan et al. 2020; de Andres et al.
2024; Ho et al. 2023). The greater flexibility of these approaches typically
comes with the requirement of significant numbers of images on which to
“train” them, typically in the form of mock observations of clusters formed in
hydrodynamical simulations. While the variety of observed morphologies in
detail poses a significant challenge to even the most advanced hydrodynamical
simulations, given the wide dynamic range involved and the uncertain role of
magnetic fields and feedback physics at small scales, realistic features on
intermediate and large scales can be reproduced (e.g. Darragh-Ford et al.
2023). Given that such hydrodynamic simulations remain costly to produce in
large numbers, there have been numerous proposed approaches to predicting the
gas properties or appearance of clusters in less expensive N-body simulations,
either based on simple physical models or by calibrating a translation from
hydrodynamic to N-body simulations (sometimes called “baryon pasting”; e.g.
Ostriker et al. 2005; Bode et al. 2007, 2009; Shaw et al. 2010; Sehgal et al.
2010; Schneider & Teyssier 2015; Flender et al. 2016, 2017; Osato & Nagai
2023).

Given an existing set of hydrodynamical simulations to start from, however,
the specific and limited task of generating novel X-ray images (or other
observables) for training purposes can be accomplished simply and without
requiring new simulations. In this work, we employ mock X-ray images produced
for clusters simulated in The Three Hundred project (Cui et al., 2018;
Ansarifard et al., 2020), a suite of 324 high-resolution hydrodynamical re-
simulations of halos from the dark-matter-only MultiDark simulations. We use
the known self-similarity of cluster brightness over a wide range in radius,
as well as physical considerations such as isotropy, to maximize the
similarity of their images before encoding their morphologies via a high-order
polynomial expansion. This representation is then diagonalized, providing a
straightforward if high-dimensional basis for generating realistic cluster
X-ray morphologies. Throughout, we use the Symmetry, Peakiness and Alignment
(SPA) morphological metrics of Mantz et al. (2015) as a benchmark for how well
this representation reproduces the information present in the original images.

The layout of the paper is as follows. The simulated image data are described
in Section 2. In Section 3, we review the definition of the SPA metrics that
we compare to and describe the construction of the new, high-dimensional
cluster image basis. Section 4 discusses the resulting distribution of
morphological features as a function of dynamical state and redshift. We
consider potential applications and future directions, and conclude, in
Section 5.

##  2 Data

Our data set is composed of 324 galaxy clusters from The Three Hundred project
hydrodynamical simulations (Cui et al., 2018), which are re-simulations of the
most massive clusters at $z=0$ identified using the Amiga halo finder
(Knollmann & Knebe, 2009) from the (1 Gpc/h)3 dark-matter-only Multi-Dark
Plank 2 simulations, which assume a flat $\Lambda$CDM cosmological model with
parameters $h=0.678$ and $\Omega_{\mathrm{m}}=0.307$ (Klypin et al., 2016).
These halos were re-simulated using gadget-x (Rasia et al., 2015), including
baryonic and hydrodynamic physics, in smaller regions of radius $15h^{-1}$
Mpc. We use mock X-ray maps produced at multiple redshifts, centered on the
most massive halo present in each box at a given redshift (see Ansarifard et
al. 2020 for details). In detail, these are maps of X-ray emissivity in the
rest-frame energy band 0.5–2.0 keV, integrated along the line of sight and
produced along 3 orthogonal projections. All maps are $4\times 4$ Mpc in size
with a resolution of 3.9 kpc and a line-of-sight integration length of 10 Mpc,
regardless of redshift. The maps were converted to images of X-ray surface
brightness in the observer-frame 0.6–2.0 keV band using temperature-dependent
K-corrections (see Darragh-Ford et al. 2023). We employ such maps generated at
redshifts 0.333, 0.592 and 0.986, between which the mass range of the cluster
sample evolves from $M_{200}\sim(0.6$–$12)\times 10^{14}\,M_{\sun}$ to
$\sim(2.5$–$25)\times 10^{14}\,M_{\sun}$.

##  3 Methods

In this section, we detail the development of our new generative model of
cluster X-ray morphology. To provide a baseline for how well the model encodes
key morphological information, we first extract the SPA metrics for each
cluster and projection, as described in Section 3.1. Sections 3.2–3.5 then
step through the procedure used to define our model.

Below, we employ the conventional “overdensity”, $\Delta$, which jointly
defines a family of characteristic cluster masses and radii as

$M_{\Delta}=\frac{4\pi}{3}\Delta\rho_{\mathrm{cr}}(z)r_{\Delta}^{3},$ (1)

where $\rho_{\mathrm{cr}}(z)$ is the critical density at the cluster’s
redshift. Broadly speaking, archival data from modern X-ray observatories like
Chandra and XMM-Newton can probe the 2-dimensional shape of many clusters out
to approximately $r_{2500}$ and firmly detect emission out to $r_{500}\approx
2.3\,r_{2500}$, although there is naturally variation with exposure time,
extent, redshift and overall luminosity. The masses of the simulated clusters
noted in Section 2 correspond to values of $r_{500}$ evolving from $\sim
0.8$–1.3 Mpc to $\sim 0.5$–0.8 Mpc between redshifts 0.333 and 0.986.

###  3.1 SPA Metrics

Darragh-Ford et al. (2023) describe the procedure for computing SPA metrics of
the mock X-ray images from The Three Hundred simulations. We reproduce their
methods, with the exception that the metrics are computed only within a
circular region of radius $r_{500}$ (see Section 3.2.1) rather than on the
complete images. This allows a fair comparison of the metrics derived from the
original images and from images produced from the model, which is only defined
within a circle. Below, we review the most important aspects of SPA for this
discussion; the algorithm is described in full detail by Mantz et al. (2015).

A key aspect of the SPA approach is to approximately remove the redshift and
temperature (i.e. mass) dependent scaling of the observed surface brightness
of a cluster’s X-ray emission (Mantz et al. 2015. As modified by Darragh-Ford
et al. 2023), this scaling factor is

$f_{S}=K(z,T)\frac{E(z)^{3}}{(1+z)^{4}}\left(\frac{kT}{\mathrm{keV}}\right)\left(\frac{5.7\times
10^{-8}\,\mathrm{erg}}{\mathrm{Ms}\,\mathrm{cm}^{2}\,(0.984^{\prime\prime})^{2}}\right).$
(2)

Here, $K$ converts between bolometric flux in energy units and photon flux in
the observed energy band,111When applied to real data, this term would also
account for Galactic absorption. $kT$ is the temperature of the cluster gas at
approximately $r_{2500}$, and $E(z)=H(z)/H_{0}$ is the normalized Hubble
parameter at the cluster’s redshift. In units of $f_{S}$, the regions of
similar observed surface brightness, $S$, in different clusters correspond to
broadly comparable overdensities, with the exception of central regions where
the presence or absence of a cool core causes significant variations in
brightness.

The first step of the SPA algorithm is to define a global cluster center as
the median photon position of the flat-fielded, background-subtracted image.
Given this center, the SPA metrics themselves can be understood as:

  1. 1.

Peakiness ($p$) quantifies the average scaled surface brightness in the
cluster center, defined as radii where an azimuthally averaged surface
brightness profile exceeds $\sim 0.05f_{S}$.

  2. 2.

Symmetry ($s$) quantifies the agreement between the global center and the
centers of 5 ellipses, fit to isophotes that logarithmically span the range
0.002–0.05 $f_{S}$.

  3. 3.

Alignment ($a$) quantifies the agreement among the centers of the above
ellipses.

A cluster image is categorized as relaxed if it simultaneously exceeds
specific thresholds in each metric ($s>0.87$, $p>-0.82$, $a>1.00$); complete
details and formulae are given by Mantz et al. (2015).

###  3.2 Constructing an Image Basis

Our starting point is the set of Zernike polynomials (ZPs), comprising an
orthogonal basis of functions over the unit circle that can, in principle,
encode an arbitrary circumscribed image. Capalbo et al. (2021) have used this
basis to quantify the morphology of simulated maps of the thermal Sunyaev-
Zel’dovich effect of The Three Hundred clusters, demonstrating a relationship
between the amplitude of the asymmetric polynomial coefficients and
simulation-based metrics for dynamical state. From the standpoint of
developing a generative model, the main challenge is to identify and
parameterize the subspace of this general model space containing realistic
cluster morphologies, as defined by the simulated images. This, along with
computational limitations, necessarily requires us to choose a maximum order
of polynomial (corresponding to a smallest spatial scale) to include in the
modeling. Given the large dynamic range of surface brightness (typically 4–5
orders of magnitude within $r_{500}$) and morphologically relevant spatial
scales ($\sim 10$–1000 kpc) in X-ray images of clusters, these are non-trivial
decisions.

The above considerations provide key context for our overall strategy in
developing a morphological image basis, which is detailed in the remainder of
this section. First, we apply a standardization to the original images, with
the aim of leveraging physical considerations to minimize the image-to-image
variation _before_ encoding anything in the polynomial basis (Section 3.2.1).
To reduce the polynomial order required to adequately encode the presence of
sharply peaked cool cores, we then apply a non-linear radial distortion
(Section 3.3). In Section 3.5, we fit the standardized, distorted images using
the ZP basis, and apply Principal Component Analysis (PCA) to identify a
lower-dimensional model subspace encoding the most important image-to-image
variations. Figure 1 illustrates the procedure.

Figure 1: Our analysis process performed on example image (identified by an
orange square in later figures). Note that all images, including the
“original”, are log-scaled for clarity. Images are first standardized as
described in Section 3.2.1, and then radially distorted such that the center
is magnified (Section 3.3). All images in our dataset are then rotated by an
angle selected as described in Section 3.4. The rightmost panel shows the
image corresponding to the ZP representation of this processed input.

####  3.2.1 Image Standardization

The more we can reduce the variation among images a priori on the basis of
physical considerations, the lower-dimensional our final morphology model need
be. To take a simple example, images that differ only by a relative rotation
should ideally be encoded by an identical set of model parameters apart from
one. Since this behavior does not naturally emerge from the procedure
described in the previous section, we explicitly apply a series of simple,
standardizing transformations to the original images at the outset. That is,
our approach is opposite to that normally used in computer vision: rather than
augmenting the training data with many transformed replicates (e.g. rotated
versions of each image), we instead define a preferred rotation with respect
to which the model is defined. The final set of model parameters then need
only be supplemented with a position angle. A similar approach can be taken
with other simple image transformations, namely centering (translation),
extent (zoom) and scaling. However, while it is desirable for the part of the
model encoding cluster morphology to be invariant under rotation, these other
transformations are physically significant, providing us the opportunity to
build physical considerations into the model at a fundamental level.

  1. 1.

Scaling: Exactly the same considerations that motivate the scaling of surface
brightness by the self-similar factor $f_{S}$ in the SPA approach apply here
(Section 3.1). Once divided by $f_{S}$, surface brightness profiles of
clusters are observed to have relatively small scatter (few tens of per cent;
Mantz et al. 2015), apart from small radii where cool cores can boost the
brightness by an order of magnitude. We also logarithmically scale the images,
reducing the dynamic range of the data to be modeled from several orders of
magnitude to, typically, a single order of magnitude, making them much more
simply encoded by a basis of polynomial functions.

  2. 2.

Centering: The centers of the simulated images are, by construction, the
position of the gravitational potential minimum. We maintain this centering
for the circular region within which the modeling takes place.

  3. 3.

Radial extent: We define $r_{500}$, as computed from the spherically averaged
enclosed mass profile about the above center, as the radial extent of the
model. Subsequent analysis is thus limited to this circular region of the
original image. As noted above, this choice of radial extent should provide a
model that comfortably applies to the typically visible portions of clusters
observed by Chandra and XMM-Newton.

  4. 4.

Rotation: Following the logic above, we define a preferred rotation; however,
in practice, this is applied after the radial distortion introduced in the
next section. We therefore defer further discussion to Section 3.4.

###  3.3 Radial Distortion

The smallest scale represented by the Zernike basis shrinks only modestly
quickly with order, while the computational expense increases super-linearly.
The sharpness of the brightness peaks of cool cores, which in extreme cases
have power-law like profiles down to the resolution limit of current X-ray
observatories, thus poses a challenge, even with logarithmic scaling of the
images. To make the representation of small-scale features in cluster cores
more tractable, we therefore introduce a non-linear radial distortion of the
standardized images. Specifically, the transformation is

$x\rightarrow x^{\prime}=A\ln\left(1+\frac{x}{\xi}\right),$ (3)

where $x=r/r_{500}$, $\xi=0.05$, and $A$ normalizes the new radial coordinate
such that $x^{\prime}=1$ corresponds to $x=1$. This has the effect of greatly
magnifying the cluster center, at the expense of the outskirts, with the
transformation approaching linearity for $x\gg\xi$. The distortion thus does
not help to resolve comparably small features outside of the cluster center,
but these are relatively rare in real clusters (e.g., the merging cluster
Abell 115). Figure 1 shows the impact of this procedure on an example image.
(Note that we reverse this distortion before displaying model images in all
other figures.)

###  3.4 Zernike Polynomial Fitting and Rotation

Once a simulated image is standardized and distorted, we use the zernike
Python package to determine the ZP coefficients up to a maximum order of
$n=30$ (a total of 496 coefficients) by numerically evaluating the inner
product of each polynomial with the image.

Once the ZP coefficients for an image have been found, we standardize its
rotation. This step is done after finding the coefficients for convenience, as
the transformation of ZP coefficients under rotation is particularly simple.
To define a preferred rotation, we first find the position angle that
maximizes the $Z_{2}^{2}$ coefficient, and then break the remaining symmetry
(between two angles separated by $180^{\circ}$) by maximizing the $Z_{1}^{1}$
coefficient. This is notionally similar, though not identical, to aligning the
major axis of an elliptical cluster with the $x$ axis, with more brightness
(e.g. due to substructure) lying in the positive $x$ direction than the
negative $x$ direction.

###  3.5 Principal Component Analysis

The final step of the model development is to perform PCA on the ZP
coefficients of the standardized, distorted and rotated images. The motivation
here is to identify a new, lower-dimensional model that spans the space of
realistic cluster morphologies (as described by the simulated images) without
the additional complexity required to generate an arbitrary image.

Before finding the principal components, we augment the data using the one
simple image transformation not addressed in the preceding sections:
reflection of the standardized images over the $x$ axis. We follow the
standard practice of centering the input data (subtracting the mean and
rescaling to unit variance), and use the scikit-learn Python package to carry
out the PCA.

##  4 Results and Discussion

Figure 2:  MADs of the SPA metrics (comparing original and modeled images) are
plotted against the maximum ZP order and magnification parameter, the default
values of which are $n=30$ and $\xi=0.05$, respectively. Solid and dashed
curves respectively show the MADs for all simulated images in the $z=0.333$
snapshot, and only relaxed images.  Figure 3:  Scaled surface brightness
profiles of the cores of SPT J2344, the peakiest cluster observed by (Mantz et
al., 2015, open black circles), and two similarly peaky simulated clusters
(filled blue circles and red squares; all have $p>-0.5$). The latter, the
peakiest simulated cluster, has an extreme value of the first principal
component, PC0, of $\sim 11$. Solid curves show profiles of the images
reconstructed from the ZP model. The dashed line shows the threshold relevant
for the definition of peakiness Mantz et al. (2015).

###  4.1 Impact of the Maximum Order and Magnification

As described in Section 3 above, the procedures of ZP fitting and radial
distortion require us to choose a maximum order of ZP and a magnification
parameter, respectively. Both cases demand a compromise; in choosing a maximum
ZP order, we strike a balance between computational overheads and the ability
to reproduce small-scale features. Varying the magnification parameter, we
trade detail at the outskirts of the cluster for additional resolution in
cluster cores. To explore these trade-offs, we compare the original and
modeled images of the $z=0.333$ snapshots, using the SPA metrics as proxies
for how well important morphological features are encoded by the model.

Figure 2 shows how the median absolute deviation (MAD) of each metric varies
when either the maximum ZP order or the magnification parameter is changed.
Here deviation refers to the differences between metrics computed from the
original images and images reconstructed from the ZP model. As expected, all
MADs decrease monotonically with increasing order. As central magnification
increases ($\xi$ decreases), we also see greater agreement, although gains
within the explored range are relatively modest. The model is quantitatively
better at reproducing the SPA metrics for relaxed clusters, which is intuitive
given their overall simpler morphologies. In this context, we note that
typical statistical uncertainties on SPA measurements for real clusters with
Chandra are $\sim 0.05$ in peakiness and $\sim 0.10$ in symmetry and alignment
(Mantz et al., 2015).

Despite this agreement, we note that there are features that appear in some
simulated clusters that are not seen in real data, in particular in cluster
cores (see also Darragh-Ford et al. 2023). Figure 3 compares the surface
brightness profile of the peakiest simulated cluster, which displays a
relatively large, flat core on which an exceptionally sharp peak is
superimposed, with the much smoother profile of the peakiest real cluster from
Mantz et al. (2015). The extremely compact (of order 10 kpc) cores in some
simulated clusters, which are an expected consequence of their limited
resolution (Rasia et al., 2015), are not well reproduced by the model with any
combination of maximum order and magnification that we considered. On the
other hand, the simulations also include peaky systems that better resemble
real clusters, as shown in the figure, and that are easily described by the
baseline model with $n=30$ and $\xi=0.05$.

###  4.2 The PC Representation

The bottom row of Figure 4 visualizes the most important features encoded by
the PCA. The first of these images corresponds to the origin of PC space; that
is, the average cluster image after standardization. Each of the remaining
images is the difference (in log-brightness) between a model image
corresponding to a unit vector along a given PC and the origin. For
comparison, the top row shows the first nine ZPs (the origin of ZP space
corresponds to an image of all zeros, visually identical to the constant
$Z_{0}^{0}$ polynomial). While both representations ultimately contain the
same information, the origin and basis directions of the PC representation, by
construction, more closely resemble visually identifiable features of the
input images.

The impact of the standard rotation imposed in Section 3.4 is evident here,
with many of the features encoded in these first PCs appearing to be
concentrated near the positive $x$ axis. The shape of the cluster core is the
other clear feature represented in these first PCs, with several of them
modulating the central brightness on different scales; PC0 in particular
appears related to the unphysically sharp peaks atop flatter cores noted in
Section 4.1 (see also Figure 3).

Figure 4: Top row: visualizations of the first nine ZPs (in canonical order).
Bottom row: the leftmost image corresponds to the origin of PC space. The
others are difference images between unit vectors along the first 8 PCs and
the origin. Color tables are independent in each panel and are symmetric about
zero (except for the origin image).  Figure 5: Coefficient distributions of
the first eight PCs for the simulated cluster images (including their
reflections). Diagonal: smoothed histograms for each PC, distinguishing images
classified as relaxed (blue) and unrelaxed (red) by the SPA metrics. Lower-
left triangle: pairwise distributions of coefficients for relaxed (blue
circles) and unrelaxed (red crosses) images. Upper-right triangle: as above,
but colors and symbols distinguish 4 example images to be referred to later in
the text, with all other points shown in gray. See Figure 8 for images of the
exemplars.

Histograms and pairwise distributions of coefficients for these PCs are shown
in Figure 5, distinguishing between relaxed (blue) and unrelaxed (red) images.
Note that, by construction, the average of each coefficient is zero; the
correlations among coefficients are also effectively zero, although this need
not have been the case (correlation coefficients $<10^{-14}$ in absolute
value). In most cases, the distributions for both relaxed and unrelaxed
clusters peak at zero, with the main difference being a larger scatter for
unrelaxed clusters. However, within this abbreviated group of PCs, we see
clear differences in PC0 (encoding an unphysical peak, as noted above, as well
as an excess at relatively small radii; here relaxed clusters prefer
_negative_ values) and PC7 (encoding a larger, bright peak; here relaxed
clusters prefer _positive_ values). Due to the augmentation of the data by
reflecting over the $x$ axis in Section 3.5, each PC is either perfectly
symmetric (e.g. 0, 1, 3, 4, 7) or anti-symmetric (e.g. 2, 5, 6) with respect
to $x$-axis reflection. In the upper-right triangle of the figure, different
colors/symbols show the PC coefficient values for 4 example images chosen for
their differing visual appearances (see Figure 8).

Beyond this overall distribution, we observe formally significant trends as a
function of redshift in the $x$-reflection symmetric PCs222The anti-symmetric
PCs necessarily have a distribution that is perfectly symmetric about zero at
all redshifts. (Figure 6) when fitting a linear model, although changes in the
mean of each PC are generally small compared with the scatter. The scatter in
all coefficients consistently decreases with increasing redshift. This seems
to imply that clusters look more similar to one another at high redshifts,
regardless of their specific morphologies. It is possible that the overall
trends are related to growth, given that $r_{500}$ is larger at smaller
redshifts; we might therefore expect that some features – those that do not
scale with the cluster size – show up in different PCs at different redshifts,
even if they are present at the same level. Similarly, the contrast between
cool cores and the self-similar cluster profile on larger scales is known to
decrease with redshift (Mantz et al., 2015; McDonald et al., 2017). The
decreasing scatter might be explained by the natural reduction in sound
crossing time at correspondingly higher redshifts, although the higher merger
rate at earlier times might be expected to counteract this to some extent.

Figure 6:  Box and whisker plots depicting the redshift dependence of the
first three principal components. PC0 has consistent decreases in median value
with redshift, PC1 has less pronounced changes, and PC2 has consistent median
values across redshifts. All three PCs shown have decreased range with
increased redshift.

###  4.3 Fidelity as a Function of the Number of PCs

We next consider how much dimensional reduction the PC representation might
provide, with the caveat that this will depend on the particular application.
The left panel of Figure 7 shows a standard PCA diagnostic, the variance
explained by each PC (individually as well as cumulatively). After the first 4
PCs, the explained variance decreases slowly, with e.g. almost 300 PCs
required to explain 90 per cent of the total variance.

The slowness of this “convergence” can be seen in Figure 8, which shows 4
example images (identified in Figure 5) as modeled by increasing numbers of
PCs, in order. While the broad shape of each image is established with
relatively few PCs, and resemblance to the original images clearly improves
with additional PCs, these improvements are visually incremental after 200–300
PCs. More quantitatively, we can compare the SPA metrics obtained from the
original images with those computed from model images using a subset of PCs,
as in the right panel of Figure 7, which similarly shows steady improvement up
until 300–400 PCs are in use. As previously, we note that measurement
uncertainties in these metrics from real data are typically $\sim 0.05$–0.10.

Figure 7:  Left: Individual (blue line) and cumulative (red line) variance
explained by each principal component of the model. Right: Similarly to Figure
2, MADs of the SPA parameters (comparing original and modeled images), as a
function of the number of PCs used in the model image reconstruction. A single
projection of each $z=0.333$ simulated cluster is used in this case.  Figure
8: Shown in the left column are four example clusters, followed by partial
reconstructions of each using various numbers of PCs (in order). Each image is
color-scaled separately.

##  5 Conclusions

In this work, we have explored one strategy for defining a basis of cluster
X-ray morphologies, based on simulated X-ray emissivity maps from The Three
Hundred project. Our approach aims to maximize the similarity of the simulated
images at the outset by taking advantage of physical insights such as the
cluster mass and center, the approximately self-similar scaling of surface
brightness, and rotational alignment with surrounding large-scale structure
(as manifest in overall ellipticity and dipolar asymmetry). After this
standardization, images are distorted to reduce the sharpness of features in
cluster centers, and fitted by a high-order set of polynomial functions. PCA
of the polynomial coefficients provides a basis set of images that are more
naturally interpretable in the context of cluster images than the original
polynomials, even though individual PCs generally do not correspond to
identifiable, physically meaningful features.

One immediate application of this approach is the generation of novel,
realistic cluster images. Figure 9 shows three examples, produced by randomly
drawing PC coefficients from normal distributions with mean zero and standard
deviations equal to the scatter in the simulated data.333Code used to do so
can be found at https://github.com/kipac-xoc/cximb. One could compute a more
detailed description of the high-dimensional PC coefficient distribution, but
even this simple method produces complex images that qualitatively resemble
the simulated clusters. Such mock images may provide a useful alternative to
simple, parametric models for testing for cluster morphology or feature
detection algorithms (e.g. cavity-finding; Fort 2017; Plšek et al. 2024). Our
results thus provide a near-instantaneous alternative to running new
hydrodynamical simulations, as well as “baryon pasting” approaches (which
require N-body simulations) when _only_ novel X-ray images, as opposed to
complete physical descriptions of clusters, are needed. Naturally, care must
be taken, as unphysical features present in the original simulations will also
be reproduced; on the other hand, it may be straightforward to suppress such
features if they are primarily represented by a single PC (e.g. PC0). With
appropriate restriction of the input data, one could straightforwardly
construct more specific models, e.g. for example to generate images of only
relaxed clusters, only first-pass mergers, etc. Application to other
wavelengths would also be straightforward given input simulations, though
questions such as the appropriate maximum polynomial order and radial
distortion would need to be revisited for those cases.

Figure 9: Three fictional cluster images generated by choosing Principal
Component coefficients at random from a Gaussian distribution for each PC, as
well as a random rotation angle.

Given the high dimensionality of the model, its use with real X-ray data will
likely require application-specific dimensionality reduction. For example, we
find that more than half of the almost 500 PCs must be included for the SPA
morphology metrics to be faithfully reproduced (within typical measurement
uncertainties) in a model image. This is not entirely surprising, given that
the PCA is in no way informed by SPA, nor more generally by knowledge of what
features distinguish “relaxed” and “unrelaxed” morphologies. As the bulk of
the observational literature relies on as few as 1–3 quantitative metrics of
X-ray morphology (e.g. Postman et al. 2012; Mahdavi et al. 2013; Martino et
al. 2014; Mantz et al. 2015; Campitiello et al. 2022), it seems reasonable
that a small number of informed combinations of PCs (or some other
decomposition of the original parameter space) may be sufficient for many
purposes. Future work will investigate methods for producing such low-
dimensional models, tailored to identifying specific morphological types from
real images.

MB was supported by Stanford University’s Physics Research Program. JP was
supported by the U.S. Department of Energy, Office of Science, Office of
Workforce Development for Teachers and Scientists (WDTS) under the Science
Undergraduate Laboratory Internships Program (SULI). We acknowledge support
from the U.S. Department of Energy under contract number DE-AC02-76SF00515 to
SLAC National Accelerator Laboratory. This work has been made possible by the
“The Three Hundred” collaboration (https://www.the300-project.org).

## References

  * Ansarifard et al. (2020) Ansarifard, S., Rasia, E., Biffi, V., et al. 2020, A&A, 634, A113, doi: 10.1051/0004-6361/201936742
  * Antonello & Verhaegen (2015) Antonello, J., & Verhaegen, M. 2015, J. Opt. Soc. Am. A, 32, 1160, doi: 10.1364/JOSAA.32.001160
  * Arévalo et al. (2016) Arévalo, P., Churazov, E., Zhuravleva, I., Forman, W. R., & Jones, C. 2016, ApJ, 818, 14, doi: 10.3847/0004-637X/818/1/14
  * Ascasibar & Markevitch (2006) Ascasibar, Y., & Markevitch, M. 2006, ApJ, 650, 102, doi: 10.1086/506508
  * Bode et al. (2009) Bode, P., Ostriker, J. P., & Vikhlinin, A. 2009, ApJ, 700, 989, doi: 10.1088/0004-637X/700/2/989
  * Bode et al. (2007) Bode, P., Ostriker, J. P., Weller, J., & Shaw, L. 2007, ApJ, 663, 139, doi: 10.1086/518432
  * Böhringer et al. (2010) Böhringer, H., Pratt, G. W., Arnaud, M., et al. 2010, A&A, 514, A32, doi: 10.1051/0004-6361/200913911
  * Buote & Tsai (1995) Buote, D. A., & Tsai, J. C. 1995, ApJ, 452, 522, doi: 10.1086/176326
  * Campitiello et al. (2022) Campitiello, M. G., Ettori, S., Lovisari, L., et al. 2022, A&A, 665, A117, doi: 10.1051/0004-6361/202243470
  * Capalbo et al. (2021) Capalbo, V., De Petris, M., De Luca, F., et al. 2021, MNRAS, 503, 6155, doi: 10.1093/mnras/staa3900
  * Churazov et al. (2012) Churazov, E., Vikhlinin, A., Zhuravleva, I., et al. 2012, MNRAS, 421, 1123, doi: 10.1111/j.1365-2966.2011.20372.x
  * Cui et al. (2018) Cui, W., Knebe, A., Yepes, G., et al. 2018, MNRAS, 480, 2898, doi: 10.1093/mnras/sty2111
  * Darragh-Ford et al. (2023) Darragh-Ford, E., Mantz, A. B., Rasia, E., et al. 2023, MNRAS, 521, 790, doi: 10.1093/mnras/stad585
  * de Andres et al. (2024) de Andres, D., Cui, W., Yepes, G., et al. 2024, MNRAS, 528, 1517, doi: 10.1093/mnras/stae071
  * de Vries et al. (2023) de Vries, M., Mantz, A. B., Allen, S. W., et al. 2023, MNRAS, 518, 2954, doi: 10.1093/mnras/stac3285
  * Eckert et al. (2015) Eckert, D., Roncarelli, M., Ettori, S., et al. 2015, MNRAS, 447, 2198, doi: 10.1093/mnras/stu2590
  * Fabian (2012) Fabian, A. C. 2012, ARA&A, 50, 455, doi: 10.1146/annurev-astro-081811-125521
  * Flender et al. (2016) Flender, S., Bleem, L., Finkel, H., et al. 2016, ApJ, 823, 98, doi: 10.3847/0004-637X/823/2/98
  * Flender et al. (2017) Flender, S., Nagai, D., & McDonald, M. 2017, ApJ, 837, 124, doi: 10.3847/1538-4357/aa60bf
  * Fort (2017) Fort, S. 2017, In Workshop on Deep Learning for Physical Sciences (DLPS 2017), arXiv:1712.00523, doi: 10.48550/arXiv.1712.00523
  * Ho et al. (2023) Ho, M., Soltis, J., Farahi, A., et al. 2023, MNRAS, 524, 3289, doi: 10.1093/mnras/stad2005
  * Jeltema et al. (2005) Jeltema, T. E., Canizares, C. R., Bautz, M. W., & Buote, D. A. 2005, ApJ, 624, 606, doi: 10.1086/428940
  * Johnson et al. (2012) Johnson, R. E., Zuhone, J., Jones, C., Forman, W. R., & Markevitch, M. 2012, ApJ, 751, 95, doi: 10.1088/0004-637X/751/2/95
  * Klypin et al. (2016) Klypin, A., Yepes, G., Gottlöber, S., Prada, F., & Heß, S. 2016, MNRAS, 457, 4340, doi: 10.1093/mnras/stw248
  * Knollmann & Knebe (2009) Knollmann, S. R., & Knebe, A. 2009, ApJS, 182, 608, doi: 10.1088/0067-0049/182/2/608
  * Mahdavi et al. (2013) Mahdavi, A., Hoekstra, H., Babul, A., et al. 2013, ApJ, 767, 116, doi: 10.1088/0004-637X/767/2/116
  * Mantz et al. (2015) Mantz, A. B., Allen, S. W., Morris, R. G., et al. 2015, MNRAS, 449, 199, doi: 10.1093/mnras/stv219
  * Markevitch & Vikhlinin (2007) Markevitch, M., & Vikhlinin, A. 2007, Phys. Rep., 443, 1, doi: 10.1016/j.physrep.2007.01.001
  * Martino et al. (2014) Martino, R., Mazzotta, P., Bourdin, H., et al. 2014, MNRAS, 443, 2342, doi: 10.1093/mnras/stu1267
  * Massey et al. (2011) Massey, R., Kitching, T., & Nagai, D. 2011, MNRAS, 413, 1709, doi: 10.1111/j.1365-2966.2011.18246.x
  * McDonald et al. (2017) McDonald, M., Allen, S. W., Bayliss, M., et al. 2017, ApJ, 843, 28, doi: 10.3847/1538-4357/aa7740
  * McNamara et al. (2000) McNamara, B. R., Wise, M., Nulsen, P. E. J., et al. 2000, ApJ, 534, L135, doi: 10.1086/312662
  * Mirakhor & Walker (2021) Mirakhor, M. S., & Walker, S. A. 2021, MNRAS, 506, 139, doi: 10.1093/mnras/stab1768
  * Mohr et al. (1993) Mohr, J. J., Fabricant, D. G., & Geller, M. J. 1993, ApJ, 413, 492, doi: 10.1086/173019
  * Ntampaka et al. (2019) Ntampaka, M., ZuHone, J., Eisenstein, D., et al. 2019, ApJ, 876, 82, doi: 10.3847/1538-4357/ab14eb
  * Nurgaliev et al. (2013) Nurgaliev, D., McDonald, M., Benson, B. A., et al. 2013, ApJ, 779, 112, doi: 10.1088/0004-637X/779/2/112
  * Osato & Nagai (2023) Osato, K., & Nagai, D. 2023, MNRAS, 519, 2069, doi: 10.1093/mnras/stac3669
  * Ostriker et al. (2005) Ostriker, J. P., Bode, P., & Babul, A. 2005, ApJ, 634, 964, doi: 10.1086/497122
  * Paterno-Mahler et al. (2013) Paterno-Mahler, R., Blanton, E. L., Randall, S. W., & Clarke, T. E. 2013, ApJ, 773, 114, doi: 10.1088/0004-637X/773/2/114
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., et al. 2011, Journal of Machine Learning Research, 12, 2825
  * Peterson & Fabian (2006) Peterson, J. R., & Fabian, A. C. 2006, Phys. Rep., 427, 1, doi: 10.1016/j.physrep.2005.12.007
  * Plšek et al. (2024) Plšek, T., Werner, N., Topinka, M., & Simionescu, A. 2024, MNRAS, 527, 3315, doi: 10.1093/mnras/stad3371
  * Postman et al. (2012) Postman, M., Coe, D., Benítez, N., et al. 2012, ApJS, 199, 25, doi: 10.1088/0067-0049/199/2/25
  * Rasia et al. (2013) Rasia, E., Meneghetti, M., & Ettori, S. 2013, The Astronomical Review, 8, 010000.  https://arxiv.org/abs/1211.7040
  * Rasia et al. (2015) Rasia, E., Borgani, S., Murante, G., et al. 2015, ApJ, 813, L17, doi: 10.1088/2041-8205/813/1/L17
  * Roediger et al. (2011) Roediger, E., Brüggen, M., Simionescu, A., et al. 2011, MNRAS, 413, 2057, doi: 10.1111/j.1365-2966.2011.18279.x
  * Santos et al. (2008) Santos, J. S., Rosati, P., Tozzi, P., et al. 2008, A&A, 483, 35, doi: 10.1051/0004-6361:20078815
  * Schneider & Teyssier (2015) Schneider, A., & Teyssier, R. 2015, J. Cosmology Astropart. Phys, 2015, 049, doi: 10.1088/1475-7516/2015/12/049
  * Sehgal et al. (2010) Sehgal, N., Bode, P., Das, S., et al. 2010, ApJ, 709, 920, doi: 10.1088/0004-637X/709/2/920
  * Shaw et al. (2010) Shaw, L. D., Nagai, D., Bhattacharya, S., & Lau, E. T. 2010, ApJ, 725, 1452, doi: 10.1088/0004-637X/725/2/1452
  * Simionescu et al. (2011) Simionescu, A., Allen, S. W., Mantz, A., et al. 2011, Science, 331, 1576, doi: 10.1126/science.1200331
  * Simionescu et al. (2012) Simionescu, A., Werner, N., Urban, O., et al. 2012, ApJ, 757, 182, doi: 10.1088/0004-637X/757/2/182
  * Urban et al. (2011) Urban, O., Werner, N., Simionescu, A., Allen, S. W., & Böhringer, H. 2011, MNRAS, 414, 2101, doi: 10.1111/j.1365-2966.2011.18526.x
  * Urban et al. (2014) Urban, O., Simionescu, A., Werner, N., et al. 2014, MNRAS, 437, 3939, doi: 10.1093/mnras/stt2209
  * Vikhlinin et al. (2007) Vikhlinin, A., Burenin, R., Forman, W. R., et al. 2007, in Heating versus Cooling in Galaxies and Clusters of Galaxies, ed. H. Böhringer, G. W. Pratt, A. Finoguenov, & P. Schuecker, 48, doi: 10.1007/978-3-540-73484-0_9
  * Walker et al. (2012a) Walker, S. A., Fabian, A. C., Sanders, J. S., & George, M. R. 2012a, MNRAS, 424, 1826, doi: 10.1111/j.1365-2966.2012.21282.x
  * Walker et al. (2012b) Walker, S. A., Fabian, A. C., Sanders, J. S., George, M. R., & Tawara, Y. 2012b, MNRAS, 422, 3503, doi: 10.1111/j.1365-2966.2012.20860.x
  * Yan et al. (2020) Yan, Z., Mead, A. J., Van Waerbeke, L., Hinshaw, G., & McCarthy, I. G. 2020, MNRAS, 499, 3445, doi: 10.1093/mnras/staa3030
  * Zhuravleva et al. (2018) Zhuravleva, I., Allen, S. W., Mantz, A., & Werner, N. 2018, ApJ, 865, 53, doi: 10.3847/1538-4357/aadae3
  * Zhuravleva et al. (2019) Zhuravleva, I., Churazov, E., Schekochihin, A. A., et al. 2019, Nature Astronomy, 3, 832, doi: 10.1038/s41550-019-0794-z
  * Zhuravleva et al. (2015) Zhuravleva, I., Churazov, E., Arévalo, P., et al. 2015, MNRAS, 450, 4184, doi: 10.1093/mnras/stv900
  * Zhuravleva et al. (2016) —. 2016, MNRAS, 458, 2902, doi: 10.1093/mnras/stw520
  * ZuHone et al. (2011) ZuHone, J. A., Markevitch, M., & Lee, D. 2011, ApJ, 743, 16, doi: 10.1088/0004-637X/743/1/16


# Predicting emergent linguistic compositions through time:
Syntactic frame extension via multimodal chaining

Lei Yu1, Yang Xu1, 2, 3
1 Department of Computer Science, University of Toronto, Toronto, Canada
2 Cognitive Science Program, University of Toronto, Toronto, Canada
3 Vector Institute for Artificial Intelligence, Toronto, Canada
<EMAIL_ADDRESS>
###### Abstract

Natural language relies on a finite lexicon to express an unbounded set of
emerging ideas. One result of this tension is the formation of new
compositions, such that existing linguistic units can be combined with
emerging items into novel expressions. We develop a framework that exploits
the cognitive mechanisms of chaining and multimodal knowledge to predict
emergent compositional expressions through time. We present the syntactic
frame extension model (SFEM) that draws on the theory of chaining and
knowledge from “percept”, “concept”, and “language” to infer how verbs extend
their frames to form new compositions with existing and novel nouns. We
evaluate SFEM rigorously on the 1) modalities of knowledge and 2)
categorization models of chaining, in a syntactically parsed English corpus
over the past 150 years. We show that multimodal SFEM predicts newly emerged
verb syntax and arguments substantially better than competing models using
purely linguistic or unimodal knowledge. We find support for an exemplar view
of chaining as opposed to a prototype view and reveal how the joint approach
of multimodal chaining may be fundamental to the creation of literal and
figurative language uses including metaphor and metonymy.

##  1 Introduction

Language users often construct novel compositions through time, such that
existing linguistic units can be combined with emerging items to form novel
expressions. Consider the expression swipe your phone, which presumably came
about after the emergence of touchscreen-enabled smartphones. Here the use of
the verb swipe was extended to express one’s experience with the emerging item
“smartphone”. These incremental extensions are fundamental to adapting a
finite lexicon toward emerging communicative needs. We explore the nature of
cognitive mechanisms and knowledge in the temporal formation of previously
unattested verb-argument compositions, and how this compositionality may be
understood in principled terms.

Figure 1: Preview of the proposed approach of syntactic frame extension. Given
a query noun (circled dot) at time $t$, the framework draws on a combination
of multimodal knowledge and cognitive mechanisms of chaining to predict novel
linguistic expressions for those items. The left panel shows via PCA
projection how a newly emerged noun (i.e., car in 1920s) is assigned
appropriate verb frames by comparing the learned multimodal representation of
the query nouns with support nouns (non-circled dots) that have been
predicated by the frames. The right panel shows a similar verb construction
for a noun that already existed at the time of prediction (i.e., computer in
1980s).

Compositionality is at the heart of linguistic creativity yet a notoriously
challenging topic in computational linguistics and natural language processing
(e.g., Vecchi et al. 2017; Cordeiro et al. 2016; Blacoe and Lapata 2012;
Mitchell and Lapata 2010; Baroni and Zamparelli 2010). For instance, modern
views on the state-of-the-art neural models of language have suggested that
they show some degree of linguistic generalization but are impoverished in
systematic compositionality (see Baroni (2020) for review). Existing work has
also explored the efficacy of neural models in modeling diachronic semantics
(e.g., Hamilton et al., 2016; Rosenfeld and Erk, 2018; Hu et al., 2019;
Giulianelli et al., 2020). However, to our knowledge, no attempt has been made
to examine principles in the formation of novel verb-noun compositions through
time.

We formulate the problem as an inferential process which we call _syntactic
frame extension_. We define syntactic frame as a joint distribution over a
verb predicate, its noun arguments, and their syntactic relations, and we
focus on tackling two related predictive problems: 1) given a novel or
existing noun, infer what verbs and syntactic relations that have not
predicated the noun might emerge to describe it over time (e.g., to drive a
car vs. to fly a car), and 2) given a verb predicate and a syntactic relation,
infer what nouns can be plausibly introduced as its novel arguments in the
future (e.g., drive a car vs. drive a computer).

Figure 1 offers a preview of our framework by visualizing the process of
assigning novel verb frames to describe two query nouns over time. In the
first case, the model incorporated with perceptual and conceptual knowledge
successfully predicts the verb drive to be a better predicate than fly for
describing the novel item car that just emerged at the time of prediction
where linguistic usages are not yet observed (i.e., emergent verb composition
with a novel noun concept). In the second case, the model predicts that hold
is a better predicate than wear for describing the noun computer, which
already existed at the time of prediction (i.e., emergent verb composition
with an existing noun).

Our approach connects two strands of research that were rarely in contact:
cognitive linguistic theories of chaining and computational representations of
multimodal semantics. Work in the cognitive linguistics tradition has
suggested that frame extension is not arbitrary and involves the comparison
between a new item to existing items that are relevant to the frame Fillmore
(1986). Similar proposals lead to the theory of chaining postulating that
linguistic categories grow by linking novel referents to existing ones of a
word due to proximity in semantic space Lakoff (1987); Malt et al. (1999); Xu
et al. (2016); Ramiro et al. (2018); Habibi et al. (2020); Grewal and Xu
(2020). However, such a theory has neither been formalized nor evaluated to
predicting verb frame extensions through time. Separately, computational work
in multimodal semantics has suggested how word meanings warrant a richer
representation beyond purely linguistic knowledge (e.g., Bruni et al. 2012;
Gella et al. 2016, 2017). However, multimodal semantic representations have
neither been examined in the diachronics of compositionality nor in light of
the cognitive theories of chaining. We show that a unified framework that
incorporates the cognitive mechanisms of chaining through deep models of
categorization and multimodal semantic representations predicts the temporal
emergence of novel noun-verb compositions.

##  2 Related work

Our work synthesizes the interdisciplinary areas of cognitive linguistics,
diachronic semantics, meaning representation, and deep learning.

###  2.1 Cognitive mechanisms of chaining

The problem of syntactic frame extension concerns the cognitive theory of
chaining Lakoff (1987); Malt et al. (1999). It has been proposed that the
historical growth of linguistic categories depends on a process of chaining,
whereby novel items link to existing referents of a word that are close in
semantic space, resulting in chain-like structures. Recent studies have
formulated chaining as models of categorization from classic work in cognitive
science. Specifically, it has been shown that chaining may be formalized as an
exemplar-based mechanism of categorization emphasizing semantic neighborhood
profile Nosofsky (1986), which contrasts with a prototype-based mechanism that
emphasizes category centrality Reed (1972); Lakoff (1987); Rosch (1975). This
computational approach to chaining has been applied to explain word meaning
growth in numeral classifiers Habibi et al. (2020) and adjectives Grewal and
Xu (2020). Unlike these previous studies, we consider the open issue whether
cognitive mechanisms of chaining might be generalized to verb frame extension
which draws on rich sources of knowledge. It remains critically undetermined
how “shallow models” such as the exemplar model can function or integrate with
deep neural models Mahowald et al. (2020); McClelland (2020), and how it might
fair with the alternative mechanism of prototype-based chaining in the context
of verb frame extension. We address both of these theoretical issues in a
framework that explores these alternative mechanisms of chaining in light of
probabilistic deep categorization models.

###  2.2 Diachronic semantics in NLP

The recent surge of interest in NLP on diachronic semantics has developed
Bayesian models of semantic change (e.g., Frermann and Lapata, 2016),
diachronic word embeddings (e.g., Hamilton et al., 2016), and deep
contextualized language models (e.g., Rosenfeld and Erk, 2018; Hu et al.,
2019; Giulianelli et al., 2020). A common assumption in these studies is that
linguistic usages (from historical corpora) are sufficient to capture
diachronic word meanings. However, previous work has suggested that text-
derived distributed representations tend to miss important aspects of word
meaning, including perceptual features Andrews et al. (2009); Baroni and Lenci
(2008); Baroni et al. (2010) and relational information Necşulescu et al.
(2015). It has also been shown that both relational and perceptual knowledge
are essential to construct creative or figurative language use such as
metaphor Gibbs Jr. et al. (2004); Gentner and Bowdle (2008) and metonymy
Radden and Kövecses (1999). Our work examines the function of multimodal
semantic representations in capturing diachronic verb-noun compositions, and
the extent to which such representations can be integrated with the cognitive
mechanisms of chaining.

###  2.3 Multimodal representation of meaning

Computational research has shown the effectiveness of grounding language
learning and distributional semantic models in multimodal knowledge beyond
linguistic knowledge Lazaridou et al. (2015); Hermann et al. (2017). For
instance, Kiros et al. (2014) proposed a pipeline that combines image-text
embedding models with LSTM neural language models. Bruni et al. (2014)
identifies discrete “visual words” in images, so that the distributional
representation of a word can be extended to encompass its co-occurrence with
the visual words of images it is associated with. Gella et al. (2017) also
showed how visual and multimodal information help to disambiguate verb
meanings. Our framework extends these studies by incorporating the dimension
of time into exploring how multimodal knowledge predicts novel language use.

###  2.4 Memory-augmented deep learning

Our framework also builds upon recent work on memory-augmented deep learning
Vinyals et al. (2016); Snell et al. (2017). In particular, it has been shown
that category representations enriched by deep neural networks can effectively
generalize to few-shot predictions with sparse input, hence yielding human-
like abilities in classifying visual and textual data Pahde et al. (2020);
Singh et al. (2020); Holla et al. (2020). In our work, we consider the
scenario of constructing novel compositions as they emerge over time, where
sparse linguistic information is available. We therefore extend the existing
line of research to investigate how representations learned from naturalistic
stimuli (e.g., images) and structured knowledge (e.g., knowledge graphs) can
reliably model the emergence of flexible language use that expresses new
knowledge and experience.

Figure 2: Illustration of the syntactic frame extension model for the emerging
query car. The model integrates information from visual perception and
conceptual knowledge about cars to form a multimodal embedding
($h_{n}^{(t)}$), which supports temporal prediction of appropriate verb-syntax
usages via deep categorization models of chaining.

##  3 Computational framework

We present the syntactic frame extension model (SFEM), which is composed of
two components. First, SFEM specifies a frame as a joint probabilistic
distribution over a verb, its noun arguments, and their syntactic relations
and supports temporal prediction of verb syntax and arguments via deep
probabilistic models of categorization. Second, SFEM draws on multimodal
knowledge by incorporating perceptual, conceptual, and linguistic cues into
flexible inference for extended verb frames over time. Figure 2 illustrates
our framework.

###  3.1 Chaining as probabilistic categorization

We denote a predicate verb as $v$ (e.g., drive) and a syntactic relation as
$r$ (e.g., direct object of a verb), and consider a finite set of verb-
syntactic frame elements $f=(v,r)\in\mathcal{F}$. We define the set of nouns
that appeared as arguments for a verb (under historically attested syntactic
relations) up to time $t$ as support nouns, denoted by $n_{s}\in S(f)^{(t)}$
(e.g., horse appeared as a support noun—the direct object—for the verb drive
prior to 1880s). Given a query noun $n^{*}$ (e.g., car upon its emergence in
1880s) that has never been an argument of $v$ under relation $r$, we define
syntactic frame extension as probabilistic inference in two related problems:

  * 1.

Verb syntax prediction. Here we predict which verb-syntactic frames $f$ are
appropriate to describe the query noun $n^{*}$, operationalized as
$p(f|n^{*})$ yet to be specified.

  * 2.

Noun argument prediction. Here we predict which nouns $n^{*}$ are plausible
novel arguments for a given verb-syntax frame $f$, operationalized as
$p(n^{*}|f)$ yet to be specified.

We solve these inference problems by modeling the joint probability
$p(n^{*},f)$ for a query noun and a candidate verb-syntactic frame
incrementally through time as follows:

$\displaystyle p(n^{*},f)^{(t)}$ $\displaystyle=p(n^{*}|f)^{(t)}p(f)^{(t)}$
(1) $\displaystyle=p(n^{*}|S(f)^{(t)})p(f)^{(t)}$ (2)

Here we construct verb meaning based on its existing support nouns
$S(f)^{(t)}$ at current time $t$. We infer the most probable verb-syntax
usages for describing the query noun (Problem 1) as follows:

$\displaystyle p(f|n^{*})$
$\displaystyle=\frac{p(n^{*},f)^{(t)}}{\sum_{f\in\mathcal{F}}p(n^{*},f)^{(t)}}$
(3)
$\displaystyle=\frac{p(n^{*}|S(f)^{(t)})p(f)^{(t)}}{\sum_{f^{\prime}\in\mathcal{F}}p(n^{*}|S(f^{\prime})^{(t)})p(f^{\prime})^{(t)}}$
(4)

In the learning phase, we train our model incrementally at each time period
$t$ by minimizing the log joint probability $p(n^{*},f)^{(t)}$ in Equation 1
for every frame $f$ and each of its query noun $n^{*}\in Q(f)^{(t)}$:

$\displaystyle J=-\sum\limits_{f\in\mathcal{F}^{(t)}}\sum\limits_{n^{*}\in
Q(f)^{(t)}}\log p(n^{*},f)^{(t)}$ (5)

For each noun $n$, we consider a time-dependent hidden representation
$\textbf{h}_{n}^{(t)}\in\mathbb{R}^{M}$ derived from different sources of
knowledge (specified in Section 3.2). For the prior probability $p(f)^{(t)}$,
we consider a frequency-based approach that computes the proportion for the
number of unique noun arguments that a (verb) frame has been paired with and
attested in a historical corpus:

$\displaystyle
p(f)^{(t)}=\frac{|S(f)^{(t)}|}{\sum_{f^{\prime}\in\mathcal{F}}|S(f^{\prime})^{(t)}|}$
(6)

We formalize $p(n^{*}|f)$ (Problem 2), namely $p(n^{*}|N(f))^{(t)}$, by two
classes of deep categorization models motivated by the literature on chaining,
categorization, and memory-augmented learning.

Deep prototype model (SFEM-DPM). SFEM-DPM draws inspirations from prototypical
network for few-shot learning Snell et al. (2017) and is grounded in the
prototype theory of categorization in cognitive psychology Rosch (1975). The
model computes a set of hidden representations for every support noun
$n_{s}\in S(f)^{(t)}$, and takes the expected vector as a prototype
$\textbf{c}_{f}^{(t)}$ to represent $f$ at time $t$:

$\displaystyle\textbf{c}_{f}^{(t)}=\frac{1}{|S(f)^{(t)}|}\sum_{n_{s}\in
S(f)^{(t)}}\textbf{h}_{n_{s}}^{(t)}$ (7)

The likelihood of extending $n^{*}$ to $S(f)^{(t)}$ is then defined as a
softmax distribution over $l_{2}$ distances $d(\cdot,\cdot)$ to the embedded
prototype:

$\displaystyle
p(n^{*}|S(f)^{(t)})=\frac{\exp{(-d(\textbf{h}_{n^{*}}^{(t)},\textbf{c}_{f}^{(t)}))}}{\sum_{f^{\prime}}\exp(-d(\textbf{h}_{n^{*}}^{(t)},\textbf{c}_{f}^{(t)}))}$
(8)

Deep exemplar model (SFEM-DEM). In contrast to the prototype model, SFEM-DEM
resembles the memory-augmented matching network in deep learning Vinyals et
al. (2016), and formalizes the exemplar theory of categorization Nosofsky
(1986) and chaining-based category growth Habibi et al. (2020). Unlike DPM,
this model depends on the $l_{2}$ distances between $n^{*}$ and every support
noun:

(9)

Decade | Verb syntactic frame | Support noun | Query noun
---|---|---|---
Predicate verb | Syntactic relation
1900 | drive | direct object | horse, wheel, cart | car, van
1950 | work | prepositional object via as | mechanic, carpenter, scientist | astronaut, programmer
1980 | store | prepositional object via in | fridge, container, box | supercomputer

Table 1: Sample entries from Google Syntactic Ngram including verb syntactic
frames, support and query nouns.

###  3.2 Multimodal knowledge integration

In addition to the probabilistic formulation, SFEM draws on structured
knowledge including perceptual, conceptual, and linguistic cues to construct
multimodal semantic representations $\textbf{h}_{n}^{(t)}$ introduced in
Section 3.1.

Perceptual knowledge. We capture perceptual knowledge from image
representations in the large, taxonomically organized ImageNet database Deng
et al. (2009). For each noun $n$, we randomly sample a collection of 64 images
from the union of all ImageNet synsets that contains $n$, and encode the
images through the VGG-19 convolutional neural network Simonyan and Zisserman
(2015) by extracting the output vector from the last fully connected layer
after all convolutions (see similar procedures also in Pinto Jr. and Xu,
2021). We then average the encoded images to a mean vector
$\textbf{x}_{p}(n)\in\mathbb{R}^{1000}$ as the perceptual representation of
$n$.

Conceptual knowledge. To capture conceptual knowledge beyond perceptual
information (e.g., attributes and functions), we extract information from the
ConceptNet knowledge graph Speer et al. (2017), which connects concepts in a
network structure via different types of relations as edges. This graph
reflects commonsense knowledge of a concept (noun) such as its functional role
(e.g., a car IS_USED_FOR transportation), taxonomic information (e.g., a car
IS_A vehicle), or attributes (e.g., a car HAS_A wheel). Since the concepts and
their relations may change over time, we prepare a diachronic slice of the
ConceptNet graph at each time $t$ by removing all words with frequency up to
$t$ in a reference historical text corpus (see Section 4 for details) under a
threshold $k_{\text{c}}$ which we set to be 10. We then compute embeddings for
the remaining concepts following methods recommended in the original study by
Speer et al. (2017). In particular, we perform singular value decomposition
(SVD) on the positive pointwise mutual information matrix
$\textbf{M}_{G}^{(t)}$ of the ConceptNet $G^{(t)}$ truncated at time $t$, and
combine the top 300 dimensions (with largest singular values) of the term and
context matrix symmetrically into a concept embedding matrix. Each row of the
resulting row matrix of SVD will therefore serves as the conceptual embedding
$\textbf{x}_{c}(n)^{(t)}\in\mathbb{R}^{300}$ for its corresponding noun.

Linguistic knowledge. For linguistic knowledge, we take the HistWords
diachronic word embeddings $\textbf{x}_{l}^{(t)}\in\mathbb{R}^{300}$ pre-
trained on the Google N-Grams English corpora to represent linguistic meaning
of each noun at decade $t$ Hamilton et al. (2016).

Knowledge integration. To construct a unified representation that incorporates
knowledge from different modalities, we take the mean of the unimodal
representations described into a joint vector
$\textbf{x}_{n}\in\mathbb{R}^{300}$, and then apply an integration function
$g:\mathbb{R}^{300}\rightarrow\mathbb{R}^{M}$ parameterized by a feedforward
neural network to get the multimodal word representation $h_{n}^{(t)}$.111For
ImageNet embeddings, we apply a linear transformation to project each
$\textbf{x}_{p}^{(t)}$ into $\mathbb{R}^{300}$ so that all unimodal
representations are 300-d vectors before taking the means. Our framework
allows flexible combinations of the three modalities introduced, e.g., a full
model would utilizes all three types of knowledge, while a linguistic-only
baseline will directly take HistWords embeddings $\textbf{x}_{l}^{(t)}$ as
inputs of the integration network.

##  4 Historical noun-verb compositions

To evaluate our framework, we collected a large dataset of historical noun-
verb compositions derived from the Google Syntactic N-grams (GSN) English
corpus Lin et al. (2012) from 1850 to 2000. Specifically, we collected verb-
noun-relation triples $(n,v,r)^{(t)}$ that co-occur in the ENGALL subcorpus of
GSN over the 150 years. We focused on working with common usages and pruned
rare cases under the following criteria: 1) a noun $n$ should have at least
$\theta_{p}=64$ image representations in ImageNet, $\theta_{c}=10$ edges in
the contemporary ConceptNet network, and $\theta_{n}=15,000$ counts (with POS
tag as nouns) in GSN over the 150-year period; 2) a verb $v$ should have at
least $\theta_{v}=15,000$ counts in GSN. To facilitate feasible model
learning, we consider the top-20 most common syntactic relations in GSN,
including direct object, direct subject, and relations concerning
prepositional objects.

We binned the raw co-occurrence counts by decade $\Delta=10$. At each decade,
we define emerging query nouns $n^{*}$ for a given verb frame $f$ if their
number of co-occurrences with $f$ up to time $t$ falls below a threshold
$\theta_{q}$, while the number of co-occurrences with $f$ up to time
$t+\Delta$ is above $\theta_{q}$ (i.e., an emergent use that
conventionalizes). We define support nouns as those that co-occurred with $f$
for more than $\theta_{s}$ times before $t$. We found that $\theta_{q}=10$ and
$\theta_{s}=100$ are reasonable choices. This preprocessing pipeline yielded a
total of 10,349 verb-syntactic frames over 15 decades, where each frame class
has at least 1 novel query noun and 4 existing support nouns. Table 1 shows
sample entries of data which we make publicly available.222Data and code are
deposited here: https://github.com/jadeleiyu/frame_extension

##  5 Evaluation and results

We first describe the details of SFEM implementation and diachronic
evaluation. We then provide an in-depth analysis on the multimodal knowledge
and chaining mechanisms in verb frame extension.

###  5.1 Details of model implementation

We implemented the integration network $g(\cdot)$ of SFEM as a three-layer
feedforward neural network with an output dimension $M=100$, and keep
parameters and embeddings in other modules fixed during learning.333See
Appendix A for additional implementation details. At each decade, we randomly
sample $70\%$ of the query nouns with their associated verb-syntactic pairs as
training data, and take the remaining examples for model testing such that
there is no overlap in the query nouns between training and testing. We
trained models on the negative log-likelihood loss defined in Equation 5 at
each decade. To examine how multimodal knowledge contributes to temporal
prediction of novel language use, we trained 5 DEM and 5 DPM models using
information from different modalities.

Figure 3: Area-under-curves of SFEM and baseline models from 1850s to 1990s.
Top row: AUCs of predicting extended syntax frames for query nouns. Bottom
row: AUCs of predicting extended nouns for query verb frames.

Model | AUC – verb syntax prediction | AUC – noun argument prediction
---|---|---
novel items | existing items | combined | novel items | existing items | combined
DPM (linguistics) | 0.642 | 0.690 | 0.681 | 0.641 | 0.653 | 0.650
DPM (perceptual) | 0.632 | 0.666 | 0.657 | 0.650 | 0.624 | 0.629
DPM (conceptual) | 0.772 | 0.722 | 0.733 | 0.727 | 0.705 | 0.711
DPM (perceptual+conceptual) | 0.809 | 0.754 | 0.767 | 0.725 | 0.719 | 0.721
DPM (perceptual+linguistics) | 0.645 | 0.669 | 0.661 | 0.655 | 0.669 | 0.665
DPM (conceptual+linguistics) | 0.753 | 0.774 | 0.766 | 0.776 | 0.768 | 0.770
DPM (perceptual+conceptual+linguistics) | 0.848 | 0.810 | 0.815 | 0.799 | 0.786 | 0.788
DEM (linguistics) | 0.652 | 0.690 | 0.686 | 0.641 | 0.625 | 0.632
DEM (perceptual) | 0.737 | 0.674 | 0.684 | 0.659 | 0.650 | 0.655
DEM (conceptual) | 0.854 | 0.784 | 0.788 | 0.736 | 0.724 | 0.729
DEM (perceptual+conceptual) | 0.858 | 0.792 | 0.797 | 0.750 | 0.744 | 0.748
DEM (perceptual+linguistics) | 0.712 | 0.759 | 0.753 | 0.698 | 0.710 | 0.708
DEM (conceptual+linguistics) | 0.902 | 0.866 | 0.870 | 0.837 | 0.822 | 0.824
DEM (perceptual+conceptual+linguistics) | 0.919 | 0.872 | 0.878 | 0.856 | 0.820 | 0.827
Baseline (frequency) | 0.573 | 0.573 | 0.573 | 0.536 | 0.536 | 0.536
Baseline (random) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500

Table 2: Mean model AUC scores of verb syntax and noun argument predictions
from 1850s to 1990s.

Query noun | Decade | Predicted frames (linguistic-only DEM) | Predicted frames (tri-modal DEM)
---|---|---|---
telephone | 1860 |  | roll-nsubj, load-dobj,
---
play-pobj_prep.on
| purchase_dobj, pick-dobj,
---
remain-pobj_prep.on
microorganism | 1900 |  | decorate-pobj_prep.with,
---
play-pobj_prep.on, spread-dobj
| feed-pobj_prep.on,
---
mix-pobj_prep.with, breed-dobj
airplane | 1930 |  | load-dobj, mount-pobj_prep.on,
---
mount-dobj, blow-dobj, roll-nsubj
| fly-dobj, approach-nsubj,
---
drive-dobj, stop-nsubj
astronaut | 1950 |  | spin-dobj,work-pobj_prep.in,
---
emerge-pobj_prep.from
| work-pobj_prep.as, talk-pobj_prep.to,
---
lead-pobj_prep.by
computer | 1970 |  | purchase-dobj, fix-dobj,
---
generate-dobj, write-pobj_prep.to
| store-pobj_prep.in, move-pobj_prep.into,
---
display-pobj_prep.on, implement-pobj_prep.in

Table 3: Example predictions of novel verb-noun compositions from the full
tri-imodal and linguistic-only models.

Ablated modality | Most affected compositions
---|---
Language |  | buy a monitor, find a disk (*),
---
a resident dies,
specialized in nutrition (*),
point to the window
Percept |  | an airplane rolls,
---
talk to an entrepreneur (*),
the tree stands, the doctor says,
topped with nutella (*)
Concept |  | perform in the film (*),
---
work as a programmer (*),
work for the newspaper,
expand the market, kill the process

Table 4: Top-4 ground-truth compositions with most prominent drops in joint
probability $p(f,n)$ after ablation of one modality of knowledge from SFEM.
Phrases marked with ‘*’ include novel query nouns.

###  5.2 Evaluation against historical data

We test our models on both verb syntax and noun argument predictive tasks with
the goals of assessing 1) the contributions of multimodal knowledge, and 2)
the two alternative mechanisms of chaining. We also consider baseline models
that do not implement chaining-based mechanisms: a frequency baseline that
predicts by count in GSN up to time $t$, and a random guesser. We evaluate
model performance via standard receiver operating characteristics (ROC) curves
that reveal cumulative precision of models in their top $m$ predictions. We
compute the standard area-under-curve (AUC) statistics for the ROC curves to
get the mean precision over all values of $m$ from 1 to the candidate set
size. Figure 3 summarizes the results over the 150 years. We observe that 1)
all multimodal models perform better than their uni-/bi-modal counterparts,
and 2) the exemplar-based model performs dominantly better than the prototype-
based counterpart, and both outperform the baseline models without chaining.
In particular, a tri-modal deep exemplar model that incorporates knowledge
from all three modalities achieves the best overall performance. These results
provide strong support that verb frame extension depends on multimodal
knowledge and an exemplar-based chaining.

To further assess how the models perform in predicting emerging verb extension
toward both novel and existing nouns, we report separate mean AUC scores for
these predictive cases where query nouns are either completely novel (i.e.,
zero token frequencies) or established. (i.e., above-zero frequencies) at the
time of prediction. Table 2 summarizes these results and shows that model
performances are similar under four predictive cases. For prediction with
novel query nouns, it is not surprising that linguistic-only models fail due
to the unavailability of linguistic mentions. However, for prediction with
established query nouns, the superiority of multimodal SFEMs is still
prominent suggesting that our framework captures general principles in verb
frame extension (and not just for predicting verb extension toward novel
nouns).

Table 3 compares sample verb syntax predictions made by the full and
linguistic-only DEM models that cover a diverse range of concepts including
inventions (e.g., airplane), discoveries (e.g., microorganism), and
occupations (e.g. astronaut). We observe that the full model typically
constructs reasonable predicate verbs that reflect salient features of the
query noun (e.g., cars are vehicles that are drive-able). In contrast, the
linguistic-only model often predicts verbs that are either overly generic
(e.g., purchase a telephone) or nonsensical.

###  5.3 Model analysis and interpretation

We provide further analyses and interpret why both multimodality and chaining
mechanisms are fundamental to predicting emergent verb compositions.

####  5.3.1 The function of multimodal knowledge

To understand the function of multimodal knowledge, we compute, for each
modality, the top-4 verb compositions that were most degraded in joint
probability $p(f,n)$ after ablating a knowledge modality from the full tri-
modal SFEM (see Table 4). A drop in $p(f,n)$ indicates reliance on
multimodality in prediction. We found that linguistic knowledge helps the
model identify some general properties that are absent in the other cues
(e.g., a monitor is buy-able). Importantly, for the two extra-linguistic
knowledge modalities, we observe that visual-perceptual knowledge helps
predict many imaged-based metaphors, including “the airplane rolls" (i.e.,
based on common shapes of airplanes) and “the tree stands" (based on
verticality of trees). On the other hand, conceptual knowledge predicts cases
of logical metonymy (e.g., “work for the newspaper”) and conceptual metaphor
(e.g., “kill the process”). These examples suggest that multimodality serves
to ground and embody SFEM with commonsense knowledge that constructs novel
verb compositions for not only literal language use, but also non-literal or
figurative language use that is extensively discussed in the psycholinguistics
literature Lakoff (1982); Radden and Kövecses (1999); Gibbs Jr. et al. (2004).

We also evaluate the contributions of the three modalities in model prediction
by comparing the AUC scores from the three uni-modal DEMs. Figure 4 shows the
percentage breakdown of examples on which one of the modalities yields the
highest score (i.e., contributes most to a reliable prediction). We observe
that conceptual cues explain data the best in almost 2/3 of the cases,
followed by perceptual and linguistic cues. These results suggest that while
conceptual knowledge plays a dominant role in model prediction, all three
modalities contain complementary information in predicting novel language use
through time.

Figure 4: Percentage breakdown of the three modalities in model prediction,
with annotated examples.

####  5.3.2 General mechanisms of chaining

Figure 5: Illustrations of two mechanisms of chaining (left: exemplar; right:
prototype) in verb frame prediction for query supercomputer. Nouns are PCA-
projected in 2D, with categories color-coded and in dashed boundaries.

We next analyze general mechanisms of chaining by focusing on understanding
the superiority of exemplar-based chaining in SFEM. Figure 5 illustrates the
exemplar-based and prototype-based processes of chaining with the example verb
frame prediction for the noun “supercomputer". For simplicity, we only show
two competing frames “to store in a ___” and “to wear a ___”. In this case,
the query noun is semantically distant to most of the prototypical support
nouns in both categories, and is slightly closer to the centroid of the “wear”
class than to that of the “store” class. The prototype model would then
predict the incorrect composition “to wear a supercomputer”. In contrast, the
exemplar model is more sensitive to the semantic neighborhood profile of the
query noun and the aprototypical support noun “computer” of the “store in ___”
class, and it therefore correctly predicts that “supercomputer” is more likely
to be predicated by “to store in”. Our discovery that the exemplar-based
chaining accounts for verb composition through time mirrors existing findings
on similar mechanisms of chaining in the extensions of numeral classifiers
Habibi et al. (2020) and adjectives Grewal and Xu (2020), and together they
suggest a general cognitive mechanism may underlie historical linguistic
innovation.

##  6 Conclusion

We have presented a probabilistic framework for characterizing the process of
syntactic frame extension in which verbs extend their referential range toward
novel and existing nouns over time. Our results suggest that language users
rely on extra-linguistic knowledge from percept and concept to construct new
linguistic compositions via a process of exemplar-based chaining. Our work
creates a novel approach to diachronic compositionality and strengthens the
link between multimodal semantics and cognitive linguistic theories of
categorization.

## Acknowledgments

We would like to thank Graeme Hirst, Michael Hahn, and Suzanne Stevenson for
their feedback on the manuscript, and members of the Cognitive Lexicon
Laboratory at the University of Toronto for helpful suggestions. We also thank
the anonymous reviewers for their constructive comments. This work was
supported by a NSERC Discovery Grant RGPIN-2018-05872, a SSHRC Insight Grant
#435190272, and an Ontario ERA Award to YX.

## References

  * Andrews et al. (2009) Mark Andrews, Gabriella Vigliocco, and David Vinson. 2009.  Integrating experiential and distributional data to learn semantic representations.  _Psychological review_ , 116(3):463.
  * Baroni (2020) Marco Baroni. 2020.  Linguistic generalization and compositionality in modern artificial neural networks.  _Philosophical Transactions of the Royal Society B_ , 375(1791):20190307.
  * Baroni and Lenci (2008) Marco Baroni and Alessandro Lenci. 2008.  Concepts and properties in word spaces.  _Italian Journal of Linguistics_ , 20(1):55–88.
  * Baroni et al. (2010) Marco Baroni, Brian Murphy, Eduard Barbu, and Massimo Poesio. 2010.  Strudel: A distributional semantic model based on property and types.  _Cognitive Science_ , 34(2):222–254.
  * Baroni and Zamparelli (2010) Marco Baroni and Roberto Zamparelli. 2010.  Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space.  In _Proceedings of the 2010 conference on empirical methods in natural language processing_ , pages 1183–1193.
  * Blacoe and Lapata (2012) William Blacoe and Mirella Lapata. 2012.  A comparison of vector-based representations for semantic composition.  In _Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning_ , pages 546–556.
  * Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012.  Distributional semantics in technicolor.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 136–145.
  * Bruni et al. (2014) Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014.  Multimodal distributional semantics.  _Journal of Artificial Intelligence Research_ , 49:1–47.
  * Cordeiro et al. (2016) Silvio Cordeiro, Carlos Ramisch, Marco Idiart, and Aline Villavicencio. 2016.  Predicting the compositionality of nominal compounds: Giving word embeddings a hard time.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1986–1997.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255.
  * Fillmore (1986) Charles J Fillmore. 1986.  Frames and the semantics of understanding.  _Quaderni di Semantica_ , 6:222–254.
  * Frermann and Lapata (2016) Lea Frermann and Mirella Lapata. 2016.  A Bayesian model of diachronic meaning change.  _Transactions of the Association for Computational Linguistics_ , 4:31–45.
  * Gella et al. (2017) Spandana Gella, Frank Keller, and Mirella Lapata. 2017.  Disambiguating visual verbs.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(2):311–322.
  * Gella et al. (2016) Spandana Gella, Mirella Lapata, and Frank Keller. 2016.  Unsupervised visual sense disambiguation for verbs using multimodal embeddings.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 182–192, San Diego, California. Association for Computational Linguistics.
  * Gentner and Bowdle (2008) Dedre Gentner and Brian Bowdle. 2008.  Metaphor as structure-mapping.  _The Cambridge handbook of metaphor and thought_ , 109:128.
  * Gibbs Jr. et al. (2004) Raymond W Gibbs Jr., Paula Lenz Costa Lima, and Edson Francozo. 2004.  Metaphor is grounded in embodied experience.  _Journal of Pragmatics_ , 36(7):1189–1210.
  * Giulianelli et al. (2020) Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020.  Analysing lexical semantic change with contextualised word representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3960–3973.
  * Grewal and Xu (2020) Karan Grewal and Yang Xu. 2020.  Chaining and historical adjective extension.  In _Proceedings of the 42nd Annual Conference of the Cognitive Science Society_.
  * Habibi et al. (2020) Amir Ahmad Habibi, Charles Kemp, and Yang Xu. 2020.  Chaining and the growth of linguistic categories.  _Cognition_ , 202:104323.
  * Hamilton et al. (2016) William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016.  Diachronic word embeddings reveal statistical laws of semantic change.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1489–1501.
  * Hermann et al. (2017) Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. 2017.  Grounded language learning in a simulated 3d world.  _arXiv preprint arXiv:1706.06551_.
  * Holla et al. (2020) Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020.  Learning to learn to disambiguate: Meta-learning for few-shot word sense disambiguation.  _arXiv preprint arXiv:2004.14355_.
  * Hu et al. (2019) Renfen Hu, Shen Li, and Shichen Liang. 2019.  Diachronic sense modeling with deep contextualized word embeddings: An ecological view.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3899–3908.
  * Kiros et al. (2014) Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014.  Multimodal neural language models.  In _Proceedings of the 31st International Conference on Machine Learning_ , pages 595–603.
  * Lakoff (1982) George Lakoff. 1982.  Experiential factors in linguistics.  _Language, mind, and brain_ , pages 142–157.
  * Lakoff (1987) George Lakoff. 1987.  _Women, Fire, and Dangerous Things: What Categories Reveal about the Mind_.  University of Chicago Press.
  * Lazaridou et al. (2015) Angeliki Lazaridou, Marco Baroni, et al. 2015.  Combining language and vision with a multimodal skip-gram model.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 153–163.
  * Lin et al. (2012) Yuri Lin, Jean-Baptiste Michel, Erez Aiden Lieberman, Jon Orwant, Will Brockman, and Slav Petrov. 2012.  Syntactic annotations for the google books ngram corpus.  In _Proceedings of the ACL 2012 System Demonstrations_ , pages 169–174.
  * Mahowald et al. (2020) Kyle Mahowald, George Kachergis, and Michael C Frank. 2020.  What counts as an exemplar model, anyway? a commentary on ambridge (2020).  _First Language_ , 40(5-6):608–611.
  * Malt et al. (1999) Barbara C Malt, Steven A Sloman, Silvia Gennari, Meiyi Shi, and Yuan Wang. 1999\.  Knowing versus naming: Similarity and the linguistic categorization of artifacts.  _Journal of Memory and Language_ , 40(2):230–262.
  * McClelland (2020) James L McClelland. 2020.  Exemplar models are useful and deep neural networks overcome their limitations: A commentary on ambridge (2020).  _First Language_ , 40(5-6):612–615.
  * Mitchell and Lapata (2010) Jeff Mitchell and Mirella Lapata. 2010.  Composition in distributional models of semantics.  _Cognitive Science_ , 34(8):1388–1429.
  * Necşulescu et al. (2015) Silvia Necşulescu, Sara Mendes, David Jurgens, Núria Bel, and Roberto Navigli. 2015.  Reading between the lines: Overcoming data sparsity for accurate classification of lexical relationships.  In _Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics_ , pages 182–192.
  * Nosofsky (1986) Robert M Nosofsky. 1986.  Attention, similarity, and the identification–categorization relationship.  _Journal of Experimental Psychology: General_ , 115(1):39.
  * Pahde et al. (2020) Frederik Pahde, Mihai Puscas, Tassilo Klein, and Moin Nabi. 2020.  Multimodal prototypical networks for few-shot learning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2644–2653.
  * Pinto Jr. and Xu (2021) Renato Ferreira Pinto Jr. and Yang Xu. 2021.  A computational theory of child overextension.  _Cognition_ , 206:104472.
  * Radden and Kövecses (1999) Günter Radden and Zoltán Kövecses. 1999.  Towards a theory of metonymy.  _Metonymy in language and thought_ , 4:17–60.
  * Ramiro et al. (2018) Christian Ramiro, Mahesh Srinivasan, Barbara C Malt, and Yang Xu. 2018.  Algorithms in the historical emergence of word senses.  _Proceedings of the National Academy of Sciences_ , 115(10):2323–2328.
  * Reed (1972) Stephen K Reed. 1972.  Pattern recognition and categorization.  _Cognitive Psychology_ , 3(3):382–407.
  * Rosch (1975) Eleanor Rosch. 1975.  Cognitive representations of semantic categories.  _Journal of Experimental Psychology: General_ , 104(3):192.
  * Rosenfeld and Erk (2018) Alex Rosenfeld and Katrin Erk. 2018.  Deep neural models of semantic shift.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 474–484.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Singh et al. (2020) Pulkit Singh, Joshua C Peterson, Ruairidh M Battleday, and Thomas L Griffiths. 2020\.  End-to-end deep prototype and exemplar models for predicting human behavior.  In _Proceedings of the 42nd Annual Conference of the Cognitive Science Society (CogSci 2020)_.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017.  Prototypical networks for few-shot learning.  In _Advances in Neural Information Processing Systems_ , pages 4077–4087.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Vecchi et al. (2017) Eva M Vecchi, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2017.  Spicy adjectives and nominal donkeys: Capturing semantic deviance using compositionality in distributional spaces.  _Cognitive Science_ , 41(1):102–136.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016.  Matching networks for one shot learning.  _Advances in Neural Information Processing Systems_ , 29:3630–3638.
  * Xu et al. (2016) Yang Xu, Terry Regier, and Barbara C Malt. 2016.  Historical semantic chaining and efficient communication: The case of container names.  _Cognitive Science_ , 40(8):2081–2094.

##  Appendix A Additional details of SFEM implementation

We implemented the integration network $g(\cdot)$ as a three-layer feedforward
neural network using PyTorch, where each layer has a dimension of $300,200$
and $100$ respectively. For models that incorporates less than three
modalities, we replace the missing embeddings with a zero vector when
computing the mean vectors before knowledge integration.

During training, except for network weights in $g(\cdot)$, we keep parameters
in every modules (i.e., the VGG-19 encoder and every unimodal embedding)
constant, and optimize SFEM by minimzing the negative log-likelihood loss
function specified in Equation 5 via stochastic gradient descent (SGD).

Each training batch consists of $B=64$ syntactic frames with their associated
query and support nouns. We train each model for 200 epochs and save the
configuration that achieves the highest validation accuracy for our evaluation
described in Section 5.