# NEURAL EMBEDDING COMPRESSION FOR EFFICIENT MULTI-TASK EARTH OBSERVATION
MODELLING

###### Abstract

As repositories of large scale data in earth observation (EO) have grown, so
have transfer and storage costs for model training and inference, expending
significant resources. We introduce Neural Embedding Compression (NEC), based
on the transfer of compressed embeddings to data consumers instead of raw
data. We adapt foundation models (FM) through learned neural compression to
generate multi-task embeddings while navigating the tradeoff between
compression rate and embedding utility. We update only a small fraction of the
FM parameters ($\sim$$10\%$) for a short training period ($\sim$$1\%$ of the
iterations of pre-training). We evaluate NEC on two EO tasks: scene
classification and semantic segmentation. Compared with applying traditional
compression to the raw data, NEC achieves similar accuracy with a $75\%$ to
$90\%$ reduction in data. Even at $99.7\%$ compression, performance drops by
only $5\%$ on the scene classification task. Overall, NEC is a data-efficient
yet performant approach for multi-task EO modelling.

Index Terms—  Earth Observation, Neural Compression, Foundation Models,
Embeddings, Computer Vision

##  1 INTRODUCTION

Fig. 1: Downstream comparison: a) Raw Data Compression (RDC); b) Uniformly
Quantized Embeddings (UQE); c) Neural Embedding Compression (NEC). ‘Learned’
refers to entropy coding with the distribution learned during training.

Earth observation (EO) repositories comprise some of the largest data stores
globally. Due to their widespread use, these repositories experience extremely
large volumes of data transfers. For example, users of the Sentinel Data
Access System downloaded 78.6 PiB of data in 2022 alone [1]. The transfer of
such data volumes between data producers and consumers (Fig. 1 (a)) causes
substantial latency and requires significant amounts of energy and vast
storage capacities.

Due to its scale, manual processing of the incoming EO data is not an option,
leading to the adoption of machine learning methods, including deep learning.
Recently, several EO foundation models (EO-FM) based on self-supervised
learning (SSL), pre-trained on EO data sampled across the planet, have been
introduced [2, 3, 4, 5, 6, 7]. The general-purpose embeddings they produce
have been demonstrated as suitable for training task-specific heads for
multiple downstream tasks (e.g. scene classification, semantic segmentation).
The approach requires fewer annotated samples to train the task-specific model
and achieves superior performance and generalization to unseen data. On the
other hand, FMs consist of hundreds of millions or even billions of parameters
[5, 8] and thus are computationally demanding during pre-training and at
inference time.

To overcome these issues, we propose the exchange of EO-FM embeddings computed
at the data provider instead of the raw data. It follows that novel data only
has to be processed once by the computationally expensive encoder The
resulting embeddings can be stored and reused for multiple downstream
applications. The inference can be performed at the data consumer with a
lightweight head.

One of the main challenges is now the efficient exchange of embeddings, which
we address in this work. This is relevant as FMs do not have compression as a
goal, often producing embeddings that are even larger than the original data.
When transferring EO data, lossless compression is usually applied, as in Fig.
1 (a), resulting in a compressed representation of the data from which it can
be reconstructed with no information loss. We refer to this method as Raw Data
Compression (RDC).

One could consider applying lossless compression to the embeddings as well.
For higher compression rates, lossy compression methods could be employed at
the cost of some distortion. A suitable metric for this distortion is critical
in elaborating a lossy compression method. To minimize the distortion,
different codecs (e.g., JPEG [9] for images, MP3 [10] for audio) exist to
compress data in different domains. Each of these minimizes a domain-specific
metric for distortion, usually some proxy for human perception. Currently, no
such codec exists in the domain of embeddings for downstream tasks.

As a simple approach, depicted in Fig. 1 (b), we could consider quantizing the
embeddings to different bit levels before entropy coding. We refer to this as
Uniformly Quantized Embeddings (UQE).

In this study, we propose the application of learned neural compression, which
has been well studied in the domain of image and video compression [11, 12,
13, 14]. This data-driven approach allows us to learn an encoder that produces
compressible multi-task embeddings from data while minimizing an appropriate
distortion metric: the loss across downstream tasks. We call this approach
Neural Embedding Compression (NEC) and illustrate it in Fig. 1 (c).

Our contributions in this paper are threefold: i) The discussion on EO-FM
embedding sharing compared to raw data, ii) the conceptual integration of
neural compression into the data pipeline, and iii) the benchmarking of the
neural compression approach for various downstream applications.

##  2 RELATED WORK

Learned neural compression was pioneered by Balle  _et al._ [11] with their
seminal work on integrating information theoretical concepts into Deep Neural
Network training for data compression. A variety of works followed proposing
more powerful entropy models [12, 15] and extending its applications to video
compression [14, 16, 13] and even neural network compression [17]. Recently,
some works have leveraged masked auto-encoding (MAE) as a lossy compression
method, dropping data at the image patch [18] or embedding level [19].

CORSA [20] trains a VQVAE [21] for the purposes of compression for satellite
imagery. In their work, they briefly explore training an MLP directly on the
VQVAE embeddings for a simple binary classification task, but their method
does not show conclusive results. To the best of our knowledge, ours is the
first work that demonstrates the effectiveness of the direct use of compressed
embeddings in downstream tasks by combining FMs with neural compression. We
target EO applications as input data is often sourced through queries on large
repositories, whereas other application areas may rely on user input. This
characteristic makes it a good fit for NEC.

##  3 METHOD

Fig. 2: Adapting an FM encoder for NEC.

In this section, we frame our method as the optimization of a rate-distortion
objective following the framework of end-to-end learned compression
established in Balle _et al._ [11]. Traditionally, these methods aim to
transmit an input image or video using the smallest amount of bits possible (R
- Rate) while minimizing the loss in quality (D - Distortion) after
reconstruction by a decoder network. This trade-off is controlled by
$\lambda$, resulting in an optimization problem of the form $\min\lambda R+D$.

For NEC, we are not specifically interested in reconstructing the original
data. Instead, given an EO input, we aim to transmit the smallest amount of
bits possible (R) so that we can perform model training or inference for
various tasks, achieving the smallest loss (D) possible across them. In what
follows, we detail each component of our method, summarized in Fig. 2.

Distortion Term — SSL: Our distortion term is an aggregate loss over
downstream tasks which are a priori unknown. Therefore, we require a proxy for
this quantity. In this work, we leverage self-supervised learning for this
purpose, allowing for training on large unlabeled datasets to produce general-
purpose embeddings which have shown competitive performance in many downstream
tasks [22].

Specifically, we use MAE [23] with a standard vision transformer (ViT) [24]
architecture. MAE masks part of the input and tasks the model with recovering
it based on the unmasked input. This is achieved by minimizing the mean
squared error between the masked input and its reconstruction, which we take
as our distortion term. We choose MAE due to its simplicity and computational
efficiency and the ViT-B architecture due to its wide use in recent EO models
[5, 6, 3]. However, any loss that encourages generally useful embeddings for
EO and any compatible encoder architecture can be used.

Rate Term: Balle  _et al._ [11] establishes the general methodology for
computing the rate term in the full loss. We give a short overview and direct
the reader to [12, Appendix 6.1] for more details.

Given an embedding vector $y$, the minimum number of bits required to transmit
$y$ is given by Shannon’s source coding theorem [25] as $-\log_{2}p(y)$, where
$p$ defines the probability distribution over all embedding vectors. Our goal
is to minimize the expectation of this quantity over the distribution of all
embedding vectors $-\mathbb{E}\left[\log_{2}p(y)\right]$, which is precisely
the entropy of this distribution. We adapt the fully factorized entropy model
defined in Balle  _et al._ [12] in order to model the probability distribution
of these embeddings for MAE. Specifically, we leverage the implementation in
the CompressAI [26] library.

After patching, masking, and passing through the encoder, we obtain an
embedding $y$ of shape $e\times n$, where $e$ is the embedding dimension of
the encoder and $n$ depends on the input shape. We quantize $y$ by rounding to
the nearest integer, using uniform noise as a differentiable proxy during
training [12]. To model the distribution of $y$ we assume that all of its
elements are independent and additionally that all elements with the same
embedding dimension (the first dimension above) are identically distributed,
resulting in $p(y)=\prod_{e}\prod_{n}p_{e}(y_{e,n})$. Each $p_{e}$ is modeled
using a small neural network.

SSL Compression Loss: For an input $x$ for which the model produces the
(approximated) quantized embedding $y$ and reconstruction $x\prime$, the loss
is given as

$Loss(x,x\prime)=\lambda\cdot\text{MAE}(x,x\prime)-\log_{2}p(y)$

with $\lambda$ as the hyperparameter controlling the trade-off.

Efficient Model Adaptation: For training, we initialize our model with the
pre-trained weights from Wang  _et al._ [3], resulting from MAE training over
the MillionAid [27] dataset. We freeze all layers of this model except for the
patch embedding layers of the encoder and decoder, the final encoder layer,
and the first decoder layer, optimizing only $\sim$10% of the total
parameters.

Fine-tuning: For fine-tuning, the data consumer can train a model using the
embeddings as surrogates for the raw training data. The setup is illustrated
in Fig. 1 (c). We freeze the entire backbone, simulating the effect of the
data consumer having access only to the embeddings.

Additionally, we experiment with the consumer having access to the first
decoder layer (including the embedding layer). In this case, we unfreeze these
layers and apply them as the first step of processing the embeddings after
entropy decoding and before the task-specific head. The transfer of these
layers’ weights is a small one-time cost that would be paid a priori, as with
the probability models for the entropy codes [11].

##  4 EXPERIMENTS

For Neural Embedding Compression (NEC) we train on the MillionAid [27] dataset
for $20$ epochs, only $1.25\%$ of the 1600 epochs used for the original pre-
training. We use the same settings as Wang  _et al._ [3] except for a reduced
learning rate of $1.5\times 10^{-4}$ and 4 GPUs instead of 8, halving the
total batch size to 1024. Embeddings at different compression levels are
obtained by training models with different values of $\lambda$ (ranging from
$10^{7}$ to $10^{11}$) but with the same architecture. We benchmark NEC
against Raw Data Compression (RDC) and Uniformly Quantized Embeddings (UQE),
using these embeddings to train models for two downstream EO tasks: scene
classification and semantic segmentation.

In RDC, we compare against transferring the raw data, after entropy coding
using the popular Zstandard [28] algorithm, at 16 and 8-bit precision, and
fine-tuning the entire model, as shown in Fig. 1 (a). The 16-bit point is a
useful reference despite the fact both curated datasets use 8-bit precision
since many EO datasets (e.g., Sentinel 2) store data in 16-bit precision.

In UQE, we compare against transferring quantized embeddings produced by the
pre-trained ViT-B network from Wang  _et al._ [3], as shown in Fig. 1 (b).
These are quantized using affine uniform quantization and entropy coded using
Zstandard [28]. We show results for $32$ and $16$ bit float and $8$, $5$, $3$
and $2$ bit integer quantization.

Downstream Task Evaluation: For each task, we plot accuracy against the
average size of an embedding for a sample from the validation set of the
respective datasets.

1\. Scene Classification: We choose the UCMerced land use dataset [29] for our
scene classification task. It contains 21 classes, each with 100 images of
size $256\times 256$. We use a simple architecture comprising a pooling layer
followed by a single linear layer, discarding the [CLS] token. For the pooling
layer, we found the ‘AttentionPoolLatent’ layer, as implemented in the ‘timm’
[30] library, to perform substantially better than other methods, such as
average pooling. We train all models with a learning rate of $1.25\times
10^{-4}$ and a batch size of $32$ for 400 epochs using the AdamW [31]
optimizer with weight decay of $0.05$.

2\. Semantic Segmentation: The Potsdam [32] dataset is used for our semantic
segmentation task. This dataset contains large images of urban areas with
semantic segmentation labels for six classes. We process it as is standard
[33], obtaining tiles of size $512\times 512$. Unlike Wang  _et al._ [3], we
keep the ‘clutter’ class and train with labels where the borders of semantic
classes are ignored to avoid the ambiguity in labeling these. Additionally, we
restrict ourselves to the RGB channels to remain compatible with the
MillionAid pre-training. Most state-of-the-art models for semantic
segmentation [3, 33, 8] combine embeddings from different stages of the
encoder. While NEC can be extended to compress multiple levels of embeddings,
we limit our investigation to the single embedding case. We use a simple
decoder composed of a first convolutional layer to reduce the embedding
dimension by half, followed by six residual convolutional blocks. Then, we use
two PixelShuffle [34] blocks to upscale the result by a factor of four.
Finally, this output is passed on one branch to a main head with two
convolutional layers with $128$ channels and on another branch to an auxiliary
head with one convolutional layer with $64$ channels. The loss of the
auxiliary head is added to the loss of the main head with a multiplicative
factor of $0.4$.

##  5 Results

(a) Classification Accuracy on the UC Merced dataset

(b) Overall Accuracy on the Potsdam dataset

Fig. 3: Accuracy metric vs Average Bits per Sample for Scene Classification
and Semantic Segmentation Fig. 4: Semantic segmentation on the Potsdam
dataset. The mean size of the embedding produced by the method over the
validation set is shown in KB.

1\. Scene Classification: Fig. 3(a) plots the results for the UCM Dataset. For
the high-performance case, we observe that UQE performs well until $5$ bit
quantization and is only slightly outperformed by NEC. For higher compression
levels, this does not hold. NEC is able to achieve $94\%$ accuracy at an
average embedding size of $0.47$KB, a reduction of only $5\%$ in accuracy for
a size reduction of $\sim$$700$x, greatly outperforming 3 and 2-bit UQE. The
benefit of transmitting the first layer of the decoder is also apparent,
particularly as the compression level increases.

2\. Semantic Segmentation: Results are shown in Fig. 3(b). NEC scales better
with compression, achieving more than a $100$x reduction in size with only
$5\%$ reduced overall accuracy, compared to a $12\%$ drop in accuracy for
comparable compression using UQE. Fig. 4 shows a visual comparison of the
inference outputs. For a smaller bit budget on this difficult image, with many
separate occurrences of classes to identify, NEC produces more accurate
segmentation masks.

In both tasks, we demonstrate the practical viability of our method, with only
a small performance gap between RDC and NEC , despite not fine-tuning the
encoder.

##  6 CONCLUSION

In this work, we introduce NEC, a framework for downstream training and
inference for EO tasks based on directly transmitting compressed embeddings.
We show how any FM can be adapted to produce compressed embeddings of
different fidelities using a single architecture and with minimal compute. Our
evaluation concludes with a data transfer reduction of one to three orders of
magnitude with only limited degradation in performance.

We argue the utility of our work from a sustainability point of view, enabling
training and inference with limited bandwidth or computational resources and
reducing data transfer, storage requirements, and their associated energy
costs. Furthermore, we see potential in the long-term storage of compressed
embeddings produced from large repositories of EO data (e.g., Landsat,
Sentinel) for future analysis.

## References

  * [1] Adrianna Castriotta,  “Copernicus sentinel data access annual report,”  Tech. Rep., ESA, 2022.
  * [2] Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, and Xiao Xiang Zhu,  “Ssl4eo-s12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in earth observation,”  arXiv preprint arXiv:2211.07044, 2022.
  * [3] Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, and Liangpei Zhang,  “Advancing plain vision transformer toward remote sensing foundation model,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–15, 2023.
  * [4] Xian Sun, Peijin Wang, Wanxuan Lu, Zicong Zhu, Xiaonan Lu, Qibin He, Junxi Li, Xuee Rong, Zhujun Yang, Hao Chang, Qinglin He, Guang Yang, Ruiping Wang, Jiwen Lu, and Kun Fu,  “Ringmo: A remote sensing foundation model with masked image modeling,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–22, 2023.
  * [5] Johannes Jakubik, Sujit Roy, CE Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, et al.,  “Foundation models for generalist geospatial artificial intelligence,”  arXiv preprint arXiv:2310.18660, 2023.
  * [6] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon,  “Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery,”  Advances in Neural Information Processing Systems, vol. 35, pp. 197–211, 2022.
  * [7] Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell,  “Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4088–4099.
  * [8] Keumgang Cha, Junghoon Seo, and Taekyung Lee,  “A billion-scale foundation model for remote sensing images,”  arXiv preprint arXiv:2304.05215, 2023.
  * [9] G.K. Wallace,  “The jpeg still picture compression standard,”  IEEE Transactions on Consumer Electronics, vol. 38, no. 1, pp. xviii–xxxiv, 1992.
  * [10] ISO Central Secretary,  “Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s Part 3: Audio,”  Standard, International Organization for Standardization, Aug. 1993.
  * [11] Johannes Ballé, Valero Laparra, and Eero P. Simoncelli,  “End-to-end Optimized Image Compression,” Mar. 2017,  arXiv:1611.01704 [cs, math].
  * [12] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston,  “Variational image compression with a scale hyperprior,”  in International Conference on Learning Representations, 2018.
  * [13] Fabian Mentzer, George Toderici, David Minnen, Sergi Caelles, Sung Jin Hwang, Mario Lucic, and Eirikur Agustsson,  “VCT: A video compression transformer,”  in Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, Eds., 2022.
  * [14] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao,  “Dvc: An end-to-end deep video compression framework,”  arXiv preprint arXiv:1812.00101, 2018.
  * [15] David Minnen, Johannes Ballé, and George D Toderici,  “Joint autoregressive and hierarchical priors for learned image compression,”  in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. 2018, vol. 31, Curran Associates, Inc.
  * [16] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Ballé, Sung Jin Hwang, and George Toderici,  “Scale-space flow for end-to-end optimized video compression,”  in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8500–8509.
  * [17] Deniz Oktay, Johannes Ballé, Saurabh Singh, and Abhinav Shrivastava,  “Scalable Model Compression by Entropy Penalized Reparameterization,”  arXiv:1906.06624 [cs, stat], Feb. 2020,  arXiv: 1906.06624.
  * [18] Anqi Li, Feng Li, Jiaxin Han, Huihui Bai, Runmin Cong, Chunjie Zhang, Meng Wang, Weisi Lin, and Yao Zhao,  “You can mask more for extremely low-bitrate image compression,”  arXiv preprint arXiv:2306.15561, 2023.
  * [19] Lin Liu, Mingming Zhao, Shanxin Yuan, Wenlong Lyu, Wengang Zhou, Houqiang Li, Yanfeng Wang, and Qi Tian,  “Exploring effective mask sampling modeling for neural image compression,”  arXiv preprint arXiv:2306.05704, 2023.
  * [20] Bart Beusen, Xenia Ivashkovych, and Tanja Van Achteren,  “Image compression using vector-quantized auto-encoders with semantically meaningful feature extraction,”  05 2022.
  * [21] Aaron Van Den Oord, Oriol Vinyals, et al.,  “Neural discrete representation learning,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [22] Yi Wang, Conrad M. Albrecht, Nassim Ait Ali Braham, Lichao Mou, and Xiao Xiang Zhu,  “Self-supervised learning in remote sensing: A review,”  IEEE Geoscience and Remote Sensing Magazine, vol. 10, no. 4, pp. 213–247, 2022.
  * [23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick,  “Masked autoencoders are scalable vision learners,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16000–16009.
  * [24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in International Conference on Learning Representations, 2021.
  * [25] C. E. Shannon,  “A mathematical theory of communication,”  The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948.
  * [26] Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja,  “Compressai: a pytorch library and evaluation platform for end-to-end compression research,”  arXiv preprint arXiv:2011.03029, 2020.
  * [27] Yang Long, Gui-Song Xia, Shengyang Li, Wen Yang, Michael Ying Yang, Xiao Xiang Zhu, Liangpei Zhang, and Deren Li,  “On creating benchmark dataset for aerial image interpretation: Reviews, guidances and million-aid,”  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 4205–4230, 2021.
  * [28] Yann Collet and Murray Kucherawy,  “Zstandard Compression and the ’application/zstd’ Media Type,” RFC 8878, Feb. 2021.
  * [29] Yi Yang and Shawn Newsam,  “Bag-of-visual-words and spatial extensions for land-use classification,”  in Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, New York, NY, USA, 2010, GIS ’10, p. 270–279, Association for Computing Machinery.
  * [30] Ross Wightman,  “Pytorch image models,” https://github.com/rwightman/pytorch-image-models, 2019.
  * [31] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  in International Conference on Learning Representations, 2019.
  * [32] F. Rottensteiner, G. Sohn, J. Jung, M. Gerke, C. Baillard, S. Benitez, and U. Breitkopf,  “The isprs benchmark on urban object classification and 3d building reconstruction,”  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. I-3, pp. 293–298, 2012.
  * [33] Kashu Yamazaki, Taisei Hanyu, Minh Tran, Adrian Garcia, Anh Tran, Roy McCann, Haitao Liao, Chase Rainwater, Meredith Adkins, Andrew Molthan, et al.,  “Aerialformer: Multi-resolution transformer for aerial image segmentation,”  arXiv preprint arXiv:2306.06842, 2023.
  * [34] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang,  “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1874–1883.


# Cluster-Guided Unsupervised Domain Adaptation for Deep Speaker Embedding

Haiquan Mao, Feng Hong, , and Man-wai Mak This work was in part supported by
the National Natural Science Foundation of China under Grant 61971371.
(Corresponding author: Man-wai Mak.)Haiquan Mao is with Shanghai Acoustics
Laboratory, Chinese Academy of Sciences, Shanghai, China, and University of
Chinese Academy of Sciences, Beijing, China (e-mail:
maohaiquan20@mails.ucas.ac.cn).Feng Hong is with The Shenzhen Research
Institute, The Hong Kong Polytechnic University, Shenzhen, China, and Shanghai
Acoustics Laboratory, Chinese Academy of Sciences, Shanghai, China (e-mail:
hongfeng@mail.ioa.ac.cn).Man-wai Mak is with the Department of Electronic and
Information Engineering, The Hong Kong Polytechnic University, Hong Kong,
China (e-mail: enmwmak@polyu.edu.hk).

###### Abstract

Recent studies have shown that pseudo labels can contribute to unsupervised
domain adaptation (UDA) for speaker verification. Inspired by the self-
training strategies that use an existing classifier to label the unlabeled
data for retraining, we propose a cluster-guided UDA framework that labels the
target domain data by clustering and combines the labeled source domain data
and pseudo-labeled target domain data to train a speaker embedding network. To
improve the cluster quality, we train a speaker embedding network dedicated
for clustering by minimizing the contrastive center loss. The goal is to
reduce the distance between an embedding and its assigned cluster center while
enlarging the distance between the embedding and the other cluster centers.
Using VoxCeleb2 as the source domain and CN-Celeb1 as the target domain, we
demonstrate that the proposed method can achieve an equal error rate (EER) of
8.10% on the CN-Celeb1 evaluation set without using any labels from the target
domain. This result outperforms the supervised baseline by 39.6% and is the
state-of-the-art UDA performance on this corpus.

###### Index Terms:

Speaker verification, unsupervised domain adaptation, speaker clustering,
contrastive center loss.

##  I Introduction

Speaker verification aims at verifying whether a given speech corresponds to
the claimed speaker [1]. In recent years, deep neural networks have
significantly improved the performance of speaker verification systems [2].
However, training these models to achieve good performance requires a large
amount of labeled data. As manual labeling is laborious, well-labeled training
data in a specific domain are not always available. When a well-trained model
is applied to an unseen domain, mismatch between the source domain and the
target domain can result in severe performance degradation.

Unsupervised domain adaptation (UDA) addresses the domain-mismatch problem by
utilizing labeled source domain data and unlabeled target domain data [3, 4],
leading to performance improvement on the target domain. For better
utilization of the unlabeled data and to enhance the feature extraction
capability of the resulting network, contrastive self-supervised learning
(CSL) has been proposed to increase the similarity of positive training pairs
and decrease the similarity of negative pairs in the latent embedding space.
Typical examples of CSL are SimCLR [5] and MoCo [6] in computer vision. These
methods also work well in speaker recognition [7, 8, 9]. Chen et al. [10]
proposed a CSL-based domain adaptation approach and achieved promising results
on CN-Celeb. However, CSL requires a large batch size to attain its full
potential but the batch size is constrained by GPU memory [5]. Besides, the
performance of CSL is limited by the absence of speaker identity information
[11]. To avoid these problems, researchers applied clustering-based methods
[11]–[13] on unlabeled data to obtain pseudo speaker labels so that the
conventional cross-entropy loss can be applied. Because the pseudo labels can
be noisy, direct minimization of the cross-entropy loss derived from the
pseudo labels could lead to poorly performed speaker embedding networks [14].
Therefore, many studies [11, 12, 13] attempted to purify the pseudo labels.
Because cluster quality is highly dependent on the structure of the input
data, recent works have focused on training a deep network to map the input
data to a latent space where separation of clusters is more manageable [15].

In clustering-based UDA methods, the two main problems are that the number of
clusters is hard to determine and the estimated pseudo labels may be
incorrect. In this paper, we propose a novel UDA framework that can reduce the
impact of these two problems. First, we design the contrastive center loss to
fine-tune the network. We use this loss to simultaneously realize intra-class
compactness and inter-class separability, which is achieved by reducing the
distance between an embedding and its assigned cluster center while enlarging
the distance between the embedding and the other cluster centers. The fine-
tuning process can not only improve the recognition performance but also make
the network extract more cluster-friendly embeddings. Therefore, we can obtain
the pseudo-labeled target domain data with less noisy labels. Second, because
the supervised training result is related to the proportion of noisy labels,
we combine the large labeled source domain data and the pseudo-labeled target
domain data, which further reduces the proportion of noisy labels in the
training data. Thus, the supervised training on the combined dataset can
achieve good performance even though the number of clusters is misestimated.
Our primary contributions are as follows: (1) we validate that fine-tuning a
network using the contrastive center loss can bring improvements in both
signification performance and cluster quality on the target domain data; (2)
we propose a cluster-guided UDA framework that can maximally leverage the
labeled source domain data and the unlabeled target domain data.

Figure 1: Overview of the proposed UDA framework. “Updating cluster centers”
means using the speaker embedding network $g_{\theta}$ to extract speaker
embeddings of all unlabeled utterances followed by clustering. The obtained
cluster assignments and centers are used for computing the contrastive center
loss $L_{cc}$. The numbers in parentheses are the equation numbers. $L_{sc}$
denotes speaker classification loss.

##  II Methods

###  II-A Training Framework

As shown in Fig. 1, our UDA framework involves several steps: (1) pre-train a
model using the speaker classification loss in the source domain and the
contrastive loss in the target domain; (2) extract speaker embeddings and
perform clustering on the unlabeled target domain data using the pre-trained
model; (3) fine-tune the model with the additional contrastive center loss and
update the clusters; (4) use the fine-tuned model $g_{\theta}$ to extract new
embeddings, followed by re-clustering to obtain pseudo labels; and (5) conduct
supervised training using labeled source domain data and pseudo-labeled target
domain data. The well-trained speaker embedding network $f_{\theta}$ is used
as the final model.

###  II-B Contrastive Self-supervised Learning

CSL assumes that two augmented segments from the same utterance form a
positive pair, while two segments from different utterances form a negative
pair. The main idea is to train the network without using data labels by
pulling the samples in positive pairs together and pushing the samples in
negative pairs apart in the embedding space.

For each mini-batch $\mathcal{U}$ with $N$ utterances randomly sampled from
the unlabeled dataset, it is assumed that these $N$ utterances are from
different speakers. For each utterance $u_{i}$ in the mini-batch, we randomly
take two non-overlapping segments of equal length, denoted by $u_{i,1}$ and
$u_{i,2}$. Each segment is randomly augmented to emulate various channel
characteristics and noise effects, forcing the network to acquire robust
speaker representations that are channel-invariant [8]. Afterward, all the
segments are fed to the speaker embedding network $g_{\theta}$ to extract the
speaker embeddings:

$\boldsymbol{e}_{i,j}=g_{\theta}\left(u_{i,j}\right),\ i=1,2,\ldots,N;\
j=1,2.$ (1)

Contrastive loss is designed to maximize the similarity of positive samples
and minimize that of negative samples:

$L_{ct}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{s\left(\boldsymbol{e}_{i,1},\boldsymbol{e}_{i,2}\right)}{\sum_{m=1}^{N}s\left(\boldsymbol{e}_{i,1},\boldsymbol{e}_{m,2}\right)},$
(2)

where $s(\cdot,\cdot)$ denotes a score function that can be Euclidean distance

$s_{1}(\boldsymbol{x},\boldsymbol{y})=\exp\left\\{-\frac{1}{\lambda^{2}}\|\boldsymbol{x}-\boldsymbol{y}\|^{2}\right\\}$
(3)

or cosine similarity

$s_{2}(\boldsymbol{x},\boldsymbol{y})=\exp\\{\omega\cos(\boldsymbol{x},\boldsymbol{y})+b\\},$
(4)

where $\lambda$, $\omega$, and $b$ are learnable parameters.

###  II-C Pre-training

In our strategy, the initial clustering of unlabeled data requires a pre-
trained model, which can be obtained either by applying supervised learning on
the labeled source domain data or by applying CSL on the unlabeled target
domain data. Because Chen et al. [10] have shown that joint training with the
labeled source domain data and the unlabeled target domain data can achieve
better performance, we performed the same joint training in which the source
domain speaker classification loss $L_{sc}$ and the target domain contrastive
loss $L_{ct}$ are simultaneously minimized:

$L_{pre}=L_{sc}+\alpha L_{ct},$ (5)

where $\alpha$ is a weighting factor. Here, the speaker classification loss
$L_{sc}$ can be the common Softmax loss or some variants of it, such as
A-Softmax [16], AM-Softmax [17, 18], and AAM-Softmax [19].

###  II-D Fine-tuning the Model for Better Clustering

Generally, the performance of CSL is limited because it does not use speaker
labels. An intuitive method is to cluster the unlabeled data to produce pseudo
labels and then perform supervised training. However, considering that too
much label noise could severely hinder the effectiveness of this strategy,
directly using the pseudo-labeled data may not achieve good performance. To
reduce the label noise, we dedicatedly train a network for clustering. This
network minimizes the joint loss comprising speaker classification loss
$L_{sc}$, contrastive loss $L_{ct}$, and contrastive center loss $L_{cc}$ to
fine-tune the pre-trained model.

####  II-D1 Contrastive Center Loss

After extracting the speaker embeddings of all unlabeled utterances, the
k-means algorithm is employed to generate cluster assignments and centers.
Generally, discriminative speaker embeddings should have good intra-class
compactness and inter-class separability. Therefore, if we force each
embedding to be near its assigned cluster center and separate from other
cluster centers, k-means would result in better clustering performance. The
contrastive center loss is defined as

$L_{cc}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{s\left(\boldsymbol{e}_{i},\boldsymbol{c}_{y_{i}}\right)}{\sum_{k=1}^{K}s\left(\boldsymbol{e}_{i},\boldsymbol{c}_{k}\right)},$
(6)

where $\boldsymbol{e}_{i}$ is the average of $\boldsymbol{e}_{i,1}$ and
$\boldsymbol{e}_{i,2}$, $\boldsymbol{c}_{y_{i}}$ represents the cluster center
for $\boldsymbol{e}_{i}$, and $\boldsymbol{c}_{k}$ represents any cluster
center. $K$ denotes the total number of clusters. The contrastive center loss
estimates each speaker embedding as the center of the speaker embeddings.
Different from other centroid-based losses which require the true speaker
labels to obtain the centers, the centers in the contrastive center loss are
obtained by clustering. Although there are some incorrect data labels, most
cluster centers can be seen as correct for a given embedding. Therefore, the
fine-tuning process can also improve the signification performance of the
network.

####  II-D2 Joint Loss

During the fine-tuning process, the speaker classification loss and the
contrastive loss constrain the network to maintain the speaker discrimination
ability on the target domain, while the contrastive center loss forces the
network to map the input data to a more clustering-friendly embedding space.
The joint loss is their weighted sum:

$L_{joint}=L_{sc}+\alpha L_{ct}+\beta L_{cc},$ (7)

where $\alpha$ is the same as that in Eq. 5 and $\beta$ is a preset parameter.

####  II-D3 Cluster Updates

At the beginning, we apply k-means on the embeddings extracted from the pre-
trained model. As fine-tuning continues, the network can extract embeddings
more conducive to clustering. Correspondingly, it is necessary to update the
cluster assignments and centers. Thus, we perform re-clustering every $P$
epochs during the fine-tuning process. When the contrastive center loss
converges, the fine-tuning process ends.

###  II-E Supervised Training on the Combined Dataset

We use the fine-tuned model $g_{\theta}$ to extract speaker embeddings of the
unlabeled target domain data and then produce pseudo labels using the k-means
algorithm. To perform domain adaptation, the labeled source domain data are
combined with the pseudo-labeled target domain data, and the total number of
classes is their sum. The proportion of noisy labels is low in the combined
dataset, which reduces the impact of noisy labels on supervised training.
Finally, a new speaker embedding network $f_{\theta}$ is trained with a
classification layer and cross-entropy loss on the combined dataset.

##  III Experimental Setup

###  III-A Datasets

We conducted experiments on VoxCeleb2 [20] and CN-Celeb1 [21], which were used
as the source domain data and the target domain data, respectively. VoxCeleb2
is a large-scale English corpus that contains over one million utterances from
5,994 speakers. CN-Celeb1 is a multi-genre Chinese corpus, in which 797 and
200 speakers are reserved for training and evaluation, respectively. Because
over 30% of the utterances in CN-Celeb1 are shorter than 2 seconds, we
followed the previous work [10] and spliced the short utterances in the
training data to utterances longer than 5 seconds. The number of training
utterances after splicing is 64,150. The speaker labels of the CN-Celeb1
training data were not used in the UDA framework. Instead, they were only used
when verifying the performance of clustering. For data augmentation, the MUSAN
corpus [22] and room impulse responses (RIR) [23] were used for adding
additive noise and reverb.

We evaluated the performance of all the training strategies on the CN-Celeb1
evaluation set, using the official trial list that contains 3,484,292 trial
pairs in total. In our experiments, we used cosine similarity to score the
trial pairs and the performance was evaluated in EER and minimum detection
cost function (minDCF).

###  III-B Network Topology and Hyperparameters

The architecture of the speaker embedding network followed the ECAPA-TDNN in
[24]. In addition, attentive statistics pooling (ASP) was used as the pooling
layer. The channel size was set to 1,024. The embedding dimension of ECAPA-
TDNN was set to 192. The supervised training used the additive angular margin
loss (AAM-Softmax loss) [25] with a margin of 0.2 and softmax prescaling of
30. The network parameters were optimized by an Adam optimizer [26]. The
initial learning rate was set to 0.001 and decreased by 5% for each epoch.

All speech files have a sampling rate of 16 kHz. For each file, an
80-dimensional log Mel-spectrogram was extracted from a 25ms window with a
10ms frame shift before being fed to the network. For the supervised training,
the duration of each utterance was cropped to 2s. As for the CSL, each
utterance in the CN-Celeb1 training set was randomly truncated into two non-
overlapping 2-sec segments. Because cosine metric is more relevant and natural
with AAM-Softmax and the existing work [27] has shown that cosine-based
similarity metric outperforms Euclidean distance in metric learning
objectives, we used cosine similarity to perform k-means and calculate the
contrastive loss and the contrastive center loss. The speaker embeddings and
centers are length-normalized during the fine-tuning process.

The initial values of $\omega$ and $b$ in Eq. 4 were set to 10 and -5. The
weighting coefficients $\alpha$ and $\beta$ in Eq. 7 were set to 1.0. First we
set the number of clusters $K$ to 800, and then we varied it to investigate
its influence on the performance. We evaluated the performance of the network
$g_{\theta}$ every 5 epochs and accordingly $P$ was set to 5. The batch size
for supervised training is 256, whereas that for contrastive self-supervised
training is 128.

##  IV Results and Discussions

###  IV-A Model Performance before Domain Adaptation

In this section, we report the performance of CSL and compare it with that of
supervised training on both the source domain and the target domain. As shown
in Table I, although the performance of contrastive self-supervised training
is worse than that of supervised training on CN-Celeb1, CSL is still an
effective way to learn speaker representations when the data labels are
unknown. Besides, due to the domain mismatch and the multi-genre challenge,
the results of supervised training on both VoxCeleb2 and CN-Celeb1 are
unsatisfactory. The result of supervised training on VoxCeleb2 is considered
the baseline for subsequent experiments.

TABLE I: Results on CN-Celeb1 without domain adaptation Training Data | Training Mode | EER (%) | minDCF
---|---|---|---
VoxCeleb2 | Supervised | 13.40 | 0.56
CN-Celeb1 | Supervised | 11.04 | 0.45
CSL | 13.36 | 0.59

###  IV-B The Effectiveness of the Fine-tuned Model for Clustering

To validate that the fine-tuned model can facilitate clustering, we compared
the clustering results using different models for extracting speaker
embeddings. To evaluate the cluster quality, we used purity and normalized
mutual information (NMI) as the external metrics, which compare the clustering
results with the ground truth. The higher purity or NMI indicates better
cluster quality. Besides, we used Calinski-Harabasz (CH) indexes [28] and
Silhouette scores (SS) [29] as the internal metrics to evaluate the clustering
performance without using any actual labels. A higher CH or SS indicates that
the clustering has high intra-class compactness and inter-class separability.
The first two lines in Table II reveal that although the supervised training
($L_{sc}$) on the source domain data has poorer performance in terms of EER,
it has higher purity and NMI but lower CH and SS. A possible reason is that
without speaker labels, the contrastive self-supervised training mistakenly
decreases the distance between two utterances belonging to different speakers.
Comparing the last two lines, we can see that by adding the contrastive center
loss to fine-tune the model, the clustering performance is better in both
external and internal metrics.

TABLE II: EER and clustering performance of the speaker embedding vectors based on different combinations of training losses Training Loss | EER (%) | Purity | NMI | CH | SS
---|---|---|---|---|---
$L_{sc}$ | 13.40 | 0.717 | 0.829 | 91.356 | 0.099
$L_{ct}$ | 13.36 | 0.658 | 0.804 | 130.130 | 0.153
$L_{sc}+L_{ct}$ | 10.30 | 0.786 | 0.875 | 100.239 | 0.136
$L_{sc}+L_{ct}+L_{cc}$ | 9.48 | 0.829 | 0.898 | 137.544 | 0.185

Additionally, we further analyze the variations of EER and NMI during the
fine-tunning process. As shown in Fig.2, the fine-tuning process can bring
improvements in both recognition performance (EER) and clustering performance
(NMI), and such improvements are less affected by the misestimation of the
total number of clusters.

|
---|---
(a) EER | (b) NMI
Figure 2: The variations of EER and NMI during the fine-tuning process.

###  IV-C Performance Comparison with Existing Methods

Table III compares the performance of our framework with existing methods on
the CN-Celeb1 evaluation set. We also varied the number of clusters $K$ to
explore its effect on the performance. Evidently, when the true labels are
available on both domains, supervised training on the combined dataset
outperforms all the existing methods. This result is considered the upper
bound performance in our UDA method. When the number of clusters was set to
800, which is close to the actual number of speakers in CN-Celeb1 ($K$=797),
our method can achieve 39.6% performance improvement over the baseline model.
This result is very close to the upper bound performance. Although the
performance of the models varies with $K$, all of them outperform the baseline
model, which indicates that our method brings significant performance
improvement even when the number of clusters deviates from the true one.

TABLE III: EER Performance of the proposed method and other existing methods on CN-Celeb1 Method | Training Data | Backbone | EER (%)
---|---|---|---
Fan et al. [21] | CN1 | i-vector | 14.24
Li et al. [30] | CN1+CN2 | x-vector | 12.52
Zhang et al. [31] | Vox2 | Rep-A-TMS-TDNN | 10.25
Zeng et al. [32] | CN1+CN2 | ECAPA-TDNN | 8.93
Chen et al. [10] | Vox2+CN1 | r-vector | 8.86
Ours | Vox2+CN1 | ECAPA-TDNN | 8.05
Chen et al. [10] | Vox2+CN1.U | r-vector | 10.20
Ours ($K$=400) | Vox2+CN1.U | ECAPA-TDNN | 8.38
Ours ($K$=800) | Vox2+CN1.U | ECAPA-TDNN | 8.10
Ours ($K$=1600) | Vox2+CN1.U | ECAPA-TDNN | 8.99
CN2 means CN-Celeb2 dataset [30], which is a supplement of CN-Celeb1 and
contains 529485 from 2000 speaker. CN1.U means CN-Celeb1 without data labels.

##  V Conclusion

This paper proposes an effective cluster-guided UDA framework, which consists
of fine-tuning the network using contrastive center loss for better clustering
and the supervised training on the combined dataset for robust representation
learning. The proposed method achieves an EER of 8.10% on CN-Celeb1 without
using any target domain labels, and works well even when the number of
clusters deviates from the actual number of speakers.

## References

  * [1] J. H. L. Hansen and T. Hasan, “Speaker recognition by machines and humans: A tutorial review,” IEEE Signal Process. Mag., vol. 32, no. 6, pp. 74–99, Nov. 2015.
  * [2] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Apr. 2018, pp. 5329–5333.
  * [3] G. Wilson and D. J. Cook, “A survey of unsupervised deep domain adaptation,” ACM Trans. Intell. Syst. Technol., vol. 11, no. 5, pp. 1–46, 2020.
  * [4] Z. Bai and X.-L. Zhang, “Speaker recognition based on deep learning: An overview,” Neural Netw., vol. 140, pp. 65–99, Aug. 2021.
  * [5] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in Proc. Int. Conf. Mach. Learn. (ICML), vol. 119, 2020, pp. 1597–1607.
  * [6] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020, pp. 9729–9738.
  * [7] J. Huh, H. S. Heo, J. Kang, S. Watanabe, and J. S. Chung, “Augmentation adversarial training for unsupervised speaker recognition,” in Workshop on Self-Supervised Learning for Speech and Audio Processing, NeurIPS, 2020.
  * [8] H. Zhang, Y. Zou, and H. Wang, “Contrastive self-supervised learning for text-independent speaker verification,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Jun. 2021, pp. 6713–6717.
  * [9] W. Xia, C. Zhang, C. Weng, M. Yu, and D. Yu, “Self-supervised text-independent speaker verification using prototypical momentum contrastive learning,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Jun. 2021, pp. 6723–6727.
  * [10] Z. Chen, S. Wang, and Y. Qian, “Self-supervised learning based domain adaptation for robust speaker verification,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Jun. 2021, pp. 5834–5838.
  * [11] R. Tao, K. A. Lee, R. K. Das, V. Hautamäki, and H. Li, “Self-supervised speaker recognition with loss-gated learning,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May. 2022, pp. 6142–6146.
  * [12] J. Thienpondt, B. Desplanques, and K. Demuynck, “The IDLAB voxceleb speaker recognition challenge 2020 system description,” arXiv preprint arXiv:2010.12468, 2020.
  * [13] D. Cai, W. Wang, and M. Li, “An iterative framework for self-supervised deep speaker representation learning,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Jun. 2021, pp. 6728–6732.
  * [14] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 132–149.
  * [15] E. Aljalbout, V. Golkov, Y. Siddiqui, and D. Cremers, “Clustering with deep learning: Taxonomy and new methods,” arXiv preprint arXiv:1801.07648, 2018.
  * [16] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 212–220.
  * [17] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “Cosface: Large margin cosine loss for deep face recognition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 5265–5274.
  * [18] F. Wang, J. Cheng, W. Liu, and H. Liu, “Additive margin softmax for face verification,” IEEE Signal Process. Lett., vol. 25, no. 7, pp. 926–930, Jul. 2018.
  * [19] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in Proc. IEEE/CVF Conf. Comput.Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 4690–4699.
  * [20] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “VoxCeleb: Large-scale speaker verification in the wild,” Comput. Speech Lang., vol. 60, 2020, Art. no. 101027. [Online]. Available: http://www.sciencedirect. com/science/article/pii/S0885230819302712
  * [21] Y. Fan, J. W. Kang, L. T. Li, K. C. Li, H. L. Chen, S. T. Cheng, P. Y. Zhang, Z. Y. Zhou, Y. Q. Cai, and D. Wang, “CN-Celeb: A challenging chinese speaker recognition dataset,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May. 2020, pp. 7604–7608.
  * [22] D. Snyder, G. Chen, and D. Povey, “MUSAN: A music, speech, and noise corpus,” arXiv preprint arXiv:1510.08484, 2015.
  * [23] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Mar. 2017, pp. 5220–5224.
  * [24] B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2020, pp. 1–5.
  * [25] X. Xiang, S. Wang, H. Huang, Y. Qian, and K. Yu, “Margin matters: Towards more discriminative deep neural network embeddings for speaker recognition,” in Proc. Asia–Pacific Signal Inf. Process. Assoc. Annu. Summit Conf. (APSIPA ASC), Nov. 2019, pp. 1652–1656.
  * [26] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [27] J. S. Chung, J. Huh, S. Mun, M. Lee, H. S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In defence of metric learning for speaker recognition,” arXiv preprint arXiv:2003.11982, 2020.
  * [28] T. Calinski and J. Harabasz, “A dendrite method for cluster analysis,” Commun. Statist. Theory Methods, vol. 3, no. 1, pp. 1–27, 1974.
  * [29] P. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” J. Comp. App. Math, vol. 20, pp. 53–65, Nov. 1987.
  * [30] L. Li, R. Liu, J. Kang, Y. Fan, H. Cui, Y. Cai, R. Vipperla, T. F. Zheng, and D. Wang, “CN-Celeb: multi-genre speaker recognition,” Speech Communication, vol. 137, pp. 77-91, 2022.
  * [31] R. Zhang, J. Wei, X. Lu, W. Lu, D. Jin, J. Xu, L. Zhang, Y. Ji, and J. Dang, “TMS: A temporal multi-scale backbone design for speaker embedding,” arXiv preprint arXiv:2203.09098, 2022.
  * [32] C. Zeng, X. Wang, E. Cooper, and J. Yamagishi, “Attention back-end for automatic speaker verification with multiple enrollment utterances,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May. 2022, pp. 6717-6721.


The Structure of Systematicity in the Brain
Randall C. O’Reilly, Charan Ranganath & Jacob L. Russin
Departments of Psychology and Computer Science
Center for Neuroscience
University of California, Davis
1544 Newton Ct
Davis, CA 95618
<EMAIL_ADDRESS>
August 27, 2024
Abstract:

> A hallmark of human intelligence is the ability to adapt to new situations,
> by applying learned rules to new content (systematicity) and thereby
> enabling an open-ended number of inferences and actions (generativity).
> Here, we propose that the human brain accomplishes these feats through
> pathways in the parietal cortex that encode the abstract structure of space,
> events, and tasks, and pathways in the temporal cortex that encode
> information about specific people, places, and things (content). Recent
> neural network models show how the separation of structure and content might
> emerge through a combination of architectural biases and learning, and these
> networks show dramatic improvements in the ability to capture systematic,
> generative behavior. We close by considering how the hippocampal formation
> may form integrative memories that enable rapid learning of new structure
> and content representations.

00footnotetext: Submitted to Current Directions in Psychological Science.
We thank Jon Cohen, Alex Petrov, Erie Boorman, Ignacio Saez, Taylor Webb,
Stephen Frankland, James Antony, John Rohrlich, Mary Zolfaghar, and members of
the CCN, DML, and Cohen labs for discussion and comments. R. C. O’Reilly is
Chief Scientist at eCortex, Inc., which may derive indirect benefit from the
work presented here. Supported by: ONR grants N00014-20-1-2578,
N00014-19-1-2684/ N00014-18-1-2116, N00014-18-C-2067, N00014-17-1-2961,
N00014-15-1-0033.

The ability to directly transfer existing knowledge to new situations by
applying learned rules to new content (_systematicity_) and thereby generate
an open-ended number of different behaviors (_generativity_) is particularly
advanced in humans relative to other animals. However, this ability is
nevertheless shared in a limited but instructive way with even relatively
simple computer programs. For example, it is easy to write a function that
takes two arbitrary strings as arguments, and systematically returns the
concatenation of those two strings. With just a bit more logic, calling such a
function repeatedly can easily construct an open-ended number of sentences.
However, learning to do something like this in a neurally-plausible manner is
much more challenging, and neural network models continue to be criticized as
lacking in these signature human abilities, with much of their recent
successes potentially attributable to something closer to rote memorization of
increasingly large datasets (Fodor Pylyshyn, 1988; Lake Baroni, 2017; Marcus,
2018; O’Reilly ., 2014; Plaut ., 1996). Likewise, empirical cognitive
neuroscience is only beginning to uncover the relevant brain areas involved in
these abilities (Constantinescu ., 2016; Park ., 2020; Summerfield ., 2020;
Frankland Greene, 2020).

The essential trick employed by the simple concatenation function is that
arbitrary content can be routed into it and operated upon generically,
independent of any details of the content. The cognitive equivalent of this,
which is widely recognized as critical for human systematic behavior (Fodor
Pylyshyn, 1988) is a separation of _structure_ (i.e., the rule-processing
system, akin to the function) from _content_. In language, syntax has
traditionally been thought of as an example of content-independent structure,
and Chomsky’s famous example: “colorless green ideas sleep furiously”
demonstrates that people can accurately judge grammatical correctness with
arbitrary content. Without adopting traditional abstract symbolic syntactic
frameworks, understanding how something like content-independent syntactic
structure can be learned and represented via known biological mechanisms is
thus a critical step toward advancing our understanding of the neural basis of
our signature human abilities.

In this paper, we review cognitive and neural evidence consistent with the
separation between structure and content across various cognitive domains, and
recent neural network models that demonstrate how such a separation might
emerge through a combination of architectural biases and learning, producing
significantly greater systematicity. Critically, structure and content also
need to interact and be integrated in various ways, and certain brain areas
appear to be specialized for this integration, so as is often the case, the
brain is likely to apply multiple representational strategies in parallel.

##  1 Structure and Content in Cognitive Models

The FINST (fingers of instantiation) framework of Pylyshyn (1989) provides an
early, simple model for how structure can be represented independent of
specific content and also maps well onto cognitive neuroscience data reviewed
in the next section. The key idea is that abstract, content-independent,
pointer-like “fingers” can index a small set (up to about 4) of different
visual locations at a time. These content-independent pointers were originally
proposed to serve as place-holders for encoding relationships among items in a
scene (e.g., INSIDE(a,b)).

The core ideas in the FINST framework can be extended to represent any kind of
abstract structure that specifies the relationships between different indexed
elements. For example, to describe the action of _giving_ , there are distinct
functional roles (giver, recipient, item), and each of these could be
represented using a separate FINST-like thematic role pointer, instead of
using the semantic content specific to a particular situation (e.g., “Radhika
gave Charan a gift”). Because _giving_ is encoded in terms of content-
independent roles indexed by FINST-like pointers, these representations can
automatically generalize to new contents, and additional inferences could be
made about the properties and implications of the structural relationship
(e.g., the giver no longer has the item, and may have some expectation of
reciprocity depending on the nature of the transaction and relationship).
Encoding sufficiently rich structural knowledge to do so requires much more
than FINST’s, but encoding this knowledge in a relatively content-independent
manner would clearly support systematic reasoning.

In short, FINST-like indexes provide a plausible attention-based neural
mechanism for role-filler variable slots in the context of classical symbolic
representational frameworks. At a cognitive level, the kinds of elaborated
structural representations and processes that have been studied include a
range of different levels of complexity, from verb / action based elements as
in the _giving_ example (Boylan ., 2015) to more elaborated _schemas_ or
_scripts_ describing longer sequences of events (e.g., the schema associated
with a kid’s birthday party), as we discuss below. The ability to transfer
structural knowledge across content domains is also central to established
models of analogical reasoning (Gentner Holyoak, 1997) (e.g., the relational
structure of _orbiting_ can be transferred from the planetary to atomic
scale).

##  2 Structure Representation in the Human Brain

There is a well-established distinction between spatial and object processing
in the human brain, which can be reframed as one example of how the brain
separates structure and content via distinct, but interacting pathways. Visual
(and auditory) networks in the brain route sensory input into distinct dorsal
and ventral stream pathways (Ungerleider Mishkin, 1982), with the ventral
visual pathway extending from early visual cortex to inferotemporal (IT)
cortex characterized as the _What_ pathway, specialized for object or scene
recognition (i.e., visual _content_). The dorsal pathway through the parietal
lobe is specialized for spatial _Where_ processing, based on extensive
evidence that this pathway represents spatial and relational information in a
relatively content-independent manner.

The potential for the dorsal stream pathway to support systematic structure-
sensitive processing was already well developed by Pylyshyn (1989), and has
been incorporated into psycholinguistic theories (Landau Jackendoff, 1993;
Frankland Greene, 2020). Furthermore, there is substantial evidence that
posterior parietal cortex encodes nonspatial structural information as well.
First, it is clear that the parietal lobe plays a critical role in the sensory
guidance of action performance (Orban ., 2004), as captured in the _What_ vs.
_How_ ventral / dorsal framework (Goodale Milner, 1992). In humans, this
action coding extends to the representation of verb-based argument structure
(e.g., as in the _giving_ example above) in the inferior parietal area
(angular gyrus), supporting the core of structure at the sentence level
(Boylan ., 2015; Binder Desai, 2011; Palmer ., 2016). More abstract structural
and linguistic concepts are thought to build directly upon these parietal
action, space and time foundations (Landau Jackendoff, 1993; Frankland Greene,
2020), including event representations in higher-order parietal areas as
discussed below.

It is well established that parietal spatial and action representations
anticipate the effects of eye, head, or body movements (Cavanagh ., 2010) —
this suggests that acquisition of structure in the parietal lobe could be
based on _predictive learning_ (O’Reilly ., 2021). Specifically, learning
driven by the difference between a predicted sensory outcome of an action and
the subsequent sensory input can drive improved predictions and shape the
formation of more abstract structural representations to more efficiently
generate these predictions. While it is fairly straightforward to learn to
predict specific sensory outcomes from motor actions, an important area of
current research is to determine the extent to which more abstract, structural
representations can emerge, capturing the consistent, generalizable
relationships that hold across a large number of such actions (Summerfield .,
2020).

Even if the dorsal-ventral streams separately encode structure and content in
the brain, fundamental questions remain about how these streams interact at
the level of detailed neural mechanisms, to support systematic cognitive
function. A recent neural network model provides a useful example for how
FINST-like attentional pointers can operate on newly-learned content
information, in the context of separate structure and content representations
(Russin ., 2020). Unlike many existing models that have relied upon various
hand-coded mechanisms to directly emulate programming-language like variable-
binding functionality in neural hardware, this model learned entirely via
error-driven learning, with only very broad, biologically plausible
architectural constraints between two processing pathways.

One such pathway had full access to the temporal ordering of words within a
sentence, while the other was only able to process the single current word at
any given time (Figure 1). Furthermore, the temporal-order sensitive pathway
could only influence the network output via attentional modulation of the
other pathway (similar to the FINST-like attentional pointers). Distinctions
such as these could plausibly derive from evolved differences in the initial
wiring of the neural architecture, providing affordances upon which subsequent
learning operates.

Figure 1: Diagram depicting a neural network model (Russin et al., 2020) with
separate pathways for learning structure (syntax) and content (semantics). The
structure pathway was recurrent (as shown by the blue self-connection) and had
full access to the temporal ordering of the words in the instructions.
However, it could only influence the output actions taken at each time step
through an attention mechanism (indicated by the blue dotted line and open
circle). The character of the representations in these different pathways
facilitated systematic generalization, but was not built in, and emerged due
to the architectural constraints imposed by the separation.

With these constraints in place, the network learned syntax-like structural
representations in the temporal-order pathway, and semantic-like content
information in the other pathway, and in so doing, exhibited significantly
more systematic behavior on challenging out-of-domain generalization tests
than unstructured models (Lake Baroni, 2017). Specifically, when the model was
trained on examples generated from a simple phrase structure grammar, the
syntax-like pathway was able to direct attention to the proper sequence of
content items based on commands like “turn left twice,” and generalize this to
a novel verb, e.g., “jump twice.” The ability of attention-like mechanisms to
drive this FINST-style indexing into novel content items is consistent with
the central importance of such mechanisms in the neural basis of structure /
content dynamics. Furthermore, this model demonstrates how learned,
distributed neural representations can take on a syntax-like role without
traditional explicit symbolic grammar trees. However, more work is needed to
understand the nature of structure representations in the brain supporting the
more complex and sophisticated forms of human cognition, compared to these
simplified models.

Interestingly, a similar distinction between structure and content pathways
was achieved in a fairly different way using a variable-binding system based
on an explicit slot-filler (key-value) lookup table mechanism (Webb ., 2021).
In this model, the structure processing pathway learned to control this lookup
table independent of the specific content stored there, while the content
pathway learned representations of individual items. This architectural
structure / content separation enabled the model to exhibit much more
systematic behavior than comparison models without such a separation. Other
related architectures are discussed in Akyürek Andreas (2021).

While these models provide important initial demonstrations of how a
separation between structure and content can support systematicity, it is
critical to appreciate that many real-world situations also require strong
interactions between the two. This has perhaps been most extensively studied
in the case of pronouncing English words: as every English-as-a-second-
language learner knows too well, the number of exceptions and sub-regularities
are mind-spinning, and monolithic neural network pathways may provide the most
appropriate mechanism for learning these mappings (Plaut ., 1996). Likewise,
the difference in parsing “Fruit flies like a banana” vs. “Time flies like an
arrow” requires an interaction between syntax and semantics. In the visual
domain, these interactions can be illustrated in the case of viewing a scene
of a kitchen, where incoming content information may activate structural
representations of typical kitchens, which then guide visual attention and
semantic processing (Hayes Henderson, 2019). In other words, human cognition
likely represents a combination of separated and interacting streams (O’Reilly
., 2014; Franklin ., 2020).

##  3 Binding of Structure and Content in the Medial Temporal Lobes

So far, we have used the classic model of the dorsal and ventral streams as
one example of the separation of structure and content, but structure exists
across multiple scales. For example, structural knowledge is required to parse
and comprehend individual sentences, but comprehending a sentence within the
context of a larger text passage often requires a mental representation of the
structure of events depicted in that passage. Recent fMRI research has
uncovered a candidate set of brain regions that may represent the abstract
structure of events — the _Posterior Medial_ (PM) network — and regions that
may represent the characteristics of the specific people, places and things
that serve as event content — the _Anterior Temporal_ (AT) network (Figure 2).
The PM and AT networks can be considered as anatomically higher up from the
classic dorsal (PM) and ventral (AT) streams (Kravitz ., 2011; Ranganath
Ritchey, 2012). Although these networks are distinct, the hippocampus is in a
key position to integrate information across the two networks. AT network
regions project predominantly to the lateral entorhinal cortex (LEC), and PM
network regions mostly project to the medial entorhinal cortex (MEC). The
hippocampus proper then binds information from these streams together into
unitary episodic memories (Ranganath Ritchey, 2012). Considerable evidence
suggests that the PM and AT networks differentially represent information
about structure and content, at higher, more complex levels and over longer
spatiotemporal scales than the more posterior dorsal and ventral streams. For
example, the PM network encodes the structure of events and spatial contexts
and participates in the reconstruction of past events and the simulation of
future events (Ranganath Ritchey, 2012), and is also central for discourse
comprehension (Martín-Loeches ., 2008).

Figure 2: Two cortico-hippocampal networks. Top: Surface renderings depict the
neocortical regions in the Posterior Medial (PM) Network (cyan), implicated in
schema representation, and Anterior Temporal (AT) Network (magenta),
implicated in representations of people and things. Bottom: A diagram
schematically depicts the pathways by which information from the AT and PM
Networks is directed to parallel pathways in the medial temporal lobes before
fully converging in the hippocampus. PRC = Perirhinal cortex; PHC =
Parahippocampal cortex; RSC = Retrosplenial cortex; LEC = Lateral entorhinal
cortex; MEC = Medial entorhinal cortex.

We can think of the hippocampus, PM, and AT networks as complementary learning
systems (Russin ., 2021), where the PM and AT networks are specialized for
slow, integrative learning leading to highly generalizable representations,
complementing the rapid episodic memory supported by the hippocampus. By
capturing states of PM and AT network regions at critical moments, the
hippocampus is in a unique position to rapidly form memories that combine
information about structure and content that is encountered during a specific
period of time (i.e., an _episode_). When encountering a similar item or
situation later on, hippocampal pattern completion can lead to reinstatement
of previous states in the PM and AT networks, thereby facilitating the
generation of a mental model of the current situation in a new or ambiguous
context (Franklin ., 2020).

Although they embody principles that are largely consistent with the ones
proposed here, other computational models (Stachenfeld ., 2017; Whittington .,
2020) have placed more emphasis on the role of the hexagonally-tiled _grid
cells_ in the MEC, which are thought to represent the topological structure of
spatial environments, and hippocampal _place cells_ , which are thought to
encode specific locations within an environment. These models suggest that the
MEC and hippocampus can represent any kind of task in a 2D state-space,
consistent with fMRI studies showing MEC activity patterns characteristic of
grid cells during the performance of novel tasks (Constantinescu ., 2016).
Notably, however, these and other studies of humans suggest that grid cells
and grid cell-like fMRI activity patterns are evident throughout the PM
network and in the medial prefrontal cortex (Jacobs ., 2013).

Further work is needed to disentangle the unique contributions of the MEC
relative to earlier parietal-lobe pathways in structure representation. One
possibility is that the use of structural knowledge to guide flexible behavior
is driven by the PM network, and the MEC plays an important role in the use of
episodic memory to support rapid structure learning by the PM network. For
instance, it is possible that the MEC grid cell network might perform an
initial step of _pattern separation_ on inputs from the PM network, moving
them apart in representational space and thus making them easier to learn
without interference (Frankland et al, in prep). This is consistent with
results from lesion studies suggesting that hippocampal or MEC lesions can
significantly impair spatial learning, while generally sparing navigation in
familiar environments (Kolarik ., 2016; Hales ., 2014).

##  4 Conclusions

There is good reason to believe that the architecture of the neocortex is
optimized to support systematic and generative behavior by bifurcating sensory
information into separate structure and content pathways along the dorsal and
ventral streams. The hierarchically organized pathways through the parietal
lobe generate different levels of representations that encode structural
relationships reliably associated with different actions and events. Such
representations capture the general “logic” of actions and events: how things
move, transform, and interact over space and time, in a way that can be
readily applied to novel content. Significant work remains to be done in order
to establish the nature of structure representations. By understanding the
evolution of learning and dynamics across interacting brain networks, we can
make significant progress toward understanding the nature of the
representations that give rise to our uniquely human intelligence.

##  5 Recommended Readings

Pylyshyn, Z. (1989) (See References). An original proposal for a structure /
content separation mechanism based on attention, that we think may still have
currency in neural network models of the brain.

Kravitz, D. J., Saleem, K. S., Baker, C. I., & Mishkin, M. (2011) (See
References). Integrated review of three parietal lobe pathways supporting
looking, reaching, and navigating.

Summerfield, C., Luyckx, F., & Sheahan, H. (2020) (See References). Recent
synthetic review of a wide range of data and theory supporting the idea that
the parietal lobe represents content-independent structure.

Whittington, J. C. R., et al., (2020) (See References). Computational model
implementing structure / content separation in the hippocampal system, with
nice illustrations of different forms of structure and how they can support
systematic behavior in different domains.

## References

  * Akyürek Andreas (2021) AkyurekAndreas21Akyürek, E. Andreas, J.  202106\.  Lexicon Learning for Few-Shot Neural Sequence Modeling Lexicon Learning for Few-Shot Neural Sequence Modeling.  arXiv:2106.03993 [cs].  http://arxiv.org/abs/2106.03993
  * Binder Desai (2011) BinderDesai11Binder, JR. Desai, RH.  201111\.  The Neurobiology of Semantic Memory The neurobiology of semantic memory.  Trends in Cognitive Sciences1511527–536.  10.1016/j.tics.2011.10.001
  * Boylan . (2015) BoylanTrueswellThompson-Schill15Boylan, C., Trueswell, JC. Thompson-Schill, SL.  201511\.  Compositionality and the Angular Gyrus: A Multi-Voxel Similarity Analysis of the Semantic Composition of Nouns and Verbs Compositionality and the angular gyrus: A multi-voxel similarity analysis of the semantic composition of nouns and verbs.  Neuropsychologia78130–141.  10.1016/j.neuropsychologia.2015.10.007
  * Cavanagh . (2010) CavanaghHuntAfrazEtAl10Cavanagh, P., Hunt, AR., Afraz, A. Rolfs, M.  201004\.  Visual Stability Based on Remapping of Attention Pointers Visual stability based on remapping of attention pointers.  Trends in Cognitive Sciences144147–153.  10.1016/j.tics.2010.01.007
  * Constantinescu . (2016) ConstantinescuOReillyBehrens16Constantinescu, AO., O’Reilly, JX. Behrens, TEJ.  201606\.  Organizing Conceptual Knowledge in Humans with a Gridlike Code Organizing conceptual knowledge in humans with a gridlike code.  Science35262921464–1468.  10.1126/science.aaf0941
  * Fodor Pylyshyn (1988) FodorPylyshyn88Fodor, JA. Pylyshyn, ZW.  198804\.  Connectionism and Cognitive Architecture: A Critical Analysis. Connectionism and cognitive architecture: A critical analysis.  Cognition281-23–71.  http://www.ncbi.nlm.nih.gov/pubmed/2450716
  * Frankland Greene (2020) FranklandGreene20Frankland, SM. Greene, JD.  202001\.  Concepts and Compositionality: In Search of the Brain’s Language of Thought Concepts and Compositionality: In Search of the Brain’s Language of Thought.  Annual Review of Psychology711273–303.  10.1146/annurev-psych-122216-011829
  * Franklin . (2020) FranklinNormanRanganathEtAl20Franklin, NT., Norman, KA., Ranganath, C., Zacks, JM. Gershman, SJ.  202004\.  Structured Event Memory: A Neuro-Symbolic Model of Event Cognition Structured Event Memory: A neuro-symbolic model of event cognition.  Psychological Review1273327–361.  10.1037/rev0000177
  * Gentner Holyoak (1997) GentnerHolyoak97Gentner, D. Holyoak, KJ.  199701\.  Reasoning and Learning by Analogy: Introduction Reasoning and learning by analogy: Introduction.  American Psychologist5232–34.
  * Goodale Milner (1992) GoodaleMilner92Goodale, MA. Milner, AD.  199201\.  Separate Visual Pathways for Perception and Action Separate visual pathways for perception and action.  Trends in Neurosciences15120–25.
  * Hales . (2014) HalesSchlesigerLeutgebEtAl14Hales, JB., Schlesiger, MI., Leutgeb, JK., Squire, LR., Leutgeb, S. Clark, RE.  201411\.  Medial Entorhinal Cortex Lesions Only Partially Disrupt Hippocampal Place Cells and Hippocampus-Dependent Place Memory Medial Entorhinal Cortex Lesions Only Partially Disrupt Hippocampal Place Cells and Hippocampus-Dependent Place Memory.  Cell Reports93893–901.  10.1016/j.celrep.2014.10.009
  * Hayes Henderson (2019) HayesHenderson19Hayes, TR. Henderson, JM.  201910\.  Scene Semantics Involuntarily Guide Attention during Visual Search Scene semantics involuntarily guide attention during visual search.  Psychonomic Bulletin & Review2651683–1689.  10.3758/s13423-019-01642-5
  * Jacobs . (2013) JacobsWeidemannMillerEtAl13Jacobs, J., Weidemann, CT., Miller, JF., Solway, A., Burke, JF., Wei, XX.Kahana, MJ.  201309\.  Direct Recordings of Grid-like Neuronal Activity in Human Spatial Navigation Direct recordings of grid-like neuronal activity in human spatial navigation.  Nature Neuroscience1691188–1190.  10.1038/nn.3466
  * Kolarik . (2016) KolarikShahlaieHassanEtAl16Kolarik, BS., Shahlaie, K., Hassan, A., Borders, AA., Kaufman, KC., Gurkoff, G.Ekstrom, AD.  201601\.  Impairments in Precision, Rather than Spatial Strategy, Characterize Performance on the Virtual Morris Water Maze: A Case Study Impairments in precision, rather than spatial strategy, characterize performance on the virtual Morris Water Maze: A case study.  Neuropsychologia8090–101.  10.1016/j.neuropsychologia.2015.11.013
  * Kravitz . (2011) KravitzSaleemBakerEtAl11Kravitz, DJ., Saleem, KS., Baker, CI. Mishkin, M.  201104\.  A New Neural Framework for Visuospatial Processing A new neural framework for visuospatial processing.  Nature Reviews Neuroscience124217–230.  10.1038/nrn3008
  * Lake Baroni (2017) LakeBaroni17Lake, BM. Baroni, M.  201710\.  Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  arXiv:1711.00350 [cs].  http://arxiv.org/abs/1711.00350
  * Landau Jackendoff (1993) LandauJackendoff93Landau, B. Jackendoff, R.  1993\.  ”What” and ”Where” in Spatial Language and Spatial Cognition ”What” and ”where” in spatial language and spatial cognition.  Behavioral and Brain Sciences162217–265.  10.1017/S0140525X00029733
  * Marcus (2018) Marcus18Marcus, G.  201801\.  Deep Learning: A Critical Appraisal Deep Learning: A Critical Appraisal.  arXiv:1801.00631 [cs, stat].  http://arxiv.org/abs/1801.00631
  * Martín-Loeches . (2008) Martin-LoechesCasadoHernandez-TamamesEtAl08Martín-Loeches, M., Casado, P., Hernández-Tamames, JA. Álvarez-Linera, J.  200806\.  Brain Activation in Discourse Comprehension: A 3t fMRI Study Brain activation in discourse comprehension: A 3t fMRI study.  NeuroImage412614–622.  10.1016/j.neuroimage.2008.02.047
  * Orban . (2004) OrbanVanEssenVanduffel04Orban, GA., Van Essen, D. Vanduffel, W.  200401\.  Comparative Mapping of Higher Visual Areas in Monkeys and Humans Comparative mapping of higher visual areas in monkeys and humans.  Trends in Cognitive Sciences.87315–324.
  * O’Reilly . (2014) OReillyPetrovCohenEtAl14O’Reilly, RC., Petrov, AA., Cohen, JD., Lebiere, CJ., Herd, SA. Kriete, T.  2014\.  How Limited Systematicity Emerges: A Computational Cognitive Neuroscience Approach How Limited Systematicity Emerges: A Computational Cognitive Neuroscience Approach.  IP. Calvo J. Symons (), The Architecture of Cognition: Rethinking Fodor and Pylyshyn1s Systematicity Challenge. The architecture of cognition: Rethinking Fodor and Pylyshyn1s Systematicity Challenge.  Cambridge, MAMIT Press.
  * O’Reilly . (2021) OReillyRussinZolfagharEtAl21O’Reilly, RC., Russin, JL., Zolfaghar, M. Rohrlich, J.  2021\.  Deep Predictive Learning in Neocortex and Pulvinar Deep predictive learning in neocortex and pulvinar.  Journal of Cognitive Neuroscience3361158–1196.
  * Palmer . (2016) PalmerBonialHwang16Palmer, M., Bonial, C. Hwang, JD.  201611\.  VerbNet: Capturing English Verb Behavior, Meaning and Usage VerbNet: Capturing English verb behavior, meaning and usage.  SEF. Chipman (), The Oxford Handbook of Cognitive Science The Oxford Handbook of Cognitive Science ( 315–336).  Oxford University Press.
  * Park . (2020) ParkMillerNiliEtAl20Park, SA., Miller, DS., Nili, H., Ranganath, C. Boorman, ED.  202007\.  Map Making: Constructing, Combining, and Inferring on Abstract Cognitive Maps Map Making: Constructing, Combining, and Inferring on Abstract Cognitive Maps.  Neuron.  10.1016/j.neuron.2020.06.030
  * Plaut . (1996) PlautMcClellandSeidenbergEtAl96Plaut, DC., McClelland, JL., Seidenberg, MS. Patterson, K.  199607\.  Understanding Normal and Impaired Word Reading: Computational Principles in Quasi-Regular Domains. Understanding normal and impaired word reading: Computational principles in quasi-regular domains.  Psychological review10356–115.  http://www.ncbi.nlm.nih.gov/pubmed/8650300
  * Pylyshyn (1989) Pylyshyn89Pylyshyn, Z.  198906\.  The Role of Location Indexes in Spatial Perception: A Sketch of the FINST Spatial-Index Model The role of location indexes in spatial perception: A sketch of the FINST spatial-index model.  Cognition32165–97.  10.1016/0010-0277(89)90014-0
  * Ranganath Ritchey (2012) RanganathRitchey12Ranganath, C. Ritchey, M.  201210\.  Two Cortical Systems for Memory-Guided Behaviour. Two cortical systems for memory-guided behaviour.  Nature Reviews Neuroscience1310713–726.  http://www.ncbi.nlm.nih.gov/pubmed/22992647
  * Russin . (2020) RussinJoOReillyEtAl20Russin, JL., Jo, J., O’Reilly, RC. Bengio, Y.  2020\.  Systematicity in a Recurrent Neural Network by Factorizing Syntax and Semantics Systematicity in a Recurrent Neural Network by Factorizing Syntax and Semantics.  Proceedings for the 42nd Annual Meeting of the Cognitive Science Society Proceedings for the 42nd Annual Meeting of the Cognitive Science Society ( 7).
  * Russin . (2021) RussinZolfagharParkEtAl21Russin, JL., Zolfaghar, M., Park, SA., Boorman, E. O’Reilly, RC.  2021\.  Complementary Structure-Learning Neural Networks for Relational Reasoning Complementary Structure-Learning Neural Networks for Relational Reasoning.  Proceedings for the 43nd Annual Meeting of the Cognitive Science Society Proceedings for the 43nd Annual Meeting of the Cognitive Science Society ( 7).
  * Stachenfeld . (2017) StachenfeldBotvinickGershman17Stachenfeld, KL., Botvinick, MM. Gershman, SJ.  201711\.  The Hippocampus as a Predictive Map The hippocampus as a predictive map.  Nature Neuroscience20111643–1653.  10.1038/nn.4650
  * Summerfield . (2020) SummerfieldLuyckxSheahan20Summerfield, C., Luyckx, F. Sheahan, H.  202001\.  Structure Learning and the Posterior Parietal Cortex Structure learning and the posterior parietal cortex.  Progress in Neurobiology184101717.  10.1016/j.pneurobio.2019.101717
  * Ungerleider Mishkin (1982) UngerleiderMishkin82Ungerleider, LG. Mishkin, M.  198201\.  Two Cortical Visual Systems Two Cortical Visual Systems.  DJ. Ingle, MA. Goodale RJW. Mansfield (), The Analysis of Visual Behavior The Analysis of Visual Behavior ( 549–586).  Cambridge, MAMIT Press.
  * Webb . (2021) WebbSinhaCohen21Webb, TW., Sinha, I. Cohen, JD.  202103\.  Emergent Symbols through Binding in External Memory Emergent Symbols through Binding in External Memory.  ICLR 2021: Proceedings of the International Conference on Learning Representations.  http://arxiv.org/abs/2012.14601
  * Whittington . (2020) WhittingtonMullerMarkEtAl20Whittington, JCR., Muller, TH., Mark, S., Chen, G., Barry, C., Burgess, N. Behrens, TEJ.  202011\.  The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation The Tolman-Eichenbaum Machine: Unifying space and relational memory through generalization in the hippocampal formation.  Cell00.  10.1016/j.cell.2020.10.024