* Fountoulakis et al. [2021] N. Fountoulakis, P. van der Hoorn, T. Müller, and M. Schepers.  Clustering in a hyperbolic model of complex networks.  _Electron. J. Probab._ , 26:Paper No. 13, 132, 2021.  doi:10.1214/21-ejp583.  URL https://doi.org/10.1214/21-ejp583.
  * García-Pérez et al. [2016] G. García-Pérez, M. Boguñá, A. Allard, and M. Serrano.  The hidden hyperbolic geometry of international trade: World trade atlas 1870–2013.  _Scientific Reports_ , 6(1):1–10, 2016.
  * Gouéré [2008] J.-B. Gouéré.  Subcritical regimes in the Poisson Boolean model of continuum percolation.  _The Annals of Probability_ , 36(4):1209 – 1220, 2008.  doi:10.1214/07-AOP352.  URL https://doi.org/10.1214/07-AOP352.
  * Gracar et al. [2019] P. Gracar, A. Grauer, L. Lüchtrath, and P. Mörters.  The age-dependent random connection model.  _Queueing Syst._ , 93(3-4):309–331, 2019.  ISSN 0257-0130.  doi:10.1007/s11134-019-09625-y.  URL https://doi.org/10.1007/s11134-019-09625-y.
  * Gracar et al. [2022a] P. Gracar, M. Heydenreich, C. Mönch, and P. Mörters.  Recurrence versus transience for weight-dependent random connection models.  _Electronic Journal of Probability_ , 27:1 – 31, 2022a.  doi:10.1214/22-EJP748.  URL https://doi.org/10.1214/22-EJP748.
  * Gracar et al. [2022b] P. Gracar, L. Lüchtrath, and C. Mönch.  Finiteness of the percolation threshold for inhomogeneous long-range models in one dimension.  2022b.  URL https://arxiv.org/abs/2203.11966.
  * Hoff et al. [2002] P. D. Hoff, A. E. Raftery, and M. S. Handcock.  Latent space approaches to social network analysis.  _Journal of the american Statistical association_ , 97(460):1090–1098, 2002.
  * van der Hofstad [2021+] R. van der Hofstad.  Random Graphs and Complex Networks, Vol. 2 (in preparation).  2021+.  URL http://www.win.tue.nl/r̃hofstad/NotesRGCNII.pdf.
  * Iskhakov et al. [2018] L. Iskhakov, B. Kamiński, M. Mironov, P. Prałat, and L. Prokhorenkova.  Clustering properties of spatial preferential attachment model.  In A. Bonato, P. Prałat, and A. Raigorodskii, editors, _Algorithms and Models for the Web Graph_ , pages 30–43, Cham, 2018. Springer International Publishing.  ISBN 978-3-319-92871-5.
  * Jacob and Mörters [2015] E. Jacob and P. Mörters.  Spatial preferential attachment networks: power laws and clustering coefficients.  _Ann. Appl. Probab._ , 25(2):632–662, 2015.  ISSN 1050-5164.  doi:10.1214/14-AAP1006.  URL https://doi.org/10.1214/14-AAP1006.
  * Jorritsma and Lapinskas [2021] J. Jorritsma and J. Lapinskas.  Software for “Increasing efficacy of contact-tracing applications by user referrals and stricter quarantining”, Apr. 2021.  URL https://doi.org/10.5281/zenodo.4675115.
  * Jorritsma et al. [2023] J. Jorritsma, J. Komjáthy, and D. Mitsche.  Cluster-size decay in supercritical kernel-based spatial random graphs.  2023\.  URL https://arxiv.org/abs/2303.00724.
  * Komjáthy and Lodewijks [2020] J. Komjáthy and B. Lodewijks.  Explosion in weighted hyperbolic random graphs and geometric inhomogeneous random graphs.  _Stochastic Process. Appl._ , 130(3):1309–1367, 2020.  ISSN 0304-4149.  doi:10.1016/j.spa.2019.04.014.  URL https://doi.org/10.1016/j.spa.2019.04.014.
  * Krioukov et al. [2010a] D. Krioukov, F. Papadopoulos, M. Kitsak, A. Vahdat, and M. Boguñá.  Hyperbolic geometry of complex networks.  _Phys. Rev. E (3)_ , 82(3):036106, 18, 2010a.  ISSN 1539-3755.  doi:10.1103/PhysRevE.82.036106.  URL https://doi.org/10.1103/PhysRevE.82.036106.
  * Krioukov et al. [2010b] D. Krioukov, F. Papadopoulos, M. Kitsak, A. Vahdat, and M. Boguñá.  Hyperbolic geometry of complex networks.  _Phys. Rev. E_ , 82:036106, Sep 2010b.  doi:10.1103/PhysRevE.82.036106.  URL https://link.aps.org/doi/10.1103/PhysRevE.82.036106.
  * Krot and Ostroumova Prokhorenkova [2015] A. Krot and L. Ostroumova Prokhorenkova.  Local clustering coefficient in generalized preferential attachment models.  In _International Workshop on Algorithms and Models for the Web-Graph_ , pages 15–28. Springer, 2015.
  * Last and Penrose [2018] G. Last and M. Penrose.  _Lectures on the Poisson process_ , volume 7 of _Institute of Mathematical Statistics Textbooks_.  Cambridge University Press, Cambridge, 2018.  ISBN 978-1-107-45843-7; 978-1-107-08801-6.
  * Leskovec [2008] J. Leskovec.  _Dynamics of large networks_.  Carnegie Mellon University, 2008.
  * Meester and Roy [1996] R. Meester and R. Roy.  _Continuum Percolation_.  Cambridge Tracts in Mathematics. Cambridge University Press, 1996.  doi:10.1017/CBO9780511895357.
  * Michielan et al. [2022] R. Michielan, N. Litvak, and C. Stegehuis.  Detecting hyperbolic geometry in networks: Why triangles are not enough.  _Physical Review E_ , 106(5):054303, 2022.
  * Newman [2003] M. E. J. Newman.  Properties of highly clustered networks.  _Physical Review E_ , 68:026121, Aug 2003.  doi:10.1103/PhysRevE.68.026121.  URL https://link.aps.org/doi/10.1103/PhysRevE.68.026121.
  * Serrano and Boguñá [2006] M. A. Serrano and M. Boguñá.  Clustering in complex networks. i. general formalism.  _Physical Review E_ , 74:056114, Nov 2006.  doi:10.1103/PhysRevE.74.056114.  URL https://link.aps.org/doi/10.1103/PhysRevE.74.056114.
  * Stegehuis et al. [2019] C. Stegehuis, R. van der Hofstad, and J. S. H. van Leeuwaarden.  Scale-free network clustering in hyperbolic and other random graphs.  _Journal of Physics A: Mathematical and Theoretical_ , 52(29):295101, jun 2019.  doi:10.1088/1751-8121/ab2269.  URL https://doi.org/10.1088/1751-8121/ab2269.
  * van der Hofstad et al. [2023] R. van der Hofstad, P. van der Hoorn, and N. Maitra.  Local limits of spatial inhomogeneous random graphs.  _Advances in Applied Probability_ , page 1–48, 2023.  doi:10.1017/apr.2022.61.
  * van der Kolk et al. [2022] J. van der Kolk, M. Serrano, and M. Boguñá.  An anomalous topological phase transition in spatial random graphs.  _Communications Physics_ , 5(1):1–7, 2022.  doi:https://doi.org/10.1103/PhysRevE.82.036106.
  * Vázquez et al. [2002] A. Vázquez, R. Pastor-Satorras, and A. Vespignani.  Large-scale topological and dynamical properties of the internet.  _Physical Review E_ , 65(6):066130, 2002.  doi:https://doi.org/10.1103/PhysRevE.65.066130.


# Collective Human Opinions in Semantic Textual Similarity

Yuxia Wang♠ Shimin Tao♣ Ning Xie♣ Hao Yang♣
Timothy Baldwin♠♡ Karin Verspoor♠♢
$\spadesuit$ The University of Melbourne, Melbourne, Victoria, Australia
$\clubsuit$ Huawei TSC, Beijing, China $\heartsuit$ MBZUAI, Abu Dhabi, UAE
$\diamondsuit$RMIT University, Melbourne, Victoria, Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Despite the subjective nature of semantic textual similarity (STS) and
pervasive disagreements in STS annotation, existing benchmarks have used
averaged human ratings as gold standard. Averaging masks the true distribution
of human opinions on examples of low agreement, and prevents models from
capturing the semantic vagueness that the individual ratings represent. In
this work, we introduce USTS, the first Uncertainty-aware STS dataset with
$\sim$15,000 Chinese sentence pairs and 150,000 labels, to study collective
human opinions in STS. Analysis reveals that neither a scalar nor a single
Gaussian fits a set of observed judgements adequately. We further show that
current STS models cannot capture the variance caused by human disagreement on
individual instances, but rather reflect the predictive confidence over the
aggregate dataset.

##  1 Introduction

Semantic textual similarity (STS) is a fundamental natural language
understanding (NLU) task, involving the prediction of the degree of semantic
equivalence between two pieces of text (S1,S2). STS has been approached in
various ways, ranging from early efforts using string- or knowledge-based
measures and count-based co-occurrence models (Resnik, 1999; Barrón-Cedeño et
al., 2010; Matveeva et al., 2005), to modern neural networks.

Broadly speaking, the goal of the STS task is to train models to make a
similarity assessment that matches what a human would make. Gold-standard
scores are typically assigned by asking multiple raters to label a pair of
sentences and then taking the average (Agirre et al., 2012, 2013, 2014, 2015,
2016; Marelli et al., 2014; Soğancıoğlu et al., 2017; Wang et al., 2018). The
underlying assumption here is that there is a single “true” similarity score
between S1 and S2, and that this label can be approximated by averaging
multiple — possibly noisy — human ratings. While this assumption might be
reasonable in settings such as educational testing with well-defined knowledge
or norms (Trask and Trask, 1999), it is not the case for more subjective NLU
tasks.

Pavlick and Kwiatkowski (2019) show that in natural language inference (NLI),
disagreements often persist even if more ratings are collected or when the
amount of context provided to raters is increased. High disagreement has been
observed in a number of existing NLI datasets Nie et al. (2020). In STS,
concerns about inconsistent judgements have been raised, particularly for
difficult boundary cases in complex domains, where even expert annotators can
disagree about the “true” label (Wang et al., 2020; Olmin and Lindsten, 2022).
Identifying and discarding “noisy” labels during training can reduce
generalisation error (Wang et al., 2022a, b). We reexamine whether the
disagreement observed among raters should be attributed to “noise” and
resolved via dismissing, or should rather treated as an inherent quality of
the STS labels. Specifically, our primary contributions are:

  1. 1.

We develop USTS, the first Uncertainty-aware STS dataset with a total of
$\sim$15,000 Chinese sentence pairs and 150,000 labels. We study the human
assessments and investigate how best to integrate them into a gold label
across varying degrees of observed human disagreement.

  2. 2.

We show that state-of-the-art STS models cannot capture disagreement when
trained using a single averaged rating, and argue that STS evaluation should
incentivise models to predict distributions over human judgements, especially
for cases of low agreement.

  3. 3.

We discuss the practicalities of transferring labels across languages in
building a multilingual STS corpus, and present evidence to suggest that this
may be problematic in the continuous labelling space.

##  2 Background

###  2.1 Semantic Textual Similarity Task

Data Collection and Annotation: As STS requires a sentence pair, to construct
a dataset, ideally sentence pairs should be sampled to populate the spectrum
of differing degrees of semantic equivalence, which is a huge challenge. If
pairs of sentences are taken at random, the vast majority would be totally
unrelated, and only a very small fraction would have some degree of semantic
equivalence (Agirre et al., 2012). Accordingly, previous work has either
resorted to string similarity metrics (e.g. edit distance or bag-of-word
overlap) (Agirre et al., 2013, 2014, 2015, 2016; Soğancıoğlu et al., 2017;
Wang et al., 2018), or reused existing datasets from tasks related to STS,
such as paraphrasing based on news/video descriptions (Agirre et al., 2012)
and NLI (Marelli et al., 2014).

In terms of annotation, for general text (e.g. news, glosses, or image
descriptions), it has mostly been performed using crowdsourcing via platforms
such as Amazon Mechanical Turk with five crowd workers (Cer et al., 2017). For
knowledge-rich domains such as clinical and biomedical text, on the other
hand, a smaller number of expert annotators has been used, such as two
clinical experts for MedSTS (Wang et al., 2018). Raters are asked to assess
similarity independently on the basis of semantic equivalence using a
continuous value in range $[0,5]$. Then a gold label is computed by averaging
these human ratings.

No. 1 |  Low Human Disagreement
---|---
S1 |  Kenya Supreme Court upholds election result.
S2 |  Kenya SC upholds election result.
Old label |  5.0
New label |  $\mathcal{N}(\mu=4.9,\sigma=0.1)$
Annotations |  [4.5, 4.7, 4.8, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]
Prediction |  3.5
Reason |  Lack of knowledge of the correspondence between Supreme Court and SC.
No. 2 |  High Human Disagreement
S1 |  A man is carrying a canoe with a dog.
S2 |  A dog is carrying a man in a canoe.
Old label |  1.8
New label |  $\mathcal{N}(\mu=1.7,\sigma=1.0)$
Annotations |  [0.0, 0.3, 0.5, 0.5, 1.2, 1.5, 1.5, 1.8, 2.0, 2.0, 2.0, 2.0, 2.5, 3.5, 3.5]
Prediction |  4.3
Reason |  Uncertainty about the impact of key differences in event participants on instances of high lexical overlap
No. 3 |  High Human Disagreement
S1 |  Someone is grating a carrot.
S2 |  A woman is grating an orange food.
Old label |  2.5
New label |  $\mathcal{N}(\mu=2.4,\sigma=1.1)$
Annotations |  [0.5, 1.0, 1.0, 1.8, 1.8, 1.8, 2.0, 2.2, 2.5, 3.0, 3.0, 3.2, 3.5, 3.6, 4.5]
Prediction |  0.6
Reason |  Failure to associate carrot with orange food.

“

Table 1: Examples with varying levels of human disagreement from the STS-B
validation set. “Old label” = gold label of STS-B; “New label” = full
distribution aggregated by 15 new ratings; and “Prediction” = similarity score
predicted by SOTA STS model.222The individual annotations for STS-B are not
available, so we collected new ratings from 15 PhD NLPers. bert-base fine-
tuned on the STS-B training data ($r$=0.91) is used for prediction, same as
the one in Section 3.1.1 for selection.

Is averaging appropriate? Averaging has been the standard approach to
generating gold labels since Lee et al. (2005). However, this approach relies
on the assumption that there is a well-defined gold-standard interpretation +
score, and that any variance in independent ratings is arbitrary rather than
due to systematic differences in interpretation. An example of this effect can
be seen in case No. 1 in Table 2. In practice, however, high levels of
disagreement can be observed among annotators in different
domains.333$\sigma>0.5$ for 9% and 11% pairs in biomedical STS corpora:
BIOSSES and EBMSASS; inter-annotator agreement Cohen’s $\kappa$=0.60/0.67 for
two clinical datasets (Wang et al., 2020).

In such cases, a simple average fails to capture the latent distribution of
human opinions/interpretations, and masks the uncertain nature of subjective
assessments. With Nos. 2 and 3 in Table 2, e.g., the average scores $\mu$ of
1.7 and 2.4 do not convey the fact that the ratings vary substantially
($\sigma>1.0$). While the integrated score may reflect the average opinion, it
neither captures the majority viewpoint nor exposes the inherent disagreements
among raters. Put differently, not all average scores of a given value convey
the same information. Consider three scenarios that all average to 3.0:
(3,3,3,3,3)/5, (1,3.5,3.5,3.5,3.5)/5, and (2,4,2,4,3)/5. The inherent level of
human agreement varies greatly in these three cases.

Looking to the system predictions, the model prediction of 3.5 for No. 1 in
Table 2 is clearly incorrect, as it lies well outside the (tight) range of
human annotations in the range $[4.5,5.0]$. While the model prediction of 4.3
for No. 2 also lies outside the annotation range of $[0.0,3.5]$, it is closer
to an extremum, and there is much lower agreement here, suggesting that the
prediction is better than that for No. 1. No. 3 seems to be better again, as
the model prediction of 0.6 is both (just) within the annotation range of
$[0.5,4.5]$ and closer to the average for a similarly low-agreement instance.
Based on the standard evaluation methodology in STS research of calculating
the Pearson correlation over the mean rating, however, No. 1 would likely be
assessed as being a more accurate prediction than Nos. 2 or 3, based solely on
how close the scalar prediction is to the annotator mean. A more nuanced
evaluation should take into consideration the relative distribution of
annotator scores, and assuming a model which outputs a score distribution
rather than a simple scalar, the relative fit between the two. We return to
explore this question in Section 5.

Based on these observations, we firstly study how to aggregate a collection of
ratings into a representation which better reflects the ground truth, and
further go on to consider evaluation metrics which measure the fit between the
distribution of annotations and score distribution of a given model.

###  2.2 Human Disagreements in Annotations

Individual Annotation Uncertainty Past discussions of disagreement on STS have
mostly focused on uncertainty stemming from an individual annotator and the
noisiness of the data collection process. They tend to attribute an outlier
label to “inattentive” raters. This has led to the design of annotation
processes to control the reliability of individual ratings and achieve high
inter-annotator agreement (Wang et al., 2018). However, disagreements persist.
Inherent Disagreements Among Humans Studies in NLI have demonstrated that
disagreements among annotations are reproducible signals (Pavlick and
Kwiatkowski, 2019). It has also been acknowledged that disagreement is an
intrinsic property of subjective tasks (Nie et al., 2020; Wang et al., 2022c;
Plank, 2022).

Despite this, most work in STS still has attributed high levels of
disagreement to poor-quality data (Wang et al., 2022a), and has focused on
reducing the uncertainty in STS modelling and providing reliable predictions
(Wang et al., 2022b). Little attention has been paid to analysing the inherent
underlying variation in STS annotations on a continuous rating scale, or how
to fit the collective human opinions to a mathematical representation. Does a
real value, Gaussian distribution, Gaussian mixture model, or a more
complicated distribution most effectively approximate the latent truth?

The shortage of individual annotator labels in STS has been a critical
obstacle to in-depth analysis of disagreements among human judgements, since
only the averaged similarity scores are available to the public for almost all
STS datasets, apart from two small-scale biomedical benchmarks with 0.1k and
1k examples, respectively. To this end, we first construct a large-scale STS
corpus in this work with 4-19 annotators for each of almost 15k sentence
pairs. We focus on analysing disagreements among annotators instead of the
individual uncertainty, presuming that each individual rater is attentive
under a quality-controlled annotation process.

###  2.3 Chinese STS Corpus

Most progress on STS, driven by large-scale investment in datasets and
advances in pre-training, has centred around English.444English STS models
have achieved $r=0.91$, while for Chinese the best results are markedly lower
at $r=0.82$ for STS-B test. Efforts to build comparable datasets for other
languages have largely focused on (automatically) translating existing English
STS datasets Huertas-García et al. (2021); Yang et al. (2019). However, this
approach may come with biases (see Section 6). Our dataset is generated from
Chinese rather than English sources, and we employ native Chinese speakers as
annotators, producing the first large-scale Chinese STS dataset.555Apart from
translated STS-B, there are only two Chinese corpora related to STS: BQ Chen
et al. (2018) and LCQMC Liu et al. (2018) for paraphrase detection (binary).

##  3 Data Collection

We collected STS judgements from multiple annotators to estimate the
distribution, for sentence pairs drawn from three multilingual sources.
Sections 3.1 and 3.2 provide details of the collection, along with challenges
in the annotation and how we ensure data quality. All data and annotations are
available at https://github.com/yuxiaw/USTS.

###  3.1 Data Sources

The first step is to gather sentence pairs. In response to rapid rises in STS
performance and insights into the shortcomings of current models and
limitations of existing datasets, we create a new corpus that not only
incorporates inherent human disagreements in the gold label representation,
but also includes more challenging examples, on which state-of-the-art STS
models tend to make wrong predictions.
Common errors: our analysis over general STS-B and clinical N2C2-STS exposes
three major error types. More than half of errors lie in subsets where human
agreement is low. High uncertainty in STS labelling leads to pervasive
disagreement among human judgements.

Another is attributed to the lack of reasoning, as Nos. 1 and 3 in Table 2
reveal: (1) matching an abbreviation with its full name, e.g. Supreme Court to
SC; and (2) building connections between descriptions that are lexically
divergent but semantically related, e.g. carrot and orange food. The other is
the failure to distinguish pairs with high lexical overlap but opposite
meaning, due to word substitution or reordering.

However, these types of examples account for only a tiny proportion of
existing test sets and have minimal impact on results. Thus, our goal is to
gather more cases of high ambiguity, requiring reasoning abilities and more
semantic attention in annotation.

As our data sources, we use sentences from TED talks, and sentence pairs from
NLI and paraphrase corpora, as detailed below. The combined dataset contains
14,951 pairs, which we perform basic data cleaning over to remove repeated
punctuation marks (e.g. multiple quotation marks, dashes, or blank spaces).

####  3.1.1 TED-X

Compared to written texts such as essays, spoken texts are more spontaneous
and typically less formal Clark (2002). Without any contextual cues such as
prosody or multi-modality to help interpret utterances, readers may have
trouble understanding, especially for single sentences out of context Chafe
(1994), resulting in high uncertainty in labelling. We therefore choose TED
speech transcriptions to gather high-ambiguity examples.

##### Selecting Single Sentences

TED2020 contains a crawl of nearly 4000 TED and TED-X transcripts, translated
into more than 100 languages. Sentences are aligned to create a parallel
corpus Reimers and Gurevych (2020). We extracted 157,047 sentences for zh-cn
with character length ranging between 20 and 100, and aligned it with the
other 8 languages of en, de, es, fr, it, ja, ko, ru, and traditional zh.

##### Pairing by Retrieval

Sentence pairs generated by random sampling are prone to be semantically
distant. To avoid pairs with similarity scores overwhelmingly distributed in
the range $[0,1]$, we use embedding-based retrieval. For each sentence, we
search for the two most similar sentences based on faiss Johnson et al. (2017)
using the SimCSE sentence embedding of sup-simcse-bert-base-uncased Gao et al.
(2021), obtaining 155,659 pairs after deduplication.666Note that we base this
on the English versions of each sentence, due to the higher availability of
pre-trained language models and sentence encoders for English. That is, we use
(approximate) cosine similarity based on contextualised sentence embeddings
instead of the surface string-based measures of previous work to sample
sentence pairs. This is expected to find pairs with a higher level of semantic
overlap, rather than some minimal level of lexical match.

Score | Description
---|---
5 | The two sentences are completely equivalent, as they mean the same thing.
4 | The two sentences are mostly equivalent, but some unimportant details differ.
3 | The two sentences are roughly equivalent, but some important information differs/missing.
2 | The two sentences are not equivalent, but share some details.
1 | The two sentences are not equivalent, but are on the same topic.
0 | The two sentences are completely dissimilar.

Table 2: Similarity scores with descriptions Agirre et al. (2013).

##### Selecting Low-agreement Examples

To select what we expect to be examples with low agreement, we leverage the
observation that high-variance examples tend to be associated with low human
agreement (Nie et al., 2020). That is, we keep pairs with large predictive
variance, and predictions that differ greatly between two agents.

We use a bert-base-uncased-based STS model fine-tuned on the STS-B training
data for prediction. We obtain the mean $\mu$ and standard deviation $\sigma$
for each example from sub-networks based on MC-Dropout, where $\mu$ is re-
scaled to the same magnitude $[0,1]$ as the normalised $L_{2}$ using SimCSE
embedding $\mathbf{x}$, and $len_{word}(S_{en})$ is the word-level length of
the English sentence. We then select instances which satisfy the three
criteria: (1) $|\frac{1}{5}\mu-(1.0-L_{2}(\mathbf{x}_{1},\mathbf{x}_{2}))|\geq
0.25$; (2) $\sigma\geq 0.16$; and (3) $len_{word}(S_{en})\geq 12$.777We tuned
these threshold values empirically, until the majority of sampled instances
fell into the range $[1,3]$ — the score interval most associated with
ambiguous instances. This results in 9,462 sentence pairs.

####  3.1.2 XNLI

Though sentence pairs from SICK-R and UNLI (Chen et al., 2020) are annotated
with entailment and contradiction relations and also continuous labels, they
don’t specifically address semantic equivalence: the scores in SICK-R reflect
semantic relatedness rather than similarity, and in UNLI the annotators were
asked to estimate how likely the situation described in the hypothesis
sentence would be true given the premise.

We use sentence pairs from Cross-lingual NLI (XNLI: Conneau et al. (2018))
where there is label disagreement (which we hypothesise reflects ambiguity),
noting that the dataset was annotated for textual entailment in en, and
translated into 14 languages: fr, es, de, el, bg, ru, tr, ar, vi, th, zh, hi,
sw and ur. From the development (2,490) and test sets (5,010), we select
examples where there is not full annotation agreement among the five
annotators, resulting in 3,259 sentence pairs (1,097 dev and 2,162 test).

####  3.1.3 PAWS-X

We sample 2230 sentence pairs from PAWS-X (Yang et al., 2019) which are not
paraphrases but have high lexical overlap. Note that this is an extension of
PAWS (Zhang et al., 2019) to include six typologically-diverse languages: fr,
es, de, zh, ja and ko.

###  3.2 Annotation

We employ four professional human annotators (all Chinese native speakers) to
assign labels to the 14,951 Chinese sentence pairs in the first round, and an
additional 15 annotators to provide additional annotations for 6,051 examples
of low human agreement (as detailed below).

##### Annotation Guideline

Table 2 shows the 6-point ordinal similarity scale we use, plus definitions.

##### Quality Control

It is difficult to ensure that any divergences in annotations are more likely
due to task subjectivity or language ambiguity than inattentiveness. We
attempt to achieve this by not using crowdsourced workers, but instead
training up in-house professional annotators with expert-level knowledge in
Linguistics, and significant experience in data labelling. They were first
required to study the annotation guidelines and exemplars, and then asked to
annotate up to 15 instances of high-agreement pre-selected from the STS-B
training set. For each example, the annotation is regarded to be correct when
the difference between the assigned and gold-standard label is $<$0.5. Failing
this, the annotator is provided with the correct label and asked to annotated
another instance.

This procedure was iterated for three rounds to familiarize the annotators
with the task. On completion of the training, we only retain annotators who
achieve a cumulative accuracy of $\geq$75%.

###  3.3 Analysis of First-round Annotations

##### Dataset breakdown

Table 3 shows the breakdown of instances across the three component sets, as
well as the combined USTS dataset. In terms of average length (zh character
level), XNLI is the shortest on average (esp. for S2, the hypothesis),
followed by TED-X and PAWS-X.

##### Inter-annotator agreement

The average Pearson ($r$) and Spearman ($\rho$) correlation between the six
pairings of annotators, and standard deviation ($\sigma$) among the four
annotators, are $r=0.74$, $\rho=0.68$, $\sigma=0.47$. These numbers reflect
the fact that there is high disagreement for a substantial number of instances
in USTS, in line with the sampling criteria used to construct the dataset. As
such, aggregating ratings by averaging is not able to capture the true nature
of much of the data. Two questions naturally arise: (1) at what level of
variance does averaging noticeably bias the gold label? and (2) how should
annotations be aggregated to fit the latent truth most closely?

Source | TED-X | XNLI | PAWS-X | USTS
---|---|---|---|---
Amount |  |  |  |
raw | 9462 | 3259 | 2230 | 14951
$\sigma>0.5$ | 3458 | 1597 |  0996 | 6051
ratio | 36.5% | 49.0% | 44.7% | 40.5%
Length |  |  |  |
S1 | 39.0 | 34.0 | 43.5 | 38.6
S2 | 39.2 | 16.9 | 43.3 | 34.9
pair | 39.1 | 25.4 | 43.4 | 36.8
Raters |  |  |  |
$r$ | 0.48 | 0.61 | 0.49 | 0.74
$\rho$ | 0.50 | 0.58 | 0.41 | 0.68
$\sigma$ | 0.44 | 0.52 | 0.49 | 0.47
STSb-zh |  |  |  |
$r$ | 0.41 | 0.48 | 0.32 | 0.70
$\rho$ | 0.43 | 0.50 | 0.18 | 0.63
$\sigma$ | 0.21 | 0.22 | 0.19 | 0.21

Table 3: Details of the USTS dataset. “$r$” = Pearson’s correlation; “$\rho$”
= Spearman’s rank correlation; and “$\sigma$” = standard deviation

##### High vs. Low agreement

Figure 1 shows the first-round variance distribution, wherein $\sigma$ ranges
from 0.0 to 1.5, with 8,900 pairs being less than 0.5. It indicates that on
$\sim$60% examples, the assessments of four annotators fluctuate around the
average score in a smaller range (0.0–0.5 on average), while the judgements of
the remaining 6,051 pairs are spread out over a wider range (0.5–1.5).

We sample 100 examples and find that, when $\sigma\leq$ 0.5, generally more
than 10 out of 15 annotators highly agree with each other. This basically
satisfies the assumption that makes averaging less biased: individual ratings
do not vary significantly (Lee et al., 2005). While less than half annotators
reach consensus when $\sigma>$ 0.7, and less than 5 when $\sigma\geq$ 1.0
(referring back to our earlier examples in Table 2). Thus, we heuristically
regard $\sigma$=0.5 as a tipping point for distinguishing examples of low
($\sigma>0.5$) and high agreement ($\sigma\leq 0.5$).

Accordingly, we split the data into two subsets, reflecting the different
levels of disagreement: cases where $\sigma\leq 0.5$ are uncontroversial
(USTS-U); and cases where $\sigma>0.5$ are contentious (USTS-C).

##### Does the model agree with the annotators?

We take bert-base-chinese and fine-tune it on the Chinese STS-B training
data888Chinese STS-B has 5,231, 1,458 and 1,361 examples for training,
validation and test, respectively; see https://github.com/pluto-junzeng/CNSD
with a learning rate of 2e-5 for 3 epochs, obtaining $r{=}0.82$/$\rho{=}0.82$
on the validation set, and $r{=}0.80$/$\rho{=}0.79$ on the test set; we refer
to this model as “STSb-zh”. We compute $r$ and $\rho$ between the model
prediction and each of the four annotations, and present the average results
in Table 3.

Both $r$ and $\rho$ across TED-X, XNLI, and PAWS-X are below 0.5, with PAWS-X
being particularly bad with half of the pairs being predicted to be in the
range $[4,5]$. Predictions of USTS are primarily concentrated in the range
$[1,3]$, when majority annotations are in the range $[0,2]$

This suggests it is non-trivial for current models to perform well without
training on USTS, and models tend to over-assign high scores (Figure 1:
predictive $\sigma$ is $<0.3$ vs. annotator $\hat{\sigma}=0.47$). However, it
also leads us to consider whether the distribution estimated based on the four
annotators is adequate to generate a gold standard. To this end, we
investigate the question How does the collective distribution vary when
increasing the number of annotators, on cases of uncontroversial USTS-U and
contentious USTS-C?

Figure 1: Standard deviation distribution of the four first-stage annotators
(left) and model predictions (right).

###  3.4 Collective Distribution Analysis

We measure the distributional variation through (1) fluctuation of $\mu$ and
$\sigma$; and (2) distributional divergence between first-round and second-
round annotators.

Study design: we sample 100 instances from USTS-U and 100 from USTS-C, with a
ratio of 4:3:3 from TED-X, XNLI, and PAWS-X, resp. We then had another 15
qualified Chinese native annotators score the 200 Chinese sentence pairs.

Formally, the annotation matrix $A^{N\times M}$ represents a data set with $N$
examples annotated by $M$ annotators. In our setting, $N{=}100$ and $M{=}19$
for both USTS-U and USTS-C. We capture the variation of $\mu$ and $\sigma$
over 100 examples by averaging $\boldsymbol{\mu}$=mean(A[:,:i], axis=1) and
$\boldsymbol{\sigma}$=std(A[:,:i], axis=1), where $i$ ranges from 4 to 19,
incorporating the new ratings incrementally.

The collective distribution for the first-round annotation A[:,:4] is denoted
as $\mathbf{p}{=}\mathcal{N}(\boldsymbol{\mu_{1}},\boldsymbol{\sigma_{1}})$,
and $\mathbf{q}{=}\mathcal{N}(\boldsymbol{\mu_{2}},\boldsymbol{\sigma_{2}})$
for A[:,4:4+j] as we add new annotators. We observe the KL-Divergence$(p\|q)$
as we increase $j$.

Hypothesis: We hypothesise that the distribution will remain stable regardless
of the number of annotators on the uncontroversial USTS-U, but change
substantially on the contentious USTS-C.

Results: To plot the value of $\mu$ and $\sigma$ in the same figure, we re-
scale $\mu$ by subtracting $0.9$ in Figure 2. We find that with an increased
number of annotators, $\mu$ of USTS-U remains stable with minor perturbations,
while $\mu$ of USTS-C declines and steadily flattens out.

Figure 2: Average $\mu$ and $\sigma$ over 100 examples of USTS-U and USTS-C as
we incorporate new annotators.

On USTS-U, $\sigma$ ascends slowly and converges to 0.3. This matches our
expectation that increasing annotators will result in more variance. Yet it
still varies in the range $[0.1,0.3]$ due to the high certainty of the
uncontroversial examples. In contrast, $\sigma$ of USTS-C stays consistently
high, indicating that there are still strong disagreements even with more
annotators, because of the inherent ambiguity of contentious cases. It
fluctuates in a larger range of $[0.6,1.0]$, with a steeper drop. That is,
combining more ratings results in large variations in $\mu$ and $\sigma$ for
USTS-C, but less for USTS-U.

Therefore, the distribution obtained from four annotators is adequate for
uncontroversial examples, but insufficient for USTS-C: more annotators are
needed to gain a representative distribution.

##### How many annotators should be employed?

In Figure 2, $\mu$ and $\sigma$ of USTS-C vary substantially before $M{=}15$,
then stabilise. The trend of KL-Divergence in Table 4 demonstrates the same
phenomenon: KL declines as the number of annotators increases, with a
relatively small and stable divergence when $j>10$. Combining these two, we
employ 15 extra annotators to score the 6,051 cases for USTS-C in the second-
round annotation.

$j$ | 4 | 6 | 8 | 10 | 14 | 15
---|---|---|---|---|---|---
USTS-U |  04.26 | 2.58 | 0.98 | 1.03 | 0.91 | 0.93
USTS-C | 12.83 | 5.08 | 5.45 | 3.51 | 2.99 | 2.82

Table 4: KL-Divergence between the first-round distribution and the second-
round, for increasing $j$.

| # | Annotators | $\mu$ | $\sigma$ | $r$ | $\rho$
---|---|---|---|---|---|---
STS-B | 8,085 | 5 | 0.0–5.0 | – | – | –
USTS-U | 8,900 | 4 | 0.0–5.0 | 0.27 | 0.91 | 0.73
USTS-C | 6,051 | 19 | 0.2–4.4 | 0.56 | 0.72 | 0.63

Table 5: Statistical breakdown of STS-B (zh) and USTS-U/USTS-C; $\mu$= the
range of integrated score.

##### First-round vs. second-round:

Figure 3: Distribution of $\sigma$ (top) and $\mu_{1}-\mu_{2}$ (bottom) of the
first- and second-round annotation distribution.

We compare $\sigma$ and $\mu$ between the first-round (in green) and second-
round (in red) annotations in Figure 3 (top). The shape of the $\sigma$
distributions is very similar, but the green bars ($\sigma_{1}$) move towards
the right by 0.3 or so, with respect to the red bars ($\sigma_{2}$), leading
to the average $\hat{\sigma_{2}}$=0.42 $\ll$ $\hat{\sigma_{1}}$=0.76. This
indicates that the second-round distribution is more stable, with less overall
variance. Nonetheless, 87% of pairs exceed the average deviation of 0.27 for
USTS-U, reflecting the higher number of disagreements. Additionally, the
distribution of $\mu_{1}-\mu_{2}$ in Figure 3 (bottom) is close to a normal
distribution, within the range of $[-1,2]$. The majority are to the right of
zero, indicating that annotators in the first round tend to assign higher
scores than in the second, resulting in a larger $\mu$.

###  3.5 The Resulting Corpus

USTS-U vs. USTS-C The number of examples in USTS-U and USTS-C is 8,900 and
6,051, respectively, with largely comparable $\mu$ range of $[0,5]$ and
$[0.2,4.4]$ (see Table 5). USTS-U has a much smaller $\hat{\sigma}$ of 0.27
than USTS-C ($\hat{\sigma}{=}0.56$), consistent with their inherent
uncertainty level. Analogously, USTS-U has a higher correlation of $r{=}0.91$
among annotators, compared to $r{=}0.72$ for USTS-C.

##  4 Aggregation of Human Judgements

For the high-agreement cases of USTS-U, gold labels can be approximated by
aggregating multiple annotations into either a scalar or a single Gaussian
distribution. However, for low-agreement examples, how to aggregate the human
ratings remains an open question.

Are all distributions unimodal Gaussian? Though most distributions of human
assessments can be assumed to be sampled from an underlying (generative)
distribution defined by a single Gaussian, we observed judgements that a
unimodal Gaussian struggles to fit. The annotations of examples Nos. 2 and 3
in Figure 4 exhibit clear bi- or tri-modal distributions. How often, then, and
to what extent do multimodal distributions fit better?

Figure 4: Human judgement distributions of examples in Table 2, with uni-,
tri- and bi-modal Gaussian resp. The dotted black line shows the model fit
when using a single Gaussian; the shaded curve shows the model learned when
allowed to fit $k$ components of a GMM.

We answer this question by fitting human judgements using a Gaussian Mixture
Model (GMM), where the number of components is selected during training. This
means the model can still choose to fit the distribution with only one
Gaussian component where appropriate. If additional components yield a better
fit to the judgements, i.e. larger log likelihood is observed than using a
unimodal distribution, we consider the human judgements to exhibit a
multimodal distribution.

Experiments and Results We randomly split USTS-C into a training (4,051) and
test set (2,000), and use the training data to fit a GMM with: (1) one
component; or (2) the optimal number of components $k$. We compute the log
likelihood assigned to each example in the test set in Figure 5 (left), with
the unimodal results as the $x$-axis and multimodal Gaussian as the $y$-axis.
The majority of points fall on or above the diagonal line ($y=x$), with a
multimodal distribution outperforming a unimodal Gaussian distribution for 83%
of instances. However, does this suggest that most examples exhibit multiple
peaks?

Figure 5: Left: Log likelihood of test data under the single-component
Gaussian ($x$-axis) vs. the $k$-component GMM ($y$-axis). The darker the area,
the more the examples concentrate. Right: Weights of top-2 effective component
distribution.

Effective components: We count the effective components for each sentence pair
based on the weight assigned by the GMM in form of a probability for each
component. We see that, for 11.3% of pairs, there is a nontrivial second
component (weight$\geq 0.2$), and a third component on 3 pairs. Rarely are
there more than three components with significant weights (see Table 6).
Moreover, we find that the weight of the dominant component mostly (87%)
distributes over 0.8, and that the weight of the second effective component
scatters across the range 0.25–0.5 (the right of Figure 5). This reveals that
the GMM does not frequently use more than one effective component, with much
lower weights on the second or third components. The majority of held-out
human judgements fit a unimodal distribution well.

| Testing |  | Train
---|---|---|---
K | amount | prop(%) | $\hat{\sigma}$ |  | amount | prop | $\hat{\sigma}$
1 | 1772 | 88.6 | 0.55 |  | 3755 | 92.7 | 0.48
2 | 225 | 11.3 | 0.63 |  | 294 | 7.3 | 0.50
3 | 3 | 0.0 | 0.39 |  | 2 | 0.0 | 0.66

Table 6: The amount and averaged standard deviation $\hat{\sigma}$ of examples
with $k=\\{1,2,3\\}$ effective components of human judgements distributions in
the training and test splits.

Gold Labels: Given that a minority of instances in USTS-C are bimodally
distributed, and that even for these instances, the weight on the second
components is low, we conservatively use a single Gaussian to aggregate human
judgements for all cases in this work.

##  5 Analysis of Model Predictions

Most STS models predict a pointwise similarity score rather than of a
distribution over values. Wang et al. (2022b) estimated the uncertainty for
continuous labels by MC-Dropout and Gaussian process regression (GPR).
However, due to the lack of gold distributions, they only evaluate outputs
using expected calibration error (ECE) and negative log-probability density
(NLPD), assessing the predictive reliability. It’s unknown whether these
uncertainty-aware models mimic human disagreements, i.e. the predicted
deviation reflects the variance of human judgements.

To explore this, we experiment over USTS and incorporate distributional
divergence (i.e. Kullback-Leibler Divergence; “KL”) into the evaluation, to
observe the fit between the distribution of collective human judgements and
the model predictive probability. We also examine the ability of different
models to capture the averaged score for low-agreement cases, and whether a
well-calibrated model fits the distribution of annotations better.

##### Evaluation Metrics:

For singular values, STS accuracy is generally evaluated with Pearson
correlation ($r$) and Spearman rank correlation ($\rho$), measuring the linear
correlation between model outputs and the average annotation, the degree of
monotonicity under ranking, respectively.

For uncertainty-aware outputs, ECE and NLPD can be used to assess model
reliability in the absence of gold distributions. ECE measures whether the
estimated predictive confidence is aligned with the empirical correctness
likelihoods. A well-calibrated model should be less confident on erroneous
predictions and more confident on correct ones. NLPD penalises over-confidence
more strongly through logarithmic scaling, favouring under-confident models.

|  | STS-B |  | USTS-U |  | USTS-C
---|---|---|---|---|---|---
| Model |  $r$ $\uparrow$ |  $\rho$ $\uparrow$ |  ECE $\downarrow$ |  NLPD $\downarrow$ |  |  $r$ $\uparrow$ |  $\rho$ $\uparrow$ |  ECE $\downarrow$ |  NLPD $\downarrow$ |  KL $\downarrow$ |  |  $r$ $\uparrow$ |  $\rho$ $\uparrow$ |  ECE $\downarrow$ |  NLPD $\downarrow$ |  KL $\downarrow$
SBERT-NLI |  |  |  |  |  |  |  |  |  |  |  |
(1) | SBERT-cosine | 0.714 | 0.718 | n/a | n/a |  | 0.597 | 0.383 | n/a | n/a | n/a |  | 0.572 | 0.442 | n/a | n/a | n/a
(2) | SBERT-GPR | 0.741 | 0.743 | 0.001 | 0.532 |  | 0.709 | 0.433 | 0.020 | 0.033 | 2.233 |  | 0.656 | 0.455 | 0.139 |  $-$0.09 | 0.576
Domain-specific |  |  |  |  |  |  |  |  |  |  |  |
(3) | BERT-lr | 0.808 | 0.804 | n/a | n/a |  | 0.855 | 0.700 | n/a | n/a | n/a |  | 0.806 | 0.707 | n/a | n/a | n/a
BERT-lr-MC | 0.811 | 0.805 | 0.167 | 4.709 |  | 0.856 | 0.703 | 0.054 | 1.079 | 4.587 |  | 0.809 | 0.708 | 0.046 | 0.442 | 6.073
SBERT-cosine | 0.779 | 0.781 | n/a | n/a |  | 0.661 | 0.387 | n/a | n/a | n/a |  | 0.596 | 0.460 | n/a | n/a | n/a
SBERT-GPR | 0.780 | 0.782 | 0.053 | 0.917 |  | 0.683 | 0.388 | 0.137 | 0.651 | 3.050 |  | 0.606 | 0.444 | 0.415 | 0.717 | 0.950
Domain-generalised |  |  |  |  |  |  |  |  |  |  |  |
(4) | BERT-lr | 0.815 | 0.813 | n/a | n/a |  | 0.860 | 0.692 | n/a | n/a | n/a |  | 0.835 | 0.768 | n/a | n/a | n/a
BERT-lr-MC | 0.814 | 0.811 | 0.179 | 5.865 |  | 0.861 | 0.697 | 0.060 | 0.898 | 4.434 |  | 0.838 | 0.774 | 0.278 | 0.702 | 5.401
SBERT-cosine | 0.772 | 0.772 | n/a | n/a |  | 0.686 | 0.435 | n/a | n/a | n/a |  | 0.670 | 0.523 | n/a | n/a | n/a
SBERT-GPR | 0.772 | 0.775 | 0.017 | 0.645 |  | 0.707 | 0.433 | 0.098 | 0.268 | 2.578 |  | 0.674 | 0.497 | 0.157 | $-$0.04 | 0.955
Cross-domain |  |  |  |  |  |  |  |  |  |  |  |
(5) | BERT-lr | 0.675 | 0.667 | n/a | n/a |  | 0.754 | 0.650 | n/a | n/a | n/a |  | 0.725 | 0.676 | n/a | n/a | n/a
BERT-lr-MC | 0.678 | 0.671 | 0.348 | 12.90 |  | 0.755 | 0.695 | 1.296 | 10.55 | 13.95 |  | 0.729 | 0.687 | 1.298 | 8.956 | 12.62
SBERT-cosine | 0.695 | 0.692 | n/a | n/a |  | 0.647 | 0.449 | n/a | n/a | n/a |  | 0.606 | 0.481 | n/a | n/a | n/a
SBERT-GPR | 0.726 | 0.726 | 0.001 | 0.555 |  | 0.723 | 0.481 | 0.020 | 0.012 | 2.215 |  | 0.675 | 0.494 | 0.148 |  $-$0.11 | 0.555

Table 7: Test set correlation ($r$/$\rho$), ECE, NLPD and KL using end-to-end
(BERT) and pipeline (SBERT), over STS-B, USTS-U and USTS-C, under five
settings. The bold number is the best result for BERT, and the underlined
number is that for SBERT.

###  5.1 Models and Setup

##### BERT with Two-layer MLP:

The hidden state $\mathbf{h}$ from the last-layer hidden state of BERT CLS
token Devlin et al. (2019) is passed through a two-layer MLP with $\tanh$
activation function. We refer to this model as BERT-lr when making
deterministic predictions, and BERT-lr-MC when using MC-Dropout Gal and
Ghahramani (2016) for uncertainty estimation.

##### SBERT with GPR:

In contrast with end-to-end training, sparse GPR is applied to estimate
distributions, taking encoded sentences from Sentence-BERT (SBERT: Reimers and
Gurevych (2019)) as input. We also calculate the cosine similarity between S1
and S2 using SBERT, as a non-Bayesian counterpart.

##### Setup:

bert-base-chinese is used with input format [CLS] S1 [SEP] S2 [SEP] for text
pair $(S1,S2)$, implemented based on the hugging-face Transformer framework.
We fine-tune SBERT separately over each STS corpus based on bert-base-chinese-
nli, using the same configuration as the original paper. We use the
concatenation of the embeddings $u\oplus v$, along with their absolute
difference $|u-v|$ and element-wise multiplication $v\times t$ to represent a
sentence pair, implemented in pyro.999https://pyro.ai/

We evaluate STS-B, USTS-U, and USTS-C under five training settings, as
presented in Table 7:

  1. 1.

Zero-shot: SBERT with no tuning;

  2. 2.

GPR trained on sbert-nli;

  3. 3.

Domain-specific: fine-tuned on each dataset separately;

  4. 4.

Domain-generalised: fine-tuned using the three datasets combined;

  5. 5.

Cross-domain: train with STS-B training data for USTS-U and USTS-C, and with
USTS for STS-B.

###  5.2 Results and Analysis

USTS is challenging. In setting (1) of Table 7, purely depending on pre-
trained semantic representation and cosine similarity, correlations over
USTS-U and USTS-C are much lower than STS-B. This suggests that USTS is a
challenging dataset, but can be learned. USTS-U in particular achieves large
improvements in performance after domain-specific training in experiments
(3)–(4).

Critical differences exist between model outputs and human annotations. The
models can capture average opinion, resulting in reasonable $r$/$\rho$ between
the predicted target value and the averaged annotations. However, they cannot
capture the variance of human opinions. To quantify how well the predicted
variance $\sigma_{M}$ captures the variance $\sigma_{H}$ of human judgements,
we analyse the outputs of the top-2 settings: BERT-lr-MC from setting (4) and
SBERT-GPR from setting (5), for USTS-U and USTS-C. We compute the correlation
$r$ and $\rho$ between $\boldsymbol{\sigma}_{M}$ and $\boldsymbol{\sigma}_{H}$
in Table 8, and visualise the $\sigma_{M}$ with increasing human disagreement
in Figure 6.

|  USTS-U ($\hat{\sigma_{H}}$=0.26) |  |  USTS-C ($\hat{\sigma_{H}}$=0.56)
---|---|---|---
Model | $r$ | $\rho$ | $\hat{\sigma_{M}}$ |  | $r$ | $\rho$ | $\hat{\sigma_{M}}$
(4) BERT-lr-MC | 0.13 | 0.12 | 0.19 |  | 0.24 | 0.23 | 0.20
(5) SBERT-GPR |  $-$0.07 |  $-$0.06 | 0.67 |  |  $-$0.05 |  $-$0.06 | 0.54

Table 8: Test set correlation between the predicted variance and collective
human variance. Figure 6: Predicted variance $\sigma_{M}$ ($y$-axis) with
increasing human disagreement ($x$-axis). The Red and blue triangles = USTS-U
and USTS-C from experiment setting (4) in Table 7, orange and green circles =
USTS-U and USTS-C from experiment setting (5), and black line is $y=x$. USTS-U
disperses at the left of the $x$-axis and low-agreement USTS-C scatters to the
right.

There is no apparent correlation between $\sigma_{M}$ and $\sigma_{H}$. A
given model displays similar deviation $\sigma_{M}$ regardless of the relative
amount of human disagreement. Different models concentrate on different parts
of the spectrum, e.g. BERT-lr-MC is distributed in the range $[0.1,0.2]$ while
SBERT-GPR is distributed in the range $[0.5,0.7]$, and neither follows the
line of $y=x$. This suggests that the uncertainty captured by current models
is not the uncertainty underlying human disagreements. Rather it may reflect
the model’s predictive confidence on the data set as a whole. This finding is
not surprising since none of the models are optimised to capture collective
human opinions, but suggests an important direction for future improvement.

Being trustworthy is orthogonal to being accurate. We see that ECE and NLPD do
not mirror the results for $r$/$\rho$ and distributional divergence KL. This
implies the ability required to improve model reliability differs from that
required to perform accurately, regardless of whether a target value or a
target distribution is predicted.

Low human-agreement USTS is detrimental to training sentence embeddings.
Comparing the performance of experiment settings (2) and (5) in Table 7,
tuning SBERT on USTS hurts results over STS-B across the board, while training
on STS-B benefits both USTS-U and USTS-C. We speculate that the examples in
USTS with larger annotator variance are more ambiguous than STS-B. Forcing
networks to learn from high-ambiguity signals may inhibit generalisation,
resulting in worse representations.

Discussion For instances of high disagreement, neither a scalar nor a single
Gaussian fits a set of observed judgements adequately. As a direction for
future work, we suggest exploring the direct estimation of individual ratings
(e.g. by few-shot prompt-based prediction) and evaluating against the raw
collective opinions. This could circumvent the ineffective training and
evaluation caused by aggregation.

##  6 Multilingual USTS

Figure 7: Scatter plot of 100 examples sampled from USTS-U (top) and USTS-C
(bottom) annotated by Chinese native, English native, professional translator
(PT), overseas students (OS), and general users (GU). We plot ($\mu$,$\sigma$)
as coordinate points.

Before extending USTS into a multilingual benchmark, we question the validity
of previous approaches involving direct transfer of annotations collected for
one language to other languages Liu et al. (2021); Yang et al. (2019). This
strategy assumes that the nuanced semantics of the component sentences is not
changed under translation, and hence the label will be identical. To test
whether this assumption is reasonable, we analyse the impact of language on
the annotations, and discuss whether such ratings are transferable across
languages.

Specifically, we establish whether the label distribution varies based on
language, and how annotator proficiency affects the distribution given the
same text.

##### Collecting Labels

Taking English as a pivot language, we employ native English speakers (“NT”)
and bilingual raters whose mother language is Mandarin Chinese, including 5
professional translators (“PT”), 5 overseas students (“OS”), and 5 general
users (“GU”). Each annotator assigns labels to 100 examples sampled from each
of USTS-U and USTS-C (the same data set used in Section 3.4), which have been
manually post-edited by professional translators to ensure content alignment.

##### Results

We average the KL between collective distributions drawn from 19 raters given
zh text, and 5 native English speakers (NT) given en text. Table 9 shows there
is not a substantial distributional divergence. Differences decline further as
annotations of the other three groups of bilingual raters are incorporated.

en-rater | NT | +PT | +OS | +GU
---|---|---|---|---
USTS-U | 0.69 | 0.67 | 0.53 | 0.38
USTS-C | 0.94 | 0.78 | 0.73 | 0.68
Table 9: KL-divergence of labels as ratings from less proficient language
speakers are incorporated.

Detailed analysis of distributions across each of these groups (Figure 7)
reveals that the language of the text affects the distribution of human
opinions. On both USTS-U and USTS-C, the distribution differs substantially
between native Chinese speakers and native English speakers when given zh and
en sentence pairs, respectively. While the zh annotations cluster in the lower
$\sigma$ region, those for en are dispersed across a large $\sigma$ span.

Figure 7 also shows that the distribution of professional translators mirrors
that of English natives, while general users differ substantially from both
these groups, but are similar to native-speaker Chinese annotators who are
given zh text. We suspect that translators make judgements based on the
meaning of en text directly, but general users may use translation tools to
translate en text back to zh to support their understanding, meaning they are
in fact rating a Chinese text pair. Intermediate-level overseas students may
mix strategies and thus are somewhere in between these two extremes.

Discussion The differences we observe may be attributed to bias introduced
during manual translation. Each sentence in a pair is translated separately,
so while a source pair may have lexical overlap, this may not carry over under
independent translation. We examine this effect by calculating the word
overlap similarity as Eq (1) for zh/en pairs, where $T_{1}$ and $T_{2}$ are
whitespace-tokenised words for English and based on the jieba segment tool for
Chinese. We calculate string similarity as:

$\displaystyle Sim=\frac{len(T_{1}\cap
T_{2})+1}{max(len(T_{1}),len(T_{2}))+1}$ (1)

As detailed in Table 10, the lexical overlap similarity for en and zh is
similar for USTS-U and USTS-C, suggesting that inconsistencies under
translation are not a primary cause of the observed discrepancy.

lan | USTS-U | USTS-C | USTS
---|---|---|---
zh | 0.42 | 0.45 | 0.44
en | 0.40 | 0.43 | 0.41
Table 10: Lexical similarity between en and zh pairs sampled from USTS-U,
USTS-C, and the combination of the two.

##### In summary

The language of the text impacts the distribution of human judgements. In our
analysis, English results in higher-uncertainty labelling than Chinese, for
both uncontroversial and contentious cases. This suggests that the previous
assumption that labels remain identical across languages as long as the
meaning of the text is kept the same, is potentially problematic, even though
pairwise lexical overlap remains similar.

##  7 Discussion

We focus on the STS task in this work. However, the methods we propose can be
transferred to other subjective textual regression tasks, such as sentiment
analysis (SA) rating and machine translation quality estimation in the format
of direct assessment (DA). Similar findings stemming from task subjectivity
may be relevant to other types of NLP tasks relying on human annotation. High
disagreement among annotators may occur due to ambiguous labelling, where it
is challenging to compile guidelines that are widely accepted and consistently
interpreted by all individual annotators.

In practice, it may be difficult to estimate the distribution of human
annotations in instances where multiple annotators are difficult to source,
such as occurs in clinical and biomedical STS due to the need for highly
specialised knowledge. Transfer learning, which relies on patterns learned
from general-purpose USTS, provides a means to predict such a distribution, if
noisily. We propose to explore the direct estimation of individual ratings by
in-context learning based on large language models (LLMs), e.g. GPT-3 (Brown
et al., 2020) and ChatGPT.101010https://openai.com/blog/chatgpt LLMs are able
to perform in-context learn — perform a new task via inference alone, by
conditioning on a few labelled pairs as part of the input Min et al. (2022).

ChatGPT appears to be highly effective at style transfer and tailoring of
content to specific audiences such as five-year old children or domain
experts, through learning about language style and tone from interactional
data and individual preferences. This allows it to generate more personalised
responses (Aljanabi et al., 2023). Deshpande et al. (2023) show that assigning
ChatGPT a persona through the parameter system-role, such as a bad/horrible
person, can increase the toxicity of generated outputs up to sixfold.

Additionally, Schick and Schütze (2021) show that generative LLMs can be used
to automatically generate labelled STS datasets using targeted instructions.
This data can be utilised to improve the quality of sentence embeddings.
Together, these imply that LLMs may have utility in generating personalised
semantic similarity assessments, based on annotator meta data (e.g. age,
educational background, or domain expertise).

Simulating variation in judgements between individual annotators using
synthetic personalised ratings could mitigate ineffective training and
evaluation caused by aggregation, given that neither a scalar nor a single
Gaussian fits the set of observed judgements adequately for instances of high
disagreement.

##  8 Conclusion

We presented the first uncertainty-aware STS corpus, consisting of 15k Chinese
examples with more than 150k annotations. The dataset is intended to promote
the development of STS systems from the perspective of capturing inherent
disagreements in STS labelling, and establish less biased and more nuanced
gold labels when large variances exist among individual ratings.

We additionally examine the models’ ability to capture the averaged opinion
and the distribution of collective human judgements. Results show that the
uncertainty captured by current models is not explained by the semantic
uncertainty that results in disagreements among humans. Rather, it tends to
reflect the predictive confidence over the whole data set. We also found that
the text language and language proficiency of annotators affect labelling
consistency.

## Acknowledgements

We thank the anonymous reviewers and editors for their helpful comments,
Yanqing Zhao, Samuel Luke Winfield D’Arcy, Yimeng Chen, Minghan Wang in Huawei
TSC and NLP Group colleagues in The University of Melbourne for various
discussions. Yuxia Wang is supported by scholarships from The University of
Melbourne and China Scholarship Council (CSC).

## References

  * Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015.  Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability.  In _Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015_ , pages 252–263. The Association for Computer Linguistics.
  * Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014\.  Semeval-2014 task 10: Multilingual semantic textual similarity.  In _Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014_ , pages 81–91. The Association for Computer Linguistics.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016.  Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016_ , pages 497–511. The Association for Computer Linguistics.
  * Agirre et al. (2013) Eneko Agirre, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013.  *SEM 2013 shared task: Semantic textual similarity.  In _Proceedings of the Second Joint Conference on Lexical and Computational Semantics_ , pages 32–43. Association for Computational Linguistics.
  * Agirre et al. (2012) Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation_ , pages 385–393. Association for Computational Linguistics.
  * Aljanabi et al. (2023) Mohammad Aljanabi et al. 2023.  Chatgpt: Future directions and open possibilities.  _Mesopotamian Journal of CyberSecurity_ , 2023:16–17.
  * Barrón-Cedeño et al. (2010) Alberto Barrón-Cedeño, Paolo Rosso, Eneko Agirre, and Gorka Labaka. 2010.  Plagiarism detection across distant language pairs.  In _Proceedings of the 23rd International Conference on Computational Linguistics_ , pages 37–45. Tsinghua University Press.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada.
  * Chafe (1994) Wallace Chafe. 1994.  _Discourse, consciousness, and time: The flow and displacement of conscious experience in speaking and writing_.  University of Chicago Press.
  * Chen et al. (2018) Jing Chen, Qingcai Chen, Xin Liu, Haijun Yang, Daohe Lu, and Buzhou Tang. 2018.  The BQ corpus: A large-scale domain-specific Chinese corpus for sentence semantic equivalence identification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4946–4951, Brussels, Belgium. Association for Computational Linguistics.
  * Chen et al. (2020) Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020.  Uncertain natural language inference.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8772–8779, Online. Association for Computational Linguistics.
  * Clark (2002) Herbert H Clark. 2002.  Speaking in time.  _Speech communication_ , 36(1-2):5–13.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Deshpande et al. (2023) Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023.  Toxicity in chatgpt: Analyzing persona-assigned language models.  _arXiv preprint arXiv:2304.05335_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.  In _International Conference on Machine Learning_ , pages 1050–1059.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Huertas-García et al. (2021) Álvaro Huertas-García, Javier Huertas-Tato, Alejandro Martín, and David Camacho. 2021.  Countering misinformation through semantic-aware multilingual models.  In _Intelligent Data Engineering and Automated Learning – IDEAL 2021_ , pages 312–323, Cham. Springer International Publishing.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.  Billion-scale similarity search with GPUs.  _arXiv preprint arXiv:1702.08734_.
  * Lee et al. (2005) Michael D Lee, Brandon Pincombe, and Matthew Welsh. 2005.  An empirical evaluation of models of text document similarity.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_.
  * Liu et al. (2021) Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021.  Learning domain-specialised representations for cross-lingual biomedical entity linking.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 565–574, Online. Association for Computational Linguistics.
  * Liu et al. (2018) Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018.  LCQMC:a large-scale Chinese question matching corpus.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1952–1962, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Marelli et al. (2014) Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014.  SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 1–8, Dublin, Ireland. Association for Computational Linguistics.
  * Matveeva et al. (2005) Irina Matveeva, G Levow, Ayman Farahat, and Christian Royer. 2005.  Generalized latent semantic analysis for term representation.  In _Proc. of RANLP_ , page 149.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020.  What can we learn from collective human opinions on natural language inference data? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9131–9143, Online. Association for Computational Linguistics.
  * Olmin and Lindsten (2022) Amanda Olmin and Fredrik Lindsten. 2022.  Robustness and reliability when training with noisy labels.  In _International Conference on Artificial Intelligence and Statistics_ , pages 922–942. PMLR.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Plank (2022) Barbara Plank. 2022.  The “problem” of human label variation: On ground truth in data, modeling and evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Resnik (1999) Philip Resnik. 1999.  Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language.  _Journal of Artificial Intelligence Research_ , 11:95–130.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Generating datasets with pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6943–6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Soğancıoğlu et al. (2017) Gizem Soğancıoğlu, Hakime Öztürk, and Arzucan Özgür. 2017.  BIOSSES: a semantic sentence similarity estimation system for the biomedical domain.  _Bioinformatics_ , 33(14):i49–i58.
  * Trask and Trask (1999) Robert Lawrence Trask and Robert Lawrence Trask. 1999.  _Key concepts in language and linguistics_.  Psychology Press.
  * Wang et al. (2018) Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2018.  MedSTS: a resource for clinical semantic textual similarity.  _Language Resources and Evaluation_ , pages 1–16.
  * Wang et al. (2020) Yanshan Wang, Sunyang Fu, Feichen Shen, Sam Henry, Ozlem Uzuner, and Hongfang Liu. 2020.  The 2019 n2c2/ohnlp track on clinical semantic textual similarity: Overview.  _JMIR Med Inform_ , 8(11).
  * Wang et al. (2022a) Yuxia Wang, Timothy Baldwin, and Karin Verspoor. 2022a.  Noisy label regularisation for textual regression.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4228–4240, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Wang et al. (2022b) Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. 2022b.  Uncertainty estimation and reduction of pre-trained models for text regression.  _Transactions of the Association for Computational Linguistics_ , 10:1–17.
  * Wang et al. (2022c) Yuxia Wang, Minghan Wang, Yimeng Chen, Shimin Tao, Jiaxin Guo, Chang Su, Min Zhang, and Hao Yang. 2022c.  Capture human disagreement distributions by calibrated networks for natural language inference.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1524–1535, Dublin, Ireland. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.


# Zen and the Science of Pattern Identification:
An Inquiry into Bayesian Skepticism

Deborah A. Striegel,1 Damian Wojtowicz,2 Teresa M. Przytycka,2 Vipul Periwal1∗

1Laboratory of Biological Modeling, NIDDK,
2Computational Biology Branch, NCBI, NLM,
The National Institutes of Health, Bethesda MD 20892, USA

∗To whom correspondence should be addressed<EMAIL_ADDRESS>
> Finding patterns in data is one of the most challenging open questions in
> information science. The number of possible relationships scales
> combinatorially with the size of the dataset, overwhelming the exponential
> increase in availability of computational resources. Physical insights have
> been instrumental in developing efficient computational heuristics. Using
> quantum field theory methods and rethinking three centuries of Bayesian
> inference, we formulated the problem in terms of finding landscapes of
> patterns and solved this problem exactly. The generality of our calculus is
> illustrated by applying it to handwritten digit images and to finding
> structural features in proteins from sequence alignments without any
> presumptions about model priors suited to specific datasets. Landscapes of
> patterns can be uncovered on a desktop computer in minutes.

Technological advances have made large-scale data collection and storage
increasingly practical. The problems we face today can often be reduced to
deducing a pattern of relationships in data, in science and elsewhere.
However, super-exponential combinatorial growth in the number of possible
relationships as the size of a datum increases hampers our ability to find
relationships in large datasets. Thus, the development of new techniques for
the discovery of relationships is critical to progress.

Here we propose a framework based on physical heuristics to overcome this
barrier to finding the set of fundamental interactions in the data, which we
call a pattern. Physical intuitions in computation have an illustrious
history. Thirty years ago, simulated annealing revolutionized optimization[1].
More recently, quantum computation has taken center stage[2]. Statistical
physics has been applied to data analysis but usually in the form of
approximations[3]. The data itself has unknown relationships and
approximations such as high temperature expansions, weak-coupling expansions,
mean field theory, and similar techniques[5] introduce further uncontrolled
uncertainties.

We sought an exact solution for finding patterns in data without presuming
anything about the presence of patterns. Without any loss of generality, we
consider datasets consisting of families of sequences of symbols and model
such families with the traditional weight matrices where $p_{iB}$ is the
probability of symbol $B$ at sequence position $i.$ Datasets that contain a
pattern have a non-random weight matrix. Specifically, we wanted to directly
compute the relationships between sequence positions and the common
deformations that relate different sequences in a family, which we define as
the landscape of the pattern. We developed an approach, Bayesian Skepticism
(BS), grounded in quantum field theory[4] that leads to an exact solution of
such pattern landscapes. Our derivation stands on three conceptual legs:

  1. 1.

A skeptical approach to modeling the data with a model prior probability
distribution $\exp(MK(p|M))$ favoring random models. This is analogous to a
null hypothesis, but with a continuous degree of skepticism $M$ in this
assumption. This analytic continuation is similar to dimensional
regularization in quantum field theory[4].

  2. 2.

A reversal of Bayesian model selection: For a given random model, is there a
weighting of data points that would make data frequencies consistent with
model probabilities?

  3. 3.

A method that solves the maximization of model and data likelihoods exactly,
giving the weight of each datum and simultaneously uncovering the most likely
model for the dataset.

The end result of our derivation[19] is a simple iteration of elementary
algebraic steps: With $M\times K(p|M)$ the skeptical prior log probability of
$p_{iA},$ the observed weighted frequencies are updated as

$f_{iA}^{(n+1)}\leftarrow\sum_{\rm sequences}\sigma_{i,A}({\rm
sequence})\exp\bigg{(}M\sum_{j,B}\sigma_{j,B}({\rm sequence})[\partial
K/\partial p_{jB}|_{p=f^{(n)}}-{\rm const.}]\bigg{)}$ (1)

where $(n)$ is the iteration count, and $\sigma_{i,A}$(sequence) is 1 if the
sequence has symbol $A$ at position $i$ and 0 otherwise, and the constant
ensures that the weights over all sequences sum to 1. Since we have been able
to express the iterative update using simple algebra, it is easy to find the
fixed point[19]. We can then find the probability that a given string of
symbols belongs to this family[19], and thus find the allowed deformations of
the family. These are the correlated symbol probability changes such that the
probability of the altered model does not change significantly from that of
the fixed point model. These deformations are the flat directions in the
pattern landscape.

We applied these concepts to the MNIST hand-written digit database[6]. This
database consists of 60,000 gray-scale images of numbers $0$ through $9.$ We
reduced these $28\times 28$ pixel format images to an unordered sequence of
symbols by associating symbols with each of the 16 possible combinations of
black and white pixels in a $2\times 2$ square (Fig. 1A). Each digit was thus
associated with a family of about 6000 strings of symbols, each string of
length 729 symbols (Fig. 1B,C) Our skepticism assumption, that the symbols
occur randomly, suggests that the prior probability of a model of such
sequences should be the generalization of the binomial coin-tossing
probability distribution to a 16-sided coin toss independently for each
position. The multinomial distribution at each position $i,$

$\exp(MK(p_{iA}|M,q_{A}))=M!\prod_{C}{q_{C}^{Mp_{iC}}/{(Mp_{iC})!}}$ (2)

gives the probability that the 16 symbols $A$ will appear $M\times p_{iA}$
times at position $i$ in a series of $M$ coin-tosses. We analytically
continued this formula to small non-integer positive values of $M$ using the
properties of the Gamma function. Typically, we will set $M=0.005$ in this
paper, and keep only sequence positions with less than 95% conservation.
Qualitative results do not depend on either $M$ changed to 0.007 or
conservation cutoff changed to 99% [19]. $q_{A}$ is the frequency of symbol
$A$ over all positions in the sequences corresponding to a digit.

The fixed-point of the iteration in eq. 1 can be used to find the flat
directions. Changing the model along a flat direction induces entropy changes
at each position in the sequence. By visualizing this entropy change[19], we
observe that some positions in the sequence become more specialized and lose
entropy and others gain entropy as the probabilities change along a flat
direction. The three flattest directions for each digit are shown in Fig. 1D,
and have obvious geometric meanings. One can deduce that most people are
right-handed by noting the slant towards the right. These flat directions are
global in character though the analysis made no use of the ordering of the
sequence or of continuity. They are correlated changes in symbol frequencies
that are consistent with the functional constraints of representing the digit.

The mutual information between sequence positions, computed from the pattern
landscape effective potential by simultaneous perturbing symbol frequencies at
two different positions, could recover structural connections between
different sequence positions, enabling reconstruction of the original image
much like a jig-saw puzzle. We evaluated pair-wise interaction information
(PII, defined in [19]) between all pairs of sequence positions and then
restricted to positions with high enough symbol entropy. The entire digit can
be recovered and shows the presence of long-range interactions that encode
functional relations between non-adjacent positions (Fig. 2). Replacing the
symbol entropy with other measures of positional information did not change
these results.

We turn now to sequences of symbols that do come with a particular order, the
amino-acid sequences of families of proteins. First, we show how pattern
landscapes can be used to distinguish functionally different subclasses of
homologous proteins. The NTPase Maf_Ham1 domain is shared by two subfamilies
of proteins with a structurally similar nucleotide binding cleft but altered
conserved residue location and nature[15]. The landscape computed from a
subset of Subfamily 1 assigned a lower probability to sequences of Subfamily 2
compared to a test set of Subfamily 1(Fig. 3A) and vice versa. Similar results
are found in the case of digits.

Next, we demonstrate a relation between pattern landscapes and structural
properties. The intimin/invasin family of virulence factors are produced by
Gram-negative bacteria. The structure of these proteins was recently
solved[16], and we computed the landscape from 146 aligned sequences. The
actual landscape is much flatter than the landscape of column-wise randomly
permuted sequences (Fig. 3B), as a small eigenvalue corresponds to a larger
radius of curvature around the most likely model. Following steps analogous to
the digit example, we computed the entropy changes induced by the most
significant deformation (Fig. 3 C-E), revealing that this deformation
corresponds to specializing either the side chains projecting towards the
outside of the barrel or those projecting towards the inside. Investigating
PII between sequence positions (Fig. 3F) as we lower the threshold for PII, we
find disconnected sets of clusters that coalesce into a connected graph (Fig.
3G). We mapped this minimal single cluster of PII interacting residues on the
structure revealing juxtaposition suggesting allostery as well as predicting
longer range interactions (Fig. 3H).

More subtle structural features in proteins can also be detected in the
landscapes of sequence alignments. The dihydrofolate reductase (DHFR) enzyme
has been studied for its conformational changes related to catalytic
activity[18]. We computed PII for the sequences[20](Fig. 4E) and the
deformations in the landscape. Fig. 4 B-D shows the structural elements that
are detected by these deformations. The interacting cluster for DHFR deduced
from PII is the whole protein (Fig. 4F) with the strongest interactions around
the folate binding pocket. [13] derived a residue-residue interaction matrix
by computing side-chain folding-unfolding dynamics for the E. coli DHFR
protein. This matrix is similar to the PII matrix (Fig. 4E) computed from
sequences produced by evolutionary dynamics indicating that evolutionary
history has tested side-chain compatibility.

A pioneering approach to detecting structural features in DHFR and some other
proteins from sequence data, based on random matrix theory[7,8,9] and
analogies with finance, weights a binary approximation to the residue-residue
correlation matrix with the derivative of the large $M$ limit of the binomial
distribution. Eigenvectors of the absolute values of this weighted matrix have
been associated with allosteric interactions[8]. The features we find capture
regions of structural variation consistent with function as shown by mapping
them to known protein structures (Fig. 4B-D,G) and the positioning suggests
allostery as well.

Explicit statistical physics-type energy models of residue interactions that
have been extensively studied in various approximations[10,11,12]. These works
compute contact maps for proteins with the help of additional secondary
structure predictions. The major distinction is that BS is exact and has a
consistent expansion to all orders in interactions[19], with a geometric
interpretation in terms of the curvature of the landscape around the most
likely model. The use of a skeptical model prior in BS is the key to these
properties. While high PII values are likely to be associated with residue
pairs that are spatially close, tertiary structure proximity is not
necessarily the case. In the two-dimensional digit examples (Fig. 2), PII
links between positions with high enough symbol entropy are geometrically
meaningful but do not imply two-dimensional proximity. Understanding protein
structure through clusters in graphs has been considered[14] though the graphs
had a different origin.

Our method is applicable to a wide spectrum of dataset sizes. We used about
6000 sequences for each digit, but for the intimin/invasin and the two
subfamilies of the Maf_Ham1 proteins we computed landscapes with only 146, 67
and 70 sequences, respectively. Our results are robust to choices of
sequences. For DHFR, while we used 1382 sequences from OrthoDB[21] without any
curation in our analysis, we also obtained the 417 sequences used in [7,8] and
found that, for the positions present in both alignments, there were no
striking differences between PII computed from the two alignments.

To compare and contrast BS with common Bayesian approaches, Expectation
Maximization (EM) [22] is an iterative algorithm for finding the maximum
likelihood parameters of model or latent variables that maximize the
probability of the dataset. EM requires no model prior for its interleaving of
expectation and maximization steps. Maximum Entropy (MaxEnt) model priors
attempt to be maximally unbiased or uninformative. BS has no maximization step
in its iteration and cannot work without the skeptical model prior that
explicitly precludes correlations, thus going beyond MaxEnt. Crucially, BS is
self-limiting in how large the skepticism parameter $M$ can become because, by
construction, the pattern landscape is convex[19]. A smaller dataset begets a
more restricted landscape. This is evidenced in the reduced radius of
curvature associated with larger eigenvalues (Fig. 4A) for DHFR for the
alignment used in [7,8] compared to [20].

In summary, BS provides a simple exact solution to computing patterns in data.
It is conceptually transparent with an explicit derivation and a venerable
interpretation[4] in terms of a landscape of models. The skeptical approach
introduced here of assuming a model prior that explicitly favors the lack of a
pattern as a tool to deduce the actual pattern landscape from the data is
broadly applicable, and completely side-steps questions of the biological
relevance of model priors. We have shown that it can be used to compute
interactions between sequence positions exactly and effectively. Of particular
interest is the holistic character of the deformations we find: No single
position can capture the ‘8-ness’ of a sequence of symbols associated with the
digit $8$, yet symbol entropy changes in flat directions capture the slant
differences and other flourishes in handwriting. Molecular dynamics (MD)
simulations of protein activity are hampered by the computational difficulty
of sampling transient high-energy states. These motions are predicted to occur
in micro-second to milli-second ranges, can be functionally important, and are
difficult to model with atomic-scale MD[17]. The discovery of structural
elements in protein structures from sequence alignments using BS may help
overcome some of these difficulties in MD by suggesting structural elements
capable of independent motion[17]. Further, BS can directly extract networks
of interactions from data exploring the role of heterogeneity and variation in
biological systems[23]. The protean character of BS is well-suited to diverse
applications in science.

Acknowledgements: K. Reynolds and R. Ranganathan shared alignments of
proteins. This work was supported by the Intramural Research Program of the
National Institutes of Health, NIDDK and NLM.

References and Notes

  1. 1.

S. Kirkpatrick, C.D. Gelatt, M.P. Vecchi, Science 220, 671-680 (1983).

  2. 2.

P. Shor, SIAM J. Comput. 26, 1484-1509 (1997).

  3. 3.

S.V. Buldyrev, N.V. Dokholyan, A.L. Goldberger, S. Havlin, C.-K. Peng, H.E.
Stanley and G.M. Viswanathan, Physica A 249, 430-438 (1998).

  4. 4.

J. Zinn-Justin, Quantum Field Theory and Critical Phenomena (Oxford Univ.
Press, New York, 2002).

  5. 5.

L.D. Landau and E.M. Lifschitz, Statistical Physics (Pergamon Press, Oxford,
1980).

  6. 6.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Proc. of the IEEE 86,
2278-2324 (1998); MNIST database available at
http://yann.lecun.com/exdb/mnist/

  7. 7.

N. Halabi, O. Rivoire, S. Leibler, and R. Ranganathan, Cell 138, 774-786
(2009).

  8. 8.

K.A. Reynolds, R.N. McLaughlin, and R. Ranganathan, Cell 147, 1564-1575
(2011).

  9. 9.

R.N. McLaughlin, F.J. Poelwijk, A. Raman, W.S. Gosal, and R. Ranganathan,
Nature 138, 138-142 (2012).

  10. 10.

D.S. Marks, T.A. Hopf, and C. Sander, Nat. Biotechnol. 30, 1072-80 (2012).

  11. 11.

F. Morcos, A. Pagnani, B. Lunt, A. Bertolino, D.S. Marks, C. Sander, R.
Zecchina, J.N. Onuchic, T. Hwa, and M. Weigt, Proc Natl Acad Sci U S A 108,
E1293-301 (2011).

  12. 12.

D.S. Marks, L.J. Colwell, R. Sheridan, T.A. Hopf, A. Pagnani, R. Zecchina, and
C. Sander, PLoS ONE 6, e28766 (2011).

  13. 13.

H. Pan, J.C. Lee, and V.J. Hilser, Proc Natl Acad Sci U S A 97, 12020-5
(2000).

  14. 14.

A. Sukhwal, M. Bhattacharyya and S. Vishveshwara, Acta Cryst. D 67, 429-439
(2011).

  15. 15.

S. Chakrabarti, S.H. Bryant, and A.R. Panchencko, J. Mol. Biol. 373, 801-810
(2007).

  16. 16.

J.W. Fairman, N. Dautin, D. Wojtowicz, W. Liu, N. Noinaj,T.J. Barnard, E.
Udho, T.M. Przytycka, V. Cherezov and S.K. Buchanan, Structure 20, 1233-1243
(2012).

  17. 17.

B.K. Ho and D.A. Agard, PLoS Comp. Bio. 5, e1000343 (2009).

  18. 18.

J.R. Schnell, H.J. Dyson and P.E. Wright, Annu. Rev. Biophys. Biomol. Struct.
33, 119-140 (2004).

  19. 19.

The derivation and supplementary supporting figures are available as
supplementary text S1.

  20. 20.

DHFR alignment from OrthoDB, supplementary text S2.

  21. 21.

R.M. Waterhouse, E.M. Zdobnov, F. Tegenfeldt, J. Li, and E.V. Kriventseva,
Nucl. Acids Res. 41, D358-365 (2013); http://cegg.unige.ch/orthodb6.

  22. 22.

A.P. Dempster, N.M. Laird and D.B. Rubin, J. Royal Stat. Soc. Ser. B 39, 1-38
(1977).

  23. 23.

S.J. Altschuler and L.F. Wu, Cell 141, 559-563 (2010).

Figure 1. Sequence representation of hand-written digits. A. The sixteen
possible configurations of black and white pixels that correspond to distinct
letters. B. The grid of 28$\times$28 squares superimposed on a specific
example of the digit 5. C. The linear sequence of 729 letters obtained in this
manner is shown arranged in a $27\times 27$ square to exhibit the digit.
Overlapping $2\times 2$ squares were mapped to the letters associated with the
configurations in A. D. Symbol entropy changes in the three flattest
directions in the pattern landscapes of all 10 digits, relative to the most
likely model, with 25 example images for each digit.

Figure 2. Interactions between sequence positions for digits. First column:
PII between position pairs represented as a heat map for each digit. Second
column: PII values vs. distance in the two-dimensional image. All black PII
values (PII $>-0.77$) are shown along with a sampling of the other regions (1%
and 0.5% of PII values from blue and brown, and orange and red regions,
respectively) Inset: Histogram of all PII values (log (base 10) scale on
$y$-axis). Third column: PII values plotted as edges on a graph on the digit
image. Edges correspond to points observed in column 2. The heat map
background color scale represents symbol entropy at each position. Fourth
column: Edges consist of the strongest PII-value pairs for each position that
also have high symbol entropy. Most long-range interactions are removed by
this cutoff, with the remaining interactions of clear geometric meaning. Fifth
column: PII values above the symbol entropy threshold vs. two-dimensional
distance. All black PII values (PII $>-0.77$) are shown along with a sampling
of the other regions (1% and 0.5% of PII values from blue and brown, and
orange and red regions, respectively) Inset: Histogram of all PII values above
threshold (log (base 10) scale on $y$-axis).

Figure 3. Applications to Maf$\\_$Ham1 and intimin/invasin proteins. A. Upper
panel: Probability of Maf$\\_$Ham1 sequences in the pattern landscape of
Subfamily 1. Sequence index is plotted on the $x$-axis and $-\log$ Probability
on the $y$-axis. The first 12 sequences are test sequences from Subfamily 1
not used in the computation of the pattern landscape and the rest are all the
sequences in Subfamily 2. Only the linear term in the pattern probability
$M\Gamma(p)$ between positions with above-median symbol entropy is used in
this computation, consistent with the results shown in Fig. 2. Lower panel:
Similar probabilities with the roles of Subfamily 1and Subfamily 2 reversed.
$M=0.01$ for these probabilities. B. First 20 eigenvalues plotted in
increasing order for intimin/invasin. The second eigenvalue is almost twice as
large as the lowest eigenvalue, suggesting that the landscape has only one
major flat direction. Inset shows eigenvalues from two instances of column-
wise randomized intimin/invasin sequences showing much larger eigenvalues, and
no corresponding isolated flattest direction. C,D,E. Front, cut-through and
back views of the intimin/invasin beta barrel. Red positions are losing symbol
entropy and blue positions are gaining entropy, relative to the most likely
model (or vice-versa due to linearity). The symbol entropy changes are
consistent with beta-barrel shear and this consistency is absent in column-
wise randomized sequences[19]. F. PII heat map between all pairs of sequence
positions. G. Graph of correlated residues with edges corresponding to PII
values above a minimal threshold and nodes for only positions with high enough
symbol entropy. The minimal threshold PII cutoff is obtained by lowering a
cutoff until the graph has only one component. Edges are colored according to
PII value. H. The resulting correlated residues shown on the structure,
colored according to their strongest PII link. As in the digit examples (Fig.
2), the graph is not restricted to tertiary structure proximity.

Figure 4. Identification of structural features from flat directions for DHFR.
A. Similar pattern of eigenvalues in ascending order for DHFR, for DHFR
(metazoa, fungi, bacteria) sequences from OrthoDB[21] and for DHFR sequences
used in [7] obtained from R. Rangananthan and K. Reynolds. Inset: Eigenvalues
for column-wise randomizations are much higher than the eigenvalues for the
actual sequences. B. The flattest direction in the DHFR pattern landscape
picks out structural changes in the folate binding pocket. Symbol entropy
increases (blue) in the folate binding pocket and decreases in the surface
residues (red), or vice versa, relative to the most likely model. C: The
strongest association of the second flat direction is with the G-H loop which
plays a stabilizing role in interactions with the Met20 loop[18]. D: The
junction between the $\alpha 4$ helix and the $\beta 6$ sheet is associated
with the third flat direction. B,C,D Insets: Top to bottom: Specific
structural changes in E. coli, chicken and human DHFR associated with these
flat directions. E. PII heat map for DHFR. F. Graph of correlated residues.
For DHFR, all residues appear in the graph, consistent with known conformation
changes associated with ligand binding. Edges are colored according to PII
value. G. DHFR structure residues colored according to their strongest PII
link value. The strongest PII interaction is shown with side-chains.

empty