# The Causal News Corpus: Annotating Causal Relations in Event Sentences from
News

###### Abstract

Despite the importance of understanding causality, corpora addressing causal
relations are limited. There is a discrepancy between existing annotation
guidelines of event causality and conventional causality corpora that focus
more on linguistics. Many guidelines restrict themselves to include only
explicit relations or clause-based arguments. Therefore, we propose an
annotation schema for event causality that addresses these concerns. We
annotated 3,559 event sentences from protest event news with labels on whether
it contains causal relations or not. Our corpus is known as the Causal News
Corpus (CNC). A neural network built upon a state-of-the-art pre-trained
language model performed well with 81.20% F1 score on test set, and 83.46% in
5-folds cross-validation. CNC is transferable across two external corpora:
CausalTimeBank (CTB) and Penn Discourse Treebank (PDTB). Leveraging each of
these external datasets for training, we achieved up to approximately 64% F1
on the CNC test set without additional fine-tuning. CNC also served as an
effective training and pre-training dataset for the two external corpora.
Lastly, we demonstrate the difficulty of our task to the layman in a crowd-
sourced annotation exercise. Our annotated corpus is publicly available,
providing a valuable resource for causal text mining researchers.

Keywords: causality, event causality, text mining, natural language
understanding

The Causal News Corpus: Annotating Causal Relations in Event Sentences from
News

|  Fiona Anting Tan1, Ali Hürriyetoğlu2, Tommaso Caselli3, Nelleke Oostdijk4,
---
Tadashi Nomoto5, Hansi Hettiarachchi6, Iqra Ameer7, Onur Uca8,
Farhana Ferdousi Liza9, Tiancheng Hu10
1 Institute of Data Science, National University of Singapore, Singapore, 2
Koc University, Turkey,
3 Rijksuniversiteit Groningen, Netherlands, 4 Radboud University, Netherlands,
5 National Institute of Japanese Literature, Japan, 6 Birmingham City
University, United Kingdom,
7 Centro de Investigación en Computación, Instituto Politécnico Nacional,
Mexico,
8 Department of Sociology, Mersin University, Turkey, 9 University of East
Anglia, United Kingdom,
10 ETH Zürich, Switzerland
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Causality is a core cognitive concept and appears in many natural language
processing (NLP) works that aim to tackle inference and understanding [Jo et
al. (2021, Dunietz et al. (2020, Feder et al. (2021]. Generally, a causal
relation is a semantic relationship between two arguments known as cause and
effect, in which the occurrence of one (cause argument) causes the occurrence
of the other (effect argument) [Barik et al. (2016].

Figure 1 depicts examples of sentences expressing causality, and ones that do
not. Notice that causality can be expressed in various ways: The first causal
example is signaled by the explicit causal marker _“due to”_ while the second
example has causality indicated by alternative lexicalizations such as
_“created”_. In the last example, an implicit causal relation exists between
the verbless clause and the matrix clause, conveying that the dissatisfaction
with the package leads to the workers’ sit-in. For sentences without
causality, they must be missing either a cause or effect argument, or both.

Figure 1: Annotated examples from Causal News Corpus. Causes are in pink,
Effects in green and Signals in yellow. Note that both Cause and Effect spans
must be present within one and the same sentence for us to mark it as
_Causal_.

Causal Question Answering and Generation applications [Dalal et al. (2021,
Hassanzadeh et al. (2019, Stasaski et al. (2021] are some immediate downstream
Natural Language Understanding (NLU) applications popular in NLP today.
Despite the importance of identifying causality in text, datasets are limited
[Asghar (2016, Xu et al. (2020, Tan et al. (2021, Yang et al. (2021] and
oftentimes, different researchers craft their datasets with different rules,
leaving users with no proper way to compare models across datasets.

Our work is directed at annotating parts of the multilingual protest news
detection dataset [Hürriyetoğlu et al. (2021a, Hürriyetoğlu et al. (2021b,
Hürriyetoğlu (2021] for event causality. Our contributions111Our corpus and
model scripts are available online at
https://github.com/tanfiona/CausalNewsCorpus. are as follows:

  * •

We created the Causal News Corpus (CNC), which is a corpus of event sentences
annotated with binary labels indicating whether each sentence contains a
causal relationship or not222Annotation of the corpus is an ongoing effort: We
also aim to add information like cause-effect-signal span markings and causal
types, and these additional information will also be made public once
available..

  * •

We showed that a neural network built on a state-of-the-art pre-trained
language model could predict the causality labels with 81.20% F1, 77.81%
Accuracy, and 54.52% Matthews Correlation Coefficient on test set.

  * •

We crafted annotation guidelines that align with existing event causality and
linguistic schemes. Our experiments show that our dataset is indeed compatible
with other causality corpora.

  * •

We observed that the layman struggles to replicate the annotations by our
experts in a crowd-sourced annotation exercise, demonstrating the difficulty
of our task.

The rest of the paper is organized as follows: Section 2 reviews the
literature on event causality. Section 3 describes the compilation and
annotation of the corpus. Section 4 describes the baselines we created for
causal event sentence classification and the experiments that were conducted
on CNC and two other datasets. Section 5 outlines experiences from a crowd-
sourcing annotation exercise. Finally, Section 6 concludes the paper and
discusses potential avenues for future work.

## 2\. Related Work

The extraction of causality from text remains a significant challenge because
semantic understanding of the context and world knowledge is oftentimes
needed. In computational linguistics, a lot of effort has been put into
extracting causal knowledge from text automatically [Blanco et al. (2008, Do
et al. (2011, Kontos and Sidiropoulou (1991, Riaz and Girju (2013].

Previous work has focused on event causality, creating corpora like
CausalTimeBank (CTB) [Mirza et al. (2014] from news, CaTeRS [Mostafazadeh et
al. (2016] from short stories and EventStoryLine [Caselli and Vossen (2017]
from online news articles. Such corpora include rich event features like event
class attributes and temporal expressions in TimeML annotation format.
However, these corpora are typically limited in size. For example,
EventStoryLine (ESL) has 1,770 causal event pairs, CaTeRS has 488 causal
links, while CTB has 318 causal event pairs. Our work contributes to this
strand of literature by providing more data: We have identified 1,957 causal
event sentences, each of which can contain multiple event pairs.

There is also a discrepancy between such event causality corpora and other
causality corpora. Penn Discourse Treebank (PDTB) [Prasad et al. (2008, Webber
et al. (2019, Prasad et al. (2006] is a corpus that annotates semantic
relations (including causal relations) between clauses, expressed either
explicitly or implicitly. This corpus is large (e.g. PDTB-3 has over $7,000$
examples for the Contingency.Cause sense alone) and is potentially useful for
training an accurate event sentence classifier. Therefore, we believe that it
will be beneficial to align the annotation guidelines of these two corpora
types. We did so by constructing event causality annotation guidelines using
linguistic rules adapted from PDTB-3. However, our corpus differs from PDTB-3
by focusing on event sentences and accepting more fine-grained arguments that
do not necessarily form a clause. This approach of including more varied
constructions of causality is similar to the work by BECauSE 2.0 [Dunietz et
al. (2017].

When discussing causality in text, many corpora only focus on explicit
relations [Girju and Moldovan (2002, Dunietz et al. (2017]. An explicit causal
relation refers to an example where the cause-and-effect relation is expressed
by means of a clear connective or some other causal signal. An example was
presented earlier in the first sentence of Figure 1. Rule-based approaches
[Girju and Moldovan (2002, Khoo et al. (2000, Sakaji et al. (2008] have been
shown to work well on such constructions. For example, researchers have used
such approaches to semi-automatically curate datasets for cause-effect related
tasks [Cao et al. (2016, Heindorf et al. (2020, Stasaski et al. (2021].
However, in the real world, implicit relations are more common but more
challenging to identify [Hidey and McKeown (2016]. Therefore, in our work, we
do not limit ourselves to a pre-defined list of connectives but rather,
include causal examples in more varied linguistic constructions for more
realistic use cases. With respect to this, we differ from CTB, which only
annotated explicit causal examples.

## 3\. Compilation & Annotation

### 3.1. Data Source

CNC builds on the datasets featured in a series of workshops directed at
mining socio-political events from news articles: Automated Extraction of
Socio-political Events from News (AESPEN) in 2020 [Hürriyetoğlu et al. (2020b,
Hürriyetoğlu et al. (2020a] and Challenges and Applications of Automated
Extraction of Socio-political Events from Text (CASE) in 2021 [Hürriyetoğlu et
al. (2021a, Hürriyetoğlu et al. (2021b, Hürriyetoğlu (2021]. The data is based
on randomly sampled articles from multiple sources and periods, and all
annotations were performed by two annotators, adjudicated and spot-checked by
a supervisor, and corrected further semi-automatically [Hürriyetoğlu et al.
(2021c, Yörük et al. (2021]. In total, 869 news documents and 3,559 English
sentences were available. We annotated all 3,559 sentences in this paper. We
utilized the dataset from CASE 2021 directly and some sentences were not well
processed. We noticed that a few examples were actually comprised of more than
one sentence and titles were still included in the texts. In our next data
release, we will redo the parsing of these examples.

### 3.2. Guidelines

CNC includes annotations for binary causal event classification in sentences;
We labeled sentences to be _Causal_ or _Non-causal_. Sentences must contain at
least a pair of events, defined as as things that happen or occur, or states
that are valid, following TimeML [Saurı et al. (2006]. For causality, we
utilized the definition of Contingency from PDTB-3, which assigns this label
for samples where “one argument provides the reason, explanation or
justification for the situation described by the other” [Webber et al. (2019].
The following four senses were recognized as _Causal_ :

  * •

Cause: Arg1 and Arg2 are causally influenced but not in a conditional
relation. Example connectives are _“because”_ and _“since”_.

  * •

Purpose: One Arg describes the goal while the other Arg describes the action
undertaken to achieve the goal. Example connectives are _“in order to”_ and
_“so that”_.

  * •

Condition: One Arg presents an unrealized situation, that when realized, would
lead to the situation described by the other Arg. Example connectives are
_“if”_ and _“as long as”_.

  * •

Negative-Condition: One Arg presents an unrealized situation, that if it does
not occur, would lead to the situation described by the other Arg. Example
connectives are _“till”_ and _“unless”_.

Since our work is event-based, the senses that provided reasons for the
speaker to utter a speech, or the hearer to have a belief (i.e. +SpeechAct or
+Belief types) were treated as _Non-causal_.

A major difference between our approach and PDTB is that we permitted non-
clausal elements such as phrases in our annotation scheme as arguments as long
as these spans meet our definition of event. PDTB is a discourse bank that
restricts its argument spans to discourse relations. As noted by ?), causal
spans of multiple linguistic forms are excluded from PDTB. For instance, in
PDTB, verb signals like _“caused”_ are not annotated in the sentence
_“Speculation about Coniston has caused the stock to rebound from a low of
$145.”_. Consequently, PDTB also did not annotate the spans before and after
_“caused”_ as Cause and Effect arguments. Given our focus on news events, more
expressions of causality are required for proper interpretation. Therefore, on
top of PDTB-approved clauses, we allowed for the following to be recognized as
an argument: noun phrases (including nominalizations)333Some examples of noun
phrases describing event arguments: _“the fire”_ and _“the protest in May”_.,
and verb phrases (including ones that fall outside of a coordinated structure,
provided that the corresponding span of its relation is a noun phrase that
would complete the verb phrase to form a clause). In general, these
relaxations of rules allowed us to annotate more fine-grained arguments,
including ones within clauses.

Sentence | Causality Tests | Label
---|---|---
| Why? |  Temporal Order | Counterfact. |  Ontological Asymmetry | Linguistic |
The protests spread to 15 other towns and resulted in two deaths and the destruction of property. | ✓ | ✓ | ✓ | ✓ | ✓ | _Causal_
Chale was allegedly chased by a group of about 30 people and was hacked to death with pangas, axes and spears. | ✗ | ✓ | ✗ | ✓ | ✗ |  _Not_ _Causal_
The strike will continue till our demands are conceded. | ✓ | ✓ | ✓ | ✓ | ✓ |  _Causal_ (Neg. Cond.)
Table 1: Examples illustrating the applications of the Tests for Causality
(Grivaz, 2010; Dunietz et al., 2017). Cause in pink, Effect in green;
potential Cause in gray. Signals are not marked.

To ground our annotations, we utilized the five tests for causality based on
the work by ?) and ?). In Table 1, we report some examples to illustrate the
applications of the causality tests and the resulting classification labels.
Additional details and examples are provided in Appendix A.

##### Five Tests for Causality

  1. 1.

Why: The example is not causal if the reader is unable to construct a “Why”
question regarding the Effect.

  2. 2.

Temporal order: The example is not causal if the Cause does not precede the
Effect in time.

  3. 3.

Counterfactual: The example is not causal if the Effect is equally likely to
occur or not occur without the Cause.

  4. 4.

Ontological asymmetry: The example is not causal if the reader can readily
swap the Cause and Effect claims in place.

  5. 5.

Linguistic: The example is likely to be causal if it can be rephrased into “X
causes Y” or “Due to X, Y.”

#####  Cause types

Quoting the PDTB-3 annotation manual [Webber et al. (2019], “Cause.Reason
(Cause.Result) is used when Arg2 (Arg1) gives the reason, explanation or
justification, while Arg1 (Arg2) gives its effect”. Within the Cause sense, we
noticed that its sub-senses, Cause.Reason and Cause.Result, fit into the five
tests framework well. In particular, we were able to construct a question like
“Why did <effect>?”, and answer with “Because (of) <cause>.” in an extractive
manner.

However, we required some relaxations in the construction of the “Why” test to
be able to include other senses into our corpus. We highlight the relaxations
as follows:

#####  Purpose types

We permitted the answer to include additional words like “In order to (achieve
the goal of) <cause>”. In this setting, the cause is a goal, or a
justification for the effect action. See Appendix A Example (3).

#####  Condition types

We allowed the relaxation of the question to include modal terms, for example:
“Why could/would <effect> occur?”. See Appendix A Example (7).

#####  Cause.NegResult and Negative-Condition types

Quoting the PDTB-3 annotation manual [Webber et al. (2019], “Cause.NegResult
is used when Arg1 gives the reason, explanation or justification that prevents
the effect mentioned in Arg2”. For Cause.NegResult and Negative-Condition
senses, we allowed a further relaxation of the question to include “prevent”
or “cause-to-end” terms, as in: “Why was <effect> prevented?”, “Why could be
<effect> prevented?” or “Why did <effect> end?”. See Appendix A Example (8).

For samples with multiple sentences, we annotated each sentence independently,
that is, we do not consider cross-sentence causality. We acknowledge that by
omitting cross-sentence examples, we potentially lose out between a 1/10 to a
1/5 of additional causal examples, based on proportions estimated from CTB and
ESL respectively. Titles were also not annotated.

### 3.3. Workflow

Five annotators and one curator were involved in the annotation effort. Figure
2 summarizes the different annotation iterations to build CNC.

Figure 2: Summary of annotation workflow. $n$: sample size for the subset.
$a$: number of annotators for the subset. $\kappa$: Krippendorff’s Alpha score
for samples per round. Round 1 was a trial training round where annotations
were discarded. All subsequent rounds were for the training set except Round 6
which was for the test set.

#### 3.3.1. Initial Training

The initial training phase for annotators corresponds to Round 1 in Figure 2.
Guidelines were presented at the beginning of the annotation phase, and
examples outside of CNC dataset were ran through with annotators. We found it
difficult to arrive at a consensus on the final label across annotators during
this initial discussion. This experience motivated us to update our guidelines
after each round of annotation to continually improve annotators’
understanding of the guidelines.

#### 3.3.2. Guidelines Refinement

The difficulty in achieving alignment across annotators, especially at the
initial stages, arose from the following three issues: Firstly, since our task
challenges literacy, comprehension, and cognitive understanding of a sentence,
it is a complex and involved task. Thus, a significant amount of time is
required for careful reading of the annotation manual and examples to improve
familiarity. Secondly, sensitivity to causal cues and signals (including
implicit signals) takes time to develop, and might not be apparent to some
until later rounds. Thirdly, different annotators rely on different guidelines
or feedback mechanisms to improve their accuracy to identify causal labels
faithful to the guidelines. Therefore, we were motivated to update our
guidelines in an iterative fashion after each annotation round to refine the
descriptions and include more examples that would help improve alignment
across annotators. A group discussion always followed at the end of each
annotation round for annotators and curator to share feedback and agree on the
final guidelines. We also allowed annotators to skip examples if they were
unable to come up with a suitable label.

For any major changes in guidelines that might affect previous rounds of
annotations, the curator identified such examples and discussed with all
annotators before the final annotation is amended. In total, 33 examples had
their labels changed in this manner.

#### 3.3.3. Final Annotation

In Round 2, we achieved a Krippendorff’s Alpha ($\kappa$) score of 29.77%
across five annotators. After the second annotation round, we split the five
annotators into two groups to cover more data within our time frame.
Concurrently, we also noticed an improvement in the agreement scores to
48.08%. By the third annotation round, the guidelines had no further changes.
Surprisingly, we noticed agreement scores fell to 26.15% in Round 4. This
could be due to the much larger dataset presented to annotators in this round,
causing reduced focus. In fact, we noticed the main reason for the drop arose
from one annotator in the first group. Excluding his contribution, the
agreement increased to a reasonable score of 39.34%. After working out
misunderstandings, the subsequent two rounds of annotations achieved
consistent agreement scores of 36.95% and 48.55%. Across all rounds, our
dataset has an agreement score of 34.99%.

In our final corpus, every example was annotated by at least two annotators.
If only one annotation is present because the other annotator skipped that
example, the curator’s vote is final. The curator also resolved any ties
present throughout the annotation exercise. In the end, we obtained 3,559
annotated sentences, with 1,957 marked as _Causal_ and 1,602 marked as _Non-
causal_. This is reflected in the first row of Table 2, alongside data sizes
for two other external datasets we experimented with in the next section. We
followed the train-test split of our data source, and also provide the
breakdown of data sizes by class labels in Table 2.

| _Causal_ | _Non-causal_ | Total
---|---|---|---
CNC | 1,957 | 1,602 | 3,559
CNC Training | 1,781 | 1,467 | 3,248
CNC Test | 176 | 135 | 311
PDTB-3 | 9,917 | 18,358 | 28,275
CTB | 318 | 1,418 | 1,736
Table 2: Data sizes by class label.

## 4\. Experiments

# | Training Set | F1 | P | R | Acc | MCC
---|---|---|---|---|---|---
1 | All _Causal_ | 72.28 | 56.59 | 100.00 | 56.59 | 0.00
2 | Random | 55.72 | 56.61 | 54.92 | 50.66 | 0.00
3 | CNC Training | 81.20 | 78.01 | 84.66 | 77.81 | 54.52
4 | PDTB-3 | 55.43 | 81.32 | 42.05 | 61.74 | 32.09
5 | PDTB-3 Bal | 64.45 | 77.60 | 55.11 | 65.59 | 34.75
6 | CTB | 27.36 | 80.56 | 16.48 | 50.48 | 17.49
7 | CTB Bal | 64.05 | 75.38 | 55.68 | 64.63 | 32.13
Table 3: Metrics from predictions on CNC Test Set using various settings. F1,
Precision (P), Recall (R), Accuracy (Acc) and Matthews Correlation Coefficient
(MCC) are reported in percentages (%). Highest score per column is indicated
in boldface.

In this section, we present a battery of baseline models and results of causal
sentence classification on the CNC data and the two existing news corpora,
PDTB-3 [Webber et al. (2019] and CTB [Mirza et al. (2014].

### 4.1. Evaluation Metrics

For all experiments, we evaluated the predictions against true labels using
five evaluation metrics: Precision (P), Recall (R), F1, Accuracy (Acc) and
Matthews Correlation Coefficient (MCC) [Matthews (1975]. Formulas are provided
below in Equations (1) to (5), where $TP$/ $TN$/ $FP$/ $FN$ stands for True
Positive/True Negative/False Positive/ False Negative respectively. All
metrics were calculated using the scikit-learn [Pedregosa et al. (2011]
package.

$P=\frac{TP}{TP+FP}$ (1)

$R=\frac{TP}{TP+FN}$ (2)

$F1=\frac{2\times P\times R}{P+R}$ (3)

$Acc=\frac{TP+TN}{TP+TN+FP+FN}$ (4)

$MCC=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$ (5)

### 4.2. Models

#### 4.2.1. Dummy Baselines

To create naive baselines, we obtained dummy predictions using two approaches.
For the first row indicated by “All _Causal_ ” of Table 3, we predicted all
examples to be _Causal_ , which is the majority label of our dataset. Notice
that the Recall score is perfect (100%), because all _Causal_ examples will be
identified if we predict all as _Causal_. However, Precision is low (56.59%),
because all _Non-causal_ examples were wrongly identified as _Causal_. In the
subsequent discussions, we focus our key evaluation metrics to be F1, Accuracy
and MCC. For the second row, we randomly assigned the _Causal_ or _Non-causal_
labels to each example based on the ground truth distribution from CNC’s
training set, averaged over $1,000$ runs.

#### 4.2.2. Neural Network Baselines

We constructed a baseline model by fine-tuning the pre-trained Bidirectional
Encoder Representations from Transformers (BERT) model [Devlin et al. (2019].
In particular, we used bert-base-cased from Huggingface [Wolf et al. (2020]
with AdamW Cross Entropy Loss. The learning rate was $5e-05$ with linear
scheduling. Train batch size was $32$, and number of epochs was $10$. The
model pipeline is as follows: Tokenized sentences aree run through a BERT
transformer module. The pooled output is obtained by taking the hidden state
corresponding to the first token. Finally, a binary classification head is
applied to obtain the predicted logits.

Additionally, we constructed a Long-Short Term Memory (LSTM) model [Hochreiter
and Schmidhuber (1997] with fastText embeddings [Bojanowski et al. (2017]. The
LSTM model performed on par with the BERT model for F1 but had lower MCC
values than the BERT model. Overall, the BERT model outperformed the LSTM
model. Details about the LSTM baseline and its evaluations are in Appendix B.
Our subsequent analyses focuses on the BERT baseline.

### 4.3. Training on CNC

From the third row in Table 3, when we trained on our training set and applied
on the test set, we achieved a reasonable F1 score of 81.20%, Accuracy of
77.81%, and MCC of 54.52%, each exceeding the dummy baselines scores reflected
in the first two rows. This finding demonstrates that our annotations are
internally consistent and reliable for a simple baseline model to learn
effectively.

### 4.4. Training on External Corpora

In this subsection, we check the transferability of existing event causality
and causality corpora on CNC. To conduct the experiment, we first trained on
only the external corpus (i.e. PDTB-3 or CTB) and then applied the trained
model to predict causal labels on CNC test set.

For PDTB-3, we trained on 28,275 examples (9,917 as _Causal_ and 18,358 as
_Non-causal_). We obtained this number of examples by selecting all causal
examples based on the four senses described in Section 3.2. Next, we added the
_Non-causal_ examples and performed de-duplication such that if two examples
from the same document overlap in some part but have a different causal label,
we dropped the _Non-causal_ example. Once the model is trained, we used it to
predict the causal labels of CNC test set. According to Row 4 of Table 3,
there is some overlap in the annotation schemes of PDTB and CNC. Therefore,
the model was able to achieve Accuracy and MCC scores of 61.74% and 32.09%,
which are higher than the first two dummy baselines. However, the F1 score is
only at 55.43%, comparable to the Random baseline and worse than the All-
_Causal_ assignment. The low F1 score is driven by the low Recall score, which
indicates that a model trained on PDTB-3 is unable to detect most of the true
_Causal_ examples. However, its high Precision score indicates the model does
return high-quality predictions. The low Recall could be explained by the fact
that we included guidelines in our annotation schema that are different from
PDTB’s schema. For example, we were not strict about arguments having to be a
proper clause. That is, for a sentence like _“John caused the fire and walked
away.”_ , we would have marked it as _Causal_. However, since _“John”_ and
_“the fire”_ are not clauses, PDTB would not mark this example as _Causal_. As
a result, a model trained on PDTB would also recognize fewer causal
constructions.

To tackle the issue of imbalanced class distribution in PDTB-3, which might
affect model performance, we reran an experiment with randomly sampled _Non-
causal_ examples to match the number of _Causal_ examples. This corpus is
referred to as “PDTB-3 Bal” ($n=19,834$) in Row 5 of Table 3. All metrics,
except Precision, improved relative to the imbalanced PDTB-3 setting.

We repeated the experiment with examples obtained from CTB. For this corpus,
we had 1,736 examples (318 as _Causal_ and 1,418 as _Non-causal_) to train
with. For sentences marked with “CLINK” in CTB, we denoted them as _Causal_.
All remaining sentences were marked as _Non-causal_. Note a possible
complication arises because while all _Non-causal_ examples are single
sentences, _Causal_ examples might span across a few sentences. Performance
was dismal, returning comparable or worse scores for F1 and Accuracy than both
dummy baselines. We hypothesized that this poor outcome is a result of CTB’s
extremely imbalanced class distribution. Indeed, when we retrained on a
balanced sample, “CTB Bal” ($n=636$), we observed performance improvements
across all metrics in the last row of Table 3. A reasonable F1 score of
64.05%, Accuracy of 64.63% and MCC of 32.13% was obtained, similar to or
slightly better than PDTB’s performance, despite CTB Bal being much smaller in
size.

To conclude, we trained models on external corpora and used them directly to
make predictions on our proposed CNC corpus. This demonstrates the
transferability of existing causality corpora on our corpus. Nonetheless,
since both external corpora contain multi-sentence examples and differ
slightly in annotation guidelines, some performance differences are expected.

### 4.5. Training & Testing on External Corpora

Training Set | Test Set |
---|---|---
| F1 |  | MCC
| CNC | PDTB-3 | CTB Bal | TRF$\uparrow$ |  | CNC | PDTB-3 | CTB Bal | TRF$\uparrow$
CNC | 83.46 | 58.38 | 80.65 | 74.16 |  | 61.71 | 30.68 | 59.11 | 50.50
PDTB-3 | 56.45 | 74.45 | 60.79 | 63.90 |  | 35.86 | 61.36 | 32.49 | 43.24
CTB Bal | 59.10 | 49.21 | 83.41 | 63.90 |  | 32.10 | 17.48 | 65.01 | 38.20
Table 4: Metrics from predictions using different train and test sets.
Diagonals per sub-table (i.e. when training set and test sets are the same)
refers to 5-folds CV experiments. F1 and Matthews Correlation Coefficient
(MCC) are reported in percentages (%). Transferability Rate (TRF) indicates
how well a model trained on a given corpus works for unseen, external
datasets: For example, the TRF for CNC is given by the average of its
performance on PDTB-3 and CTB Bal, based on a particular metric (such as F1 or
MCC). Highest score per column is indicated in boldface.

We continue to study the compatibility of CNC with the two external corpora.
In this section, we study if a model trained on CNC will be helpful for event
sentence causality prediction on unseen datasets like PDTB-3 and CTB.

For each of the three corpora, we trained on the whole corpus and applied the
trained model on the remaining two corpora. For example, we trained on all
examples from CNC (3,559 examples) and used the model to predict labels for
PDTB-3 and CTB Bal. We experimented with the balanced corpus of CTB because
the results of the experiments presented in Section 4.4 shown that the full
CTB returns extremely poor performance when used for training. We also
included the 5-Fold cross-validation (CV) scores as a reference: For each
dataset, we randomly split the examples into five validation sets, and for
each round, the corresponding remaining samples were used for training.

Table 4 reflects key experimental results. As expected, for every corpus, the
CV training setup always returns the best performance. After which, for both
PDTB-3 and CTB Bal, we noticed that a model trained on CNC always returns a
higher performance across metrics. For example, when testing on CTB Bal, a
model trained on PDTB-3 achieved 60.79% while a model trained on our corpus
achieved 80.65% F1 score. We introduced a metric called Transferability Rate
(TRF), which computes the mean score across test sets. TRF indicates how well
a model trained on a given corpus works on average for unseen, external
datasets. CNC obtained the highest TRF across the three corpora for both F1
and MCC, reflecting good transferability.

Our findings once again demonstrate that there exists some differences in the
annotation schemes for causality, which reduces the transferability of trained
models between corpora. Nevertheless, our corpus is the most transferable
between the event causality (i.e. CTB) and linguistic causality (i.e. PDTB)
corpora studied. Therefore, our annotation guidelines and dataset can serve as
a bridge between these two strands of causality corpora.

### 4.6. Using CNC for Pre-Training

Dataset | PTM | F1 | P | R | Acc | MCC
---|---|---|---|---|---|---
PDTB | bert-base-cased | 74.45 | 76.76 | 72.31 | 82.60 | 61.36
| CNC-PTM | 75.19 | 75.73 | 74.69 | 82.71 | 61.95
CTB Bal | bert-base-cased | 83.41 | 77.25 | 90.95 | 81.91 | 65.01
| CNC-PTM | 84.68 | 80.14 | 90.02 | 83.80 | 68.30
Table 5: Metrics from pre-trained model (PTM) experiments. F1, Precision (P),
Recall (R), Accuracy (Acc) and Matthews Correlation Coefficient (MCC) are
reported in percentages (%). Highest score per column per dataset is indicated
in boldface.

Since we believe CNC is transferable, we hypothesize that CNC will also be
useful for training a pre-trained model (PTM). Later, this PTM can be fine-
tuned onto any other causality-based corpora. We term the BERT PTM updated on
all examples of CNC as ‘CNC-PTM’. CNC-PTM was then fine-tuned on each external
dataset (i.e. PDTB and CTB Bal). We again used the 5 CV setup444For each
external dataset and each fold, CNC-PTM was fine-tuned on the training set and
applied onto the validation set. This means that for each external dataset, we
would have 5 fine-tuned models at the end of the CV experiment., but reduced
the training epochs to 2. We chose this low number in order to keep our model
intact. The fine-tuning step only serves to adjust the model’s alignment to
better fit the new data distribution.

From Table 5, we indeed found improvements in the average performance across
folds for both PDTB and CTB Bal. Logically, CNC-PTM is better initialized for
causal sentence classification than the general bert-base-cased PTM for two
reasons: (1) It had access to more causal examples and, (2) Its structure is
more adapted for the classification task. Thus, performance improvements are
expected. These findings again support our claim that CNC is generalizable to
out-of-distribution datasets. Therefore, it was useful for updating a PTM.

## 5\. Crowd-Sourcing Annotations

| F1 | P | R | Acc | MCC
---|---|---|---|---|---
All _Causal_ | 66.00 | 50.00 | 100.00 | 48.40 | -3.89
Majority | 61.97 | 47.83 | 88.00 | 46.00 | -14.74
Each Vote | 59.31 | 48.96 | 75.20 | 48.40 | -3.79

Table 6: Metrics from crowd-sourced workers for a subset of 50 examples. F1,
Precision (P), Recall (R), Accuracy (Acc) and Matthews Correlation Coefficient
(MCC) are reported in percentages (%). Highest score per column is indicated
in boldface.

We experimented with crowd-sourced workers using Amazon Mechanical Turk
(MTurk). We randomly selected 50 examples, of which 25 were _Causal_ and 25
were _Non-causal_. For each causal label, we selected 13 examples that were
more complex (with the percentage of the vote for the label being $\leq 0.75$,
while the remaining 12 examples were ‘easier’ examples with high agreements).
The annotation manual was provided to the MTurk workers, and no restrictions
were set on the type of workers we allowed for. We requested for five unique
annotators to be assigned to each example. Therefore, 250 annotations were
received555Although our dataset was balanced, crowd-sourced workers returned a
skewed distribution in votes: 192 votes for _Causal_ and 58 votes for _Non-
causal_., of which we had 44 unique workers.

Kappa scores were low at 1.62%. Evaluation metrics of these crowd-sourced
annotations against our expert labels are reflected in Table 6. The second row
reflects predictions taken based on the majority vote (i.e. mode out of five
votes per example). The third row reflects outcomes when we compared each of
the 250 annotations with our true labels independently. In both cases,
performance of crowd-sourced workers is poor. Since most annotators labelled
examples as _Causal_ , the results are similar to the results from an All-
_Causal_ dummy baseline (shown in the first row). In fact, in terms of F1,
Accuracy, and MCC, the workers’ scores are close, if not worse than, the All
_Causal_ baseline. As highlighted in Section 3.3, the task is challenging and
requires a thorough understanding of the annotation guideline to do well.
Given that MTurk provides a low monitoring environment with no room for
iterative feedback, it is understandable that crowd-sourced workers perform
poorly (close to random assignment) at their tasks.

Nevertheless, this experiment provided a few learning points: Firstly, our
manual annotations are not easy to identify by a layman, and skilled
annotators are required. Similar to our annotation experience, it is necessary
to have repeated discussions and feedback sessions with annotators so as to
improve agreement scores. Secondly, compared to a BERT model trained on
internal examples or external examples annotated with similar logic, the
layman identifies causality poorly666In terms beating dummy baselines,
especially for Accuracy and MCC metrics.. ?) argues that discrepancies in the
quality of crowd-sourced annotations from experts should be used as an
estimate on how complex the task is. Together, these findings highlight that
our corpus is a unique and valuable resource, requiring an immense amount of
time and effort to create with experts.

## 6\. Conclusion

Causality is an important cognitive concept that deserves to have a corpus
dedicated to the definition and identification. CNC focuses on annotating
causality in text, and so far, we have created an annotation schema to
identify if event sentences contain causal relations or not.

CNC’s annotation guidelines are constructed based on linguistic rules, and
covers a wider array of causal linguistic constructions than previous works.
We also demonstrated transferability between CNC and existing datasets that
include causal relations like CTB and PDTB. Finally, the binary causal event
classification task is challenging: layman workers only achieved close poor
performance in our crowd-sourced annotation experiment. Therefore, CNC, which
has been annotated by experts, is a valuable resource for researchers working
in the causality and event causality space. CNC could potentially be relevant
to NLU researchers working on downstream tasks too.

We are also organizing a shared task using CNC to promote the development of
automatic causal text mining solutions777Our shared task competition page is
at https://codalab.lisn.upsaclay.fr/competitions/2299.. In our next dataset
release, we will clean some parts of our current dataset and perform
additional curation of mispredicted examples in CV. We are also in the midst
of adding fine-grained annotations, such as cause-effect-signal spans and
causal concept labels.

## 7\. Acknowledgements

This project is supported by the National Research Foundation, Singapore under
its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any
opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not reflect the views of National
Research Foundation, Singapore.

## 8\. Bibliographical References

## References

  * Asghar (2016 Asghar, N.  (2016).  Automatic extraction of causal relations from natural language texts: a comprehensive survey.  arXiv preprint arXiv:1605.07895.
  * Barik et al. (2016 Barik, B., Marsi, E., and Öztürk, P.  (2016).  Event causality extraction from natural science literature.  Res. Comput. Sci., 117:97–107.
  * Blanco et al. (2008 Blanco, E., Castell, N., and Moldovan, D.  (2008).  Causal relation extraction.  In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08).
  * Bojanowski et al. (2017 Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T.  (2017).  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135–146.
  * Cao et al. (2016 Cao, M., Sun, X., and Zhuge, H.  (2016).  The role of cause-effect link within scientific paper.  In Hai Zhuge et al., editors, 12th International Conference on Semantics, Knowledge and Grids, SKG 2016, Beijing, China, August 15-17, 2016, pages 32–39. IEEE Computer Society.
  * Caselli and Inel (2018 Caselli, T. and Inel, O.  (2018).  Crowdsourcing StoryLines: Harnessing the crowd for causal relation annotation.  In Proceedings of the Workshop Events and Stories in the News 2018, pages 44–54, Santa Fe, New Mexico, U.S.A, August. Association for Computational Linguistics.
  * Caselli and Vossen (2017 Caselli, T. and Vossen, P.  (2017).  The event StoryLine corpus: A new benchmark for causal and temporal relation extraction.  In Proceedings of the Events and Stories in the News Workshop, pages 77–86, Vancouver, Canada, August. Association for Computational Linguistics.
  * Dalal et al. (2021 Dalal, D., Arcan, M., and Buitelaar, P.  (2021).  Enhancing multiple-choice question answering with causal knowledge.  In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 70–80, Online, June. Association for Computational Linguistics.
  * Devlin et al. (2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Do et al. (2011 Do, Q., Chan, Y. S., and Roth, D.  (2011).  Minimally supervised event causality identification.  In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 294–303.
  * Dunietz et al. (2017 Dunietz, J., Levin, L., and Carbonell, J.  (2017).  The BECauSE corpus 2.0: Annotating causality and overlapping relations.  In Proceedings of the 11th Linguistic Annotation Workshop, pages 95–104, Valencia, Spain, April. Association for Computational Linguistics.
  * Dunietz et al. (2020 Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu-Carroll, J., and Ferrucci, D.  (2020).  To test machine comprehension, start by defining comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7839–7859, Online, July. Association for Computational Linguistics.
  * Feder et al. (2021 Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al.  (2021).  Causal inference in natural language processing: Estimation, prediction, interpretation and beyond.  arXiv preprint arXiv:2109.00725.
  * Girju and Moldovan (2002 Girju, R. and Moldovan, D. I.  (2002).  Text mining for causal relations.  In Susan M. Haller et al., editors, Proceedings of the Fifteenth International Florida Artificial Intelligence Research Society Conference, May 14-16, 2002, Pensacola Beach, Florida, USA, pages 360–364. AAAI Press.
  * Grivaz (2010 Grivaz, C.  (2010).  Human judgements on causation in French texts.  In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta, May. European Language Resources Association (ELRA).
  * Hassanzadeh et al. (2019 Hassanzadeh, O., Bhattacharjya, D., Feblowitz, M., Srinivas, K., Perrone, M., Sohrabi, S., and Katz, M.  (2019).  Answering binary causal questions through large-scale text mining: An evaluation using cause-effect pairs from human experts.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5003–5009. International Joint Conferences on Artificial Intelligence Organization, 7.
  * Heindorf et al. (2020 Heindorf, S., Scholten, Y., Wachsmuth, H., Ngomo, A. N., and Potthast, M.  (2020).  Causenet: Towards a causality graph extracted from the web.  In Mathieu d’Aquin, et al., editors, CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 3023–3030. ACM.
  * Hidey and McKeown (2016 Hidey, C. and McKeown, K.  (2016).  Identifying causal relations using parallel Wikipedia articles.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1424–1433, Berlin, Germany, August. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997 Hochreiter, S. and Schmidhuber, J.  (1997).  Long Short-Term Memory.  Neural Computation, 9(8):1735–1780, 11.
  * Hürriyetoğlu et al. (2020a Ali Hürriyetoğlu, et al., editors.  (2020a).  Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, Marseille, France, May. European Language Resources Association (ELRA).
  * Hürriyetoğlu et al. (2020b Hürriyetoğlu, A., Zavarella, V., Tanev, H., Yörük, E., Safaya, A., and Mutlu, O.  (2020b).  Automated extraction of socio-political events from news (AESPEN): Workshop and shared task report.  In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 1–6, Marseille, France, May. European Language Resources Association (ELRA).
  * Hürriyetoğlu et al. (2021a Hürriyetoğlu, A., Mutlu, O., Yörük, E., Liza, F. F., Kumar, R., and Ratan, S.  (2021a).  Multilingual protest news detection - shared task 1, CASE 2021.  In Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), pages 79–91, Online, August. Association for Computational Linguistics.
  * Hürriyetoğlu et al. (2021b Hürriyetoğlu, A., Tanev, H., Zavarella, V., Piskorski, J., Yeniterzi, R., Mutlu, O., Yuret, D., and Villavicencio, A.  (2021b).  Challenges and applications of automated extraction of socio-political events from text (CASE 2021): Workshop and shared task report.  In Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), pages 1–9, Online, August. Association for Computational Linguistics.
  * Hürriyetoğlu et al. (2021c Hürriyetoğlu, A., Yörük, E., Mutlu, O., Duruşan, F., Yoltar, Ç., Yüret, D., and Gürel, B.  (2021c).  Cross-context news corpus for protest event-related knowledge base construction.  Data Intelligence, 3(2):308–335, 06.
  * Hürriyetoğlu (2021 Ali Hürriyetoğlu, editor.  (2021).  Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), Online, August. Association for Computational Linguistics.
  * Jo et al. (2021 Jo, Y., Bang, S., Reed, C., and Hovy, E. H.  (2021).  Classifying argumentative relations using logical mechanisms and argumentation schemes.  Trans. Assoc. Comput. Linguistics, 9:721–739.
  * Khoo et al. (2000 Khoo, C. S. G., Chan, S., and Niu, Y.  (2000).  Extracting causal knowledge from a medical database using graphical patterns.  In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 336–343, Hong Kong, October. Association for Computational Linguistics.
  * Kontos and Sidiropoulou (1991 Kontos, J. and Sidiropoulou, M.  (1991).  On the acquisition of causal knowledge from scientific texts with attribute grammars.  International Journal of Applied Expert Systems, 4(1):31–48.
  * Matthews (1975 Matthews, B. W.  (1975).  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451.
  * Mirza et al. (2014 Mirza, P., Sprugnoli, R., Tonelli, S., and Speranza, M.  (2014).  Annotating causality in the TempEval-3 corpus.  In Proceedings of the EACL 2014 Workshop on Computational Approaches to Causality in Language (CAtoCL), pages 10–19, Gothenburg, Sweden, April. Association for Computational Linguistics.
  * Mostafazadeh et al. (2016 Mostafazadeh, N., Grealish, A., Chambers, N., Allen, J., and Vanderwende, L.  (2016).  CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures.  In Proceedings of the Fourth Workshop on Events, pages 51–61, San Diego, California, June. Association for Computational Linguistics.
  * Pedregosa et al. (2011 Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E.  (2011).  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830.
  * Pennington et al. (2014 Pennington, J., Socher, R., and Manning, C.  (2014).  GloVe: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics.
  * Prasad et al. (2006 Prasad, R., Miltsakaki, E., Dinesh, N., Lee, A., Joshi, A., and Webber, B. L.  (2006).  The penn discourse treebank 1.0 annotation manual.  IRCS Technical Reports Series.
  * Prasad et al. (2008 Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A. K., and Webber, B. L.  (2008).  The penn discourse treebank 2.0.  In Proceedings of the International Conference on Language Resources and Evaluation, LREC 2008, 26 May - 1 June 2008, Marrakech, Morocco. European Language Resources Association.
  * Riaz and Girju (2013 Riaz, M. and Girju, R.  (2013).  Toward a better understanding of causality between verbal events: Extraction and analysis of the causal power of verb-verb associations.  In Proceedings of the SIGDIAL 2013 Conference, pages 21–30.
  * Sakaji et al. (2008 Sakaji, H., Sekine, S., and Masuyama, S.  (2008).  Extracting causal knowledge using clue phrases and syntactic patterns.  In Takahira Yamaguchi, editor, Practical Aspects of Knowledge Management, 7th International Conference, PAKM 2008, Yokohama, Japan, November 22-23, 2008. Proceedings, volume 5345 of Lecture Notes in Computer Science, pages 111–122. Springer.
  * Saurı et al. (2006 Saurı, R., Littman, J., Knippen, B., Gaizauskas, R., Setzer, A., and Pustejovsky, J.  (2006).  Timeml annotation guidelines version 1.2. 1.
  * Stasaski et al. (2021 Stasaski, K., Rathod, M., Tu, T., Xiao, Y., and Hearst, M. A.  (2021).  Automatically generating cause-and-effect questions from passages.  In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 158–170, Online, April. Association for Computational Linguistics.
  * Tan et al. (2021 Tan, F. A., Hazarika, D., Ng, S.-K., Poria, S., and Zimmermann, R.  (2021).  Causal augmentation for causal sentence classification.  In Proceedings of the First Workshop on Causal Inference and NLP, pages 1–20, Punta Cana, Dominican Republic, November. Association for Computational Linguistics.
  * Webber et al. (2019 Webber, B., Prasad, R., Lee, A., and Joshi, A.  (2019).  The penn discourse treebank 3.0 annotation manual.  Philadelphia, University of Pennsylvania.
  * Wolf et al. (2020 Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A.  (2020).  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October. Association for Computational Linguistics.
  * Xu et al. (2020 Xu, J., Zuo, W., Liang, S., and Zuo, X.  (2020).  A review of dataset and labeling methods for causality extraction.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 1519–1531, Barcelona, Spain (Online), December. International Committee on Computational Linguistics.
  * Yang et al. (2021 Yang, J., Han, S. C., and Poon, J.  (2021).  A survey on extraction of causal relations from natural language text.  CoRR, abs/2101.06426.
  * Yörük et al. (2021 Yörük, E., Hürriyetoğlu, A., Duruşan, F., and Yoltar, Ç.  (2021).  Random sampling in corpus design: Cross-context generalizability in automated multicountry protest event collection.  American Behavioral Scientist, 0(0):00027642211021630.

## Appendix

##  Appendix A Annotated Examples

The following are some examples of the expected causal label for each
sentence, along with justifications using the Five Tests highlighted in
Section 3.2.

  1. (1)

His attackers allegedly drank his blood .

  * •

_Non-causal_ : There is only one event in this sentence.

  1. (2)

The protests spread to 15 other towns and resulted in two deaths and the
destruction of property .

  * •

_Causal_ : This sentence contains causal events.

  1. 1.

Why: Why were there “two deaths and the destruction of property”? Because “the
protests spread to 15 other towns”.

  2. 2.

Temporal order: Protests must spread (Cause) before deaths and destruction
(Effect) can occur.

  3. 3.

Counterfactual: Deaths and destruction (Effect) are unlikely to occur if the
protests did not spread (NegCause).

  4. 4.

Ontological asymmetry: Using Cause as an Effect to construct a question: Why
did “The protests spread to 15 other towns”? Because of deaths and destruction
– does not answer the question.

  5. 5.

Linguistic: “The protests spread to 15 other towns” causes “two deaths and the
destruction of property”.

  1. (3)

The three-member Farlam Commission, chaired by retired judge Ian Farlam, was
established by President Jacob Zuma to probe into the violence and the deaths
of 44 people in wage-related protests.

  * •

_Causal_ : This sentence contains causal event of the Purpose sense.

  1. 1.

Why: Why was “the three-member Farlam Commision … established by President
Jacob Zuma”? In order “to probe into the violence and the deaths of 44 people
in wage-related protests”.

  2. 2.

Temporal order: The probe purpose (Cause) must occur before establishing the
commission (Effect).

  3. 3.

Counterfactual: Establishing the commission (Effect) is unlikely to occur if
the President did not have the goal to probe into the protests (NegCause).

  4. 4.

Ontological asymmetry: Using Cause as an Effect to construct a question: Why
did the President “probe into violence and deaths of people in wage-related
protests”? Because the Commission was established – does not answer the
question.

  5. 5.

Linguistic: “to probe into the violence and the deaths of 44 people in wage-
related protests” causes “The three-member Farlam Commission , chaired by
retired judge Ian Farlam , was established by President Jacob Zuma”.

  1. (4)

Chale was allegedly chased by a group of about 30 people and was hacked to
death with pangas, axes and spears.

  * •

_Non-causal_ : This sentence exhibits temporal events that have no causal
connections. We highlight the tests it fails below:

  1. 1.

Why: Why was Chale “hacked to death with pangas, axes and spears”? Because
“Chale was allegedly chased by a group of about 30 people”. – This test fails
since the Cause does not really explain the reason for why there was the
Effect.

  2. 2.

Counterfactual: The counterfactual is “The hacking (Effect) is unlikely to
occur if the chasing did not take place (NegCause).” – This test fails since
hacking can occur without chasing. In fact, without chasing, Chale might have
been hacked earlier.

  1. (5)

“I observed the attack on the police, I have no doubt about it,” Modiba said
during cross-examination.

  * •

_Causal_ : This sentence contains a description of an implicit causal event
relation within a speech.

  1. 1.

Why: Why did Modiba “have no doubt about it”? Because Modiba “observed the
attack on the police”.

  2. 2.

Temporal order: Modiba must observe (Cause) before claiming he had no doubts
(Effect).

  3. 3.

Counterfactual: Modiba cannot claim that he has no doubts (Effect) if he had
not observed the attack (NegCause).

  4. 4.

Ontological asymmetry: Using Cause as an Effect to construct a question: Why
was it that Modiba “observed the attack on the police”? Because he had no
doubt about it – does not answer the question.

  5. 5.

Linguistic: “I observed the attack on the police” causes “I have no doubt
about it”.

  1. (6)

Both Anoop and Ramanan are also accused in the case related to attack on the
Nitta Gelatin office at Panampilly Nagar last year.

  * •

_Non-causal_ : The inclusion of “also” in “also accused” renders that when
constructing an answer for the Effect, we need to explain why they are accused
in addition to someone else that has already been accused. Therefore, when
performing the “Why” test, the answer seems incomplete. Hence, we do not
consider this sentence to be causal.

  1. 1.

Why: Why were “both Anoop and Ramanan also accused in the case”? Because of
the “attack on the Nitta Gelatin office at Panampilly Nagar last year .” –
This does not answer the question.

  2. 2.

Linguistic: “attack on the Nitta Gelatin office at Panampilly Nagar last year
” cause “Both Anoop and Ramanan are also accused in the case”. – This does not
work.

  1. (7)

Striking mineworkers have threatened to halt all mining operations if their
employers do not accede to their pay demand.

  * •

_Causal_ : This sentence contains Condition relations, which we interpret as
causal.

  1. 1.

Why: Why could “halt all mining operations” occur? Because “their employers do
not accede to their pay demand”.

  2. 2.

Temporal order: Their pay demands must be acceded (Cause) before the halting
of mining operations (Effect).

  3. 3.

Counterfactual: The halting of mining operations (Effect) is unlikely to occur
if their pay demands were acceded (NegCause).

  4. 4.

Ontological asymmetry: Using Cause as an Effect to construct a question: Why
was it that “their employers do not accede to their pay demands”? We cannot
find an answer span for this question in the original Effect span.

  5. 5.

Linguistic: “their employers do not accede to their pay demand” causes “halt
all mining operations”.

Setting | F1 | P | R | Acc | MCC
---|---|---|---|---|---
All _Causal_ | 72.28 | 56.59 | 100.00 | 56.59 | 0.00
Random | 55.72 | 56.61 | 54.92 | 50.66 | 0.00
CNC Training | 78.22 | 72.68 | 84.66 | 73.31 | 45.15
PDTB-3 | 56.68 | 66.41 | 49.43 | 57.23 | 16.90
PDTB-3 Bal | 68.75 | 56.62 | 87.50 | 54.98 | 0.14
CTB | 33.79 | 86.05 | 21.02 | 53.38 | 23.80
CTB Bal | 65.23 | 71.14 | 60.23 | 63.67 | 28.15
Table 7: Metrics from predictions with the LSTM model on CNC Test Set using
different training sets. F1, Precision (P), Recall (R), Accuracy (Acc) and
Matthews Correlation Coefficient (MCC) are reported in percentages (%).
Highest score per result column is indicated in boldface.

  1. (8)

The strike will continue till our demands are conceded.

  * •

_Causal_ : This sentence contains Negative-Condition relations, which we
interpret as preventive causal type.

  1. 1.

Why: Why could the event that “the strike will continue” be prevented? Because
“our demands are conceded”.

  2. 2.

Temporal order: The demands must be conceded (Cause) before preventing the
strike from continuing (NegEffect).

  3. 3.

Counterfactual: The strike cannot be stopped from continuing (NegEffect) if
the demands are not conceded (NegCause).

  4. 4.

Ontological asymmetry: Using Cause as an Effect to construct a question: Why
was it that “our demands are conceded”? Because “the strike could be stopped”
– does not answer the question.

  5. 5.

Linguistic: “Our demands are conceded” prevents “the strike will continue”.

##  Appendix B Other Baseline Models

As described in Section 4, in addition to the BERT model, we constructed a
LSTM model as one of our baselines, considering its ability to account for
long-distance relations between words. In this model, firstly, an embedding
layer initialized with fastText’s Common Crawl 300-dimensional embeddings
[Bojanowski et al. (2017]. We experimented with GloVe 300-dimensional
embeddings [Pennington et al. (2014] and a concatenation of GloVe and fastText
in our initial experiments and selected fastText because it outperformed
others. This layer is followed by two bi-directional LSTM layers with a dense
layer on top. The final prediction is obtained via a softmax layer. For our
training parameters, we used a maximum sequence length of 128, batch size of
32, a learning rate of $1e^{-3}$ with Adam optimizer and epochs of 20 with
early stopping patience of 5. The obtained results are summarized in Table 7.
We observed that performance of this system is significantly lower than BERT
for all scenarios we tested. However, conclusions regarding the
transferability of PDTB-3 and CTB Bal onto CNC (based on F1 and MCC scores)
holds.


# Geometric VLAD for Large Scale Image Search

Zixuan Wang⋆, Wei Di†, Anurag Bhardwaj†, Vignesh Jagadeesh†, Robinson
Piramuthu†
⋆ Dept.of Electrical Engineering, Stanford University, CA 94305
† eBay Research Labs, eBay Inc., San Jose, CA 95125
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a novel compact image descriptor for large scale image search. Our
proposed descriptor - Geometric VLAD (gVLAD) is an extension of VLAD (Vector
of Locally Aggregated Descriptors) that incorporates weak geometry information
into the VLAD framework. The proposed geometry cues are derived as a
membership function over keypoint angles which contain evident and informative
information but yet often discarded. A principled technique for learning the
membership function by clustering angles is also presented. Further, to
address the overhead of iterative codebook training over real-time datasets, a
novel codebook adaptation strategy is outlined. Finally, we demonstrate the
efficacy of proposed gVLAD based retrieval framework where we achieve more
than $15\%$ improvement in mAP over existing benchmarks.

##  1 Introduction

Proliferation of large-scale image collections on web has made the task of
efficient image retrieval challenging. Given a query image or region, the goal
is to retrieve images of the same object or scene from a large scale database
with high accuracy, efficiency and less memory usage. One of the core problems
is how to concisely represent the visual information present in images. A
number of methods have been proposed recently that address this issue from
both computational efficiency as well as retrieval accuracy perspectives.
However, there is a growing need for algorithms that can achieve reasonable
trade-offs on both these aspects. Vector of Locally Aggregated Descriptors
(VLAD) [8] proposed by Jégou et al. is one of the seminal contributions in
this area as they show that compact and accurate VLAD representation is able
to scale to billions of descriptors (by avoiding expensive hard disk
operations) and still retain superior retrieval performance. However, one of
the limitations of this representation is its inability to incorporate more
keypoint level information that can potentially lead to enhanced performance.
One such information is the dominant angle of the detected keypoint, also
referred to as “Keypoint Angle”, which is often discarded for the sake of
obtaining rotational invariance in matches. A toy example is illustrated in
Figure 1, in which VLAD is unable to differentiate between the configurations
shown in the left and right figures where keypoints (red dots) differ only in
their orientations, while having same descriptor representation and distance
in the feature space towards the centroid $c_{i}$. Thus, we hypothesize that
keypoint angles provide useful geometric cues which can be very useful for
matching images. Integrating this information in a principled way can
substantially improve the performance of existing VLAD based representation.
In this paper, we present Geometric VLAD (gVLAD) that strengthens the VLAD
representation by incorporating weak geometric cue in form of keypoint angles.

Figure 1: gVLAD motivation: A set of key points (denoted in red dot) locates
in the feature space with same distance $r$ towards the centroid $c_{i}$,
assuming they are of same feature descriptor. VLAD is unable to differentiate
between the configurations shown in the left and right figures which differ
only in the orientations of keypoints (depicted by arrow). However, by
separating keypoints into two bins according to their dominant orientation,
and measuring distance of points from each bin towards the centroid
separately, the proposed gVLAD can successfully differentiate between the two
configurations.

Our contributions in this paper are as follows:

  * •

Angle Binning Based VLAD: We propose a novel formulation of gVLAD that
incorporates low level keypoint angles in form of a membership function into
the VLAD representation.

  * •

Circular Preserved Angle Membership Learning: We propose a simple but
effective principled technique to learn the membership function of keypoint
angles based on trigonometric transform and clustering in a fashion that
preserves their circular distribution.

  * •

Codebook Adaptation: To eliminate the need of iterative codebook training for
large scale real-world image collections, a codebook adaptation scheme is
presented.

  * •

Z-Score Normalization: Z-score based normalization technique is proposed that
outperforms existing normalization methods for VLAD-based representations.

  * •

Superior New Benchmark Results: State-of-the-art image retrieval performance
of proposed framework over a number of existing retrieval benchmarks are
achieved.

The paper is organized as follows. In section 2, we outline related work in
large-scale image search and strategies of integrating geometric information
into image representations. In section 3, we describe the geometric VLAD
representation in detail. In section 4, we demonstrate the performance gain on
Oxford, Holidays and Paris benchmarks, as well as on extended large scale
datasets. We conclude the paper and discuss the future work in section 5.

##  2 Related Work

The Bag-of-Words (BoW) representation is one of the most widely used method
for image retrieval [16, 14]. It quantizes each local descriptor SIFT [11] or
SURF [3], to its nearest cluster center and encodes each image as a histogram
over cluster centers also known as “Visual Words”. Good retrieval performance
is achieved with a high dimensional sparse BOW vector, in which case inverted
lists can be used to implement efficient search. However, the search time
grows quadratically as the number of images increase [4].

To overcome this issue, the Fisher kernel based approach proposed by Perronnin
et al. [13] transforms an set of variable-sized independent samples into a
fixed size vector representation. The samples are distributed according to a
parametric generative model, in this case a Gaussian Mixture Model (GMM)
estimated on a training set. A simplified version of Fishers kernels, the VLAD
is proposed by Jégou et al. [8, 9]. It encodes the difference from the cluster
center in a more direct manner, rather than the frequency assigned to the
cluster. It requires less computation than Fisher kernels but can achieve
comparable retrieval performance.

However, most of existing methods ignore the geometric information present in
images. Spatial re-ranking [14] is usually used as a geometric filter to
remove unrelated images from retrieval results. However, due to its expensive
computation it is applied only to top ranked images for re-ranking. The
spatial pyramid [10] is a simple extension of the BOW representation which
partitions the image into increasingly fine sub-regions and computes
histograms of local features found inside each sub-region. It shows improved
performance on scene classification tasks. The weak geometric consistency
constraints (WGC) [7] uses angle and scale information from key points to
verify the consistency of matching descriptors. It can improve the retrieval
performance significantly. Recently, Zhang et al. [17] propose a technique to
encode more spatial information through the geometry-preserving visual phrases
(GVP) which requires a pair of images to obtain geometric information. Chum et
al. [5] propose geometric min-hash, which extends min-hash by adding local
spatial extent to increase the discriminability of the descriptor. It can be
used for nearly duplicate image detection but has not achieved the state-of-
the art performance in retrieval.

|
---|---

|
---|---

Figure 2: Each figure on the left shows an input image. Each figure on the
right shows the detected key points in the image. Keypoints are grouped into
four bins based on their angles represented by the direction of the line, and
is colored by four unique colors. Length of the line corresponds to scale.
Note that each image has a distinct representation based on the orientation of
key points which suggests that this information can be potentially useful in
image representation.

##  3 Proposed Framework

In this section, we introduce the Geometric VLAD (gVLAD) to improve retrieval
performance by incorporating low level angle information from the key points
into VLAD framework.

###  3.1 Geometric VLAD

Let us represent the local descriptor $\mathbf{x}$ to be $d$-dimensional
vector (e.g. SURF or SIFT descriptors). Codebook or visual words are denoted
as $\mu=\left[\mu_{1},\mu_{2},\dots,\mu_{K}\right]$, where $K$ represents the
size of the vocabulary. Let $NN(\mathbf{x})$ represent the nearest-neighbor
function that maps an input descriptor $\mathbf{x}$ to its nearest visual word
$i$ where $1\leq i\leq K$. In the original VLAD [8, 9], to represent a given
image, a set of local descriptors are extracted first. Then, the contribution
of each visual word $v_{i}$ is defined by accumulating distances of all the
descriptors that belong to the $i^{th}$ visual word $\mu_{i}$ as:

$v_{i}=\sum_{\mathbf{x}:NN(\mathbf{x})=i}\mathbf{x}-\mu_{i}$ (1)

Such representation is further L2-normalized, and concatenated to form a
vector representation with size $d\times K$ to represent each image. However,
the above formulation suffers from the drawback that it is unable to
incorporate extra descriptor level information such as angle which can be of
very useful in providing a weak geometrical cue. Thus, we present a gVLAD
representation which encodes such angle information of the descriptor into the
VLAD framework for efficient image matching. To define gVLAD, we redefine a
local descriptor by $\mathbf{x}^{\theta}$, where $\mathbf{x}$ still represents
the appearance feature vector of the descriptor and $\theta$ represents the
angle of the descriptor, i.e. the dominant angle of the keypoint. For example,
in SIFT descriptor, the angle corresponds to the dominant direction of
gradient within a local window. To model the distribution of angles, we
introduce clustering idea and define a membership function over the angles as:
$\psi(\theta(\mathbf{x})):0\leq\theta<2\pi\to\\{1,2,\dots,M\\}$, where $M$
denotes the total number of angular bins.

The gVLAD $v_{i}^{j}$ for $i^{th}$ of the $K$ visual words (feature bin) and
$j^{th}$ of the $M$ angular bins can now be represented as:

$v_{i}^{j}=\left\\{\begin{array}[]{l
l}\sum_{\mathbf{x}^{\theta}:NN(\mathbf{x})=i}\mathbf{x}^{\theta}-\mu_{i}&\quad\text{if
$\psi(\theta(\mathbf{x}))=j$}\\\ \mathbf{0}^{d}&\quad\text{if
$\psi(\theta(\mathbf{x}))\neq j$}\end{array}\right.$ (2)

where $d$ is the dimension of feature vector of local descriptor $\mathbf{x}$.
The contribution of each visual word $V_{i}$ in the geometric VLAD can now be
written as combining individual contributions from each angle bin:

$V_{i}=[v_{i}^{1},v_{i}^{2},\cdots,v_{i}^{M-1},v_{i}^{M}]$ (3)

where $V_{i}$ is a row vector with size of $d\times M$. Our geometric VLAD
(gVLAD) representation $\mathcal{V}$ is defined by accumulating contributions
of from all $K$ visual words, and has $D$ dimensions: $D=K\times d\times M$.

$\mathcal{V}=[V_{1},V_{2},\cdots,V_{K-1},V_{K}]$ (4)

###  3.2 Learning Membership Function - $\psi(\theta)$

One principal way to learn the membership function $\psi(\theta)$ is to apply
clustering over the angle distribution and find the appropriate membership
assignments for each angle value among $M$ learned clusters. Typically angles
have a circular distribution of in the range of $[0,2\pi)$, whereas existing
clustering algorithms that based on L2 distance such as $k$-means assume a
Cartesian co-ordinate space for input data, and can not be applied directly.
To address this issue, we propose to represent each keypoint as $(r,\theta)$,
where $r$ is the radial coordinate. Since we are only interested in angles of
key point $\theta$, we fix $r$ as an arbitrary number $r>0$. We now perform a
non-linear transform from this polar co-ordinate to 2D Cartesian co-ordinate
space using the trigonometric functions:

$\displaystyle x$ $\displaystyle=r\times\cos{\theta}$ (5) $\displaystyle y$
$\displaystyle=r\times\sin{\theta}$ (6)

Thus, each angle is mapped to a point $\mathbf{z}(\theta)=(x,y)$ in this 2-d
space. To learn the membership of function $\psi(\theta)$, we perform k-means
clustering in this space satisfying:

${\arg\min}_{\\{\mathbf{\alpha}_{1},\ldots,\mathbf{\alpha}_{M}\\}}\sum_{i=1}^{M}\sum_{\mathbf{z}_{j}\in\Xi_{i}}{\|\mathbf{z}_{j}-\mathbf{\alpha}_{i}\|}^{2}$
(7)

where $\mathbf{\alpha}_{i}$ is the cluster centroid by averaging all points in
cluster set $\Xi_{i}$. The membership of each angle $\theta$ can be estimated
through:

$\psi(\theta)={\arg\min}_{i\in\\{1,2,\ldots,M\\}}{\|\mathbf{z}(\theta)-\mathbf{\alpha}_{i}\|}^{2}$
(8)

###  3.3 Codebook Adaptation

Most real-world image databases grow continuously which leads to frequent
codebook training processes that are often desirable. We propose a simple
codebook adaptation process that can adapt existing codebooks with incremental
dataset and alleviate the need of frequent large-scale codebook training.
Secondly, this technique also allows codebook training from diverse datasets
as codebook trained on one dataset (i.e. Paris building images) can be adapted
to retrieve images from another dataset (i.e. Flickr holidays images). To
define our codebook adaptation, let us represent a source dataset of images
$S$ where an initial codebook $\mu=\left[\mu_{1},\mu_{2},\dots,\mu_{K}\right]$
is trained. Given a new domain specific dataset $T$, our goal is to adapt
$\mu$ to another domain specific codebook $\hat{\mu}$ given as:

$\displaystyle\hat{\mu_{i}}$
$\displaystyle=\frac{1}{N}\sum_{t=1}^{N}\gamma_{i}(t),\mathbf{x}^{\theta}(t)\in
T$ (9) $\displaystyle\text{where}~{}\gamma_{i}(t)$
$\displaystyle=\left\\{\begin{array}[]{l l}\mathbf{x}^{\theta}(t)&\quad\text{
if $NN(\mathbf{x}^{\theta}(t))=\mu_{i}$}\\\ \mathbf{0}^{d}&\quad\text{if
$NN(\mathbf{x}^{\theta}(t))\neq\mu_{i}$}\end{array}\right.$ (12)

where $N$ is the total number of descriptors in dataset $T$ and
$\mathbf{x}^{\theta}(t)$ represents $t^{th}$ descriptor. In our experiment,
the initial codebook $\mu$ is trained using the Paris dataset. For all the
other experiments on different datasets, $\hat{\mu_{i}}$ is used in
conjunction with Equation 2 to compute the representation of the geometric
VLAD.

###  3.4 gVLAD Normalization

Normalization is important to effectively and correctly measure distance
between vector representation. Here we propose three stages of normalization.
First, we use the intra-normalization [2], where the sum of residuals of each
visual word $v_{i}^{j}$ is L2 normalized independently, where $1\leq i\leq K$
and $1\leq j\leq M$. This step is followed by inter-Z-score normalization
across different visual words. Given a vector $X$, its Z-score normalization
is computed as: $\frac{X-\mu}{\sigma}$, where $\mu$ and $\sigma$ represent the
mean and standard deviation of $X$. Let’s denote the $t^{th}$ entry of $V_{i}$
as $v_{i,t}$, where $V_{i}$ is defined in Equation 3. We apply the inter-Z-
score normalization on each
$\left[v_{1,t},v_{2,t},\ldots,v_{i,t},\ldots,v_{K,t}\right]$, where $1\leq
t\leq{M\times d}$ and $1\leq i\leq K$. At last, the gVLAD vector $\mathcal{V}$
is L2 normalized $\mathcal{V}:=\mathcal{V}/{\|\mathcal{V}\|}_{2}$.

###  3.5 PCA Whitening

Given a large collection of images, the size of representation needs to be
carefully considered so as to be feasible for practical real time retrieval.
For instance, using only $256$ visual words with $64$ dimensional SURF
descriptors and $4$ angle bins generates a feature representation of size
$D=64\times 256\times 4=65,536$. To achieve memory-efficient representation of
this vector, we use standard PCA with pre-whitening as described in [6]. The
PCA whitening matrix can be expressed in the form of:

$\textbf{P}=\textbf{D}^{-1/2}\textbf{E}^{T}$ (13)

where
$\textbf{E}\textbf{D}\textbf{E}^{T}=E\\{\bar{\mathbf{V}}\bar{\textbf{V}}^{T}\\}$
is the eigenvector decomposition of the covariance matrix of the (zero mean)
data $\bar{\textbf{V}}$, where each row
$\bar{\mathcal{V}_{l}}=\mathcal{V}_{l}-\mathcal{V}_{0}$, and $\mathcal{V}_{0}$
is the mean vector computed from all gVALD representation vectors.
$\textbf{D}=\mathrm{diag}[d_{1},d_{2},\dots,d_{D}]$ is the $D\times D$
diagonal matrix containing the eigenvalues and
$\textbf{E}=[e_{1},e_{2},\dots,e_{D}]$ is an orthogonal $N\times D$ matrix
having the eigenvectors as columns.

The obtained whitened gVLAD representation is:

$\tilde{\mathcal{V}_{l}}=\textbf{P}(:,1:\rho)^{T}\times\bar{\mathcal{V}_{l}}$
(14)

where $\rho$ is the number of eigenvectors to keep, i.e. the dimension of
reduced feature vectors. $\tilde{V}_{l}$ is then L2 normalized. The complete
algorithm is outlined in Algorithm 1.

Algorithm 1 Computation of gVLAD descriptor

1:S1: Keypoint detection and descriptor: compute image descriptors
$\mathbf{x}^{\theta}$, where $\mathbf{x}$ is the appearance vector, and
$\theta$ represents the angle.

2:S2: Generate visual vocabulary $\left[\mu_{1},\mu_{2},\dots,\mu_{K}\right]$
using $k$-means on all descriptors from training data.

3:S3: Learning membership function $\psi(\theta)$ for each
$\mathbf{x}^{\theta}$

4:
$\arg\min_{i\in\\{1,2,\ldots,M\\}}{\|\mathbf{z}(\theta)-\mathbf{\alpha}_{i}\|}^{2}$

5:S4: Compute geometric VLAD $v_{i}^{j}$:

6:$v_{i}^{j}=\left\\{\begin{array}[]{l
l}\sum_{\mathbf{x^{\theta}}:NN(\mathbf{x})=i}\mathbf{x}^{\theta}-\mu_{i}&\quad\text{if
$\psi(\theta(\mathbf{x}))=j$}\\\ \mathbf{0}^{d}&\quad\text{if
$\psi(\theta(\mathbf{x}))\neq j$}\end{array}\right.$

7: $V_{i}=[v_{i}^{1},v_{i}^{2},\cdots,v_{i}^{M-1},v_{i}^{M}]$

8: $\mathcal{V}=[V_{1},V_{2},\cdots,V_{K-1},V_{K}]$

9:S5: Codebook Adaptation:

10:
$\hat{\mu_{i}}=\frac{1}{N}\sum_{t=1}^{N}\gamma_{i}(t),\mathbf{x}^{\theta}(t)\in
T$

11:S6: Intra-normalization, Inter-Z-score normalization, and L2-normalization.

12:S7: PCA whitening:

13:
$\tilde{\mathcal{V}_{l}}=\textbf{P}(:,1:\rho)^{T}\times\bar{\mathcal{V}_{l}}$

Figure 3: The pipeline of the geometric VLAD (gVLAD) descriptor computation.

##  4 Experiments & Evaluations

###  4.1 Benchmark Dataset

We evaluate the proposed approach on several public available benchmark
datasets: Oxford buildings, Paris and Holidays. Large scale experiments are
conducted on these datasets by adding $1$M Flickr images as distractors [7].
For each of these datasets, performance is measured by mean average precision
(mAP) over a set of pre-defined queries and their annotated ground truth
matches.

Holidays Dataset: Holidays dataset [7] contains $1491$ high resolution
personal holiday photos with $500$ annotated queries. For large scale
experiments, $1$ million Flickr images are added to it to create Holidays +
Flickr 1M dataset. About 5%-10% of the images in holiday dataset have
orientations which are unnatural for human observer [12]. We manually rotate
these images to create Rotated Holidays dataset.

Oxford Dataset: This dataset, Oxford 5K contains $5062$ images of Oxford
buildings gathered from Flickr [14]. There are 55 query images each with a
rectangular bounding box specifying the region of interest. To test large
scale retrieval, it is firstly extended with a 100K Flickr
images111http://www.robots.ox.ac.uk/ vgg/data/oxbuildings/flickr100k.html to
create Oxford 105K  dataset. We further extend the dataset with 1 million
Flickr image222http://press.liacs.nl/mirflickr/ creating Oxford 5K + Flickr 1M
dataset.

Paris Dataset: The Paris Dataset [15] Paris 6K consists of $6412$ images
collected from Flickr by searching for particular Paris landmarks. There are
$60$ query images, each with a rectangular bounding box specifying the region
of interest. We found that both 100K Flickr images and Flickr 1M images
contains a large number of Paris landmarks, hence we do not extend the Paris
dataset with Flickr images.

###  4.2 Implementation Details

Descriptor computation: The pipeline of computing gVLAD descriptor is
characterized in Figure 3. First, all images are resized to $1024\times 768$.
We find that when using the original resolution of Holidays images, the
performance is inferior to the down-sampled images. We can also benefit from
the speed when using smaller images. In Oxford and Paris datasets, bounding
boxes are provided for queries. We only extract descriptors inside bounding
boxes. We use the SIFT and SURF implementations in OpenCV333http://opencv.org/
to detect keypoints and extract descriptors. Each SIFT descriptor has 128
dimensions and each SURF descriptor has 64 dimensions. We find that VLAD based
features have better performance using SURF keypoints and descriptors [3] than
SIFT keypoints and descriptors [11]. In general, we observed about 10%
improvement using SURF as compared to SIFT. More details about the performance
difference can be seen by comparing results in Table 2 and Table 3.

Angle Membership Function: The angle distribution of SURF keypoints from
Holidays dataset is shown in Figure 4 (a). We find that majority keypoints
have vertical or horizontal angles as detectors have larger response at these
points, resulting in roughly 4 centers ($\pi/2$, $\pi$, $2\pi/3$, $2\pi$). To
learn the membership function of each keypoint angle, we apply the proposed
approach in 3.2. Because larger number of bins will increase the dimension of
the final derived gVLAD feature, to gain a reasonable representation as well
as low dimensionality, we set the number of angle bins to be $4$ to fit the
distribution. A $\pi/4$ offset and a set of evenly distributed bins:
$[-\pi/4,\pi/4)$, $[\pi/4,3\pi/4)$, $[3\pi/4,5\pi/4)$ and $[5\pi/4,7\pi/4)$,
are automatically estimated from the algorithm, which is visualized in Figure
4 (b). We use this angle bin partition in following evaluations. We had also
experimented using different number of bins and offset on Rotated Holidays
dataset. We observe increasing performance as more bins are used, as shown in
Figure 5. This is because that increasing number of bins is equivalent to
increasing number of subspaces, by which the distance of descriptor towards
centroid can be computed in a more discernible way. However, gains by using 5
or 6 bins as compared to the predicted angle partition ( 4 bins with $\pi/4$
offset) by propose algorithm are marginal, also our learned setting has much
smaller dimensions.

|
---|---
(a) | (b)
Figure 4: (a) Distribution of keypoint angles from Holidays dataset. (b)
Learnt 4 angle bins with $\pi/4$ offset. Figure 5: Performance using different
angle bins and offset on Rotated Holidays dataset. Based on empirical
observation, we apply angle membership learning with $M=4$, and use the
learned partition in our experiment. Such setting gives near best performance
but lower dimension of gVLAD features as compared to using more bins for
modeling angle distribution.

Vocabulary Generation: The vocabulary consisting of $K=256$ visual words is
computed from all SURF descriptors on Paris dataset. Various different cluster
initializations of $k$-means are executed and the best clustering is used as
the vocabulary for all evaluations. As the number of extracted descriptors is
typically much larger than $K$, e.g. even the smallest Holidays dataset
contains 8.3 million SURF descriptors. This vocabulary can be considered
independent from all datasets. Such simplification has been used in literature
[2]. For every dataset, this vocabulary is used as a reference vocabulary and
a vocabulary adaptation is performed as described in section 3.3.

Retrieval: During retrieval, L2 distance is computed to rank images with
respect to input query. Since our focus is generating a compact and efficient
image descriptor, to illustrate the power of the proposed descriptor, we use
brute-force distance computation to report our results. However, our proposed
descriptors can in principle be used with approximate distance matching or
other hashing based techniques as well, which is beyond the scope of this
paper.

###  4.3 Performance Evaluation & Analysis

The performance in all retrieval experiments is evaluated using the mean
average precision (mAP), which is defined as the mean of the average precision
over all the queries given a dataset. Average Precision is computed as the
average of the precision value obtained for the set of top $k$ images after
each relevant image is retrieved. We use standard evaluation packages obtained
from the data websites.

The Power of Angle: To illustrate the power of angle, we performed a simple
experiment which uses only the angle information from each keypoint to
retrieve similar images. After obtaining the keypoints and the angle of each
keypoint, we generate an angle histogram for each of image by binning all
angles into $Q$ bins. We use L2 distance to compute the similarity between
angle descriptors. Table 1 shows the retrieval results on Rotated Holidays
dataset. It can be seen that surprisingly using only angle information
(without any appearance information from SURF or other descriptors), we can
still achieve about $26.9\%$ mAP results. Note that the dimension of the angle
bin histogram for the best result is only $72$, which is a much smaller number
compared to conventional BOW or VLAD descriptors.

|  Angle Bins
---|---
$Q$ | 2 | 4 | 8 | 18 | 36 | 72
mAP | 0.015 | 0.037 | 0.149 | 0.241 | 0.261 | 0.269
Table 1: Retrieval performance on Rotated Holidays dataset using only Angle
binning histogram with varying dimensions, no appearance information is used.
Given only $72$ dimension of angle histogram, a surprising $26.9\%$ mAP result
is achieved.

Step-by-Step Performance Evaluation: To show the performance gain obtained
from each of the proposed steps, we performed a step-by-step experiment on
Rotated Holidays dataset and baselined it with VLAD performance. Results are
listed in Table 3. All results use SURF detector and SURF descriptor. It can
be seen that by adding inter-Z-Score normalization to the original VLAD, the
performance is increased by $5.4\%$. Performing Angle Binning over VLAD leads
to a gain of $7.3\%$. By combining both Angle Binning and Z-Score
normalization, we achieve $14.7\%$ improvement over VLAD representation.
Performing vocabulary adaptation for Rotated Holidays dataset provides
additional $3.8\%$ performance gain. Finally, PCA whitening is applied which
is able to reduce the dimension significantly with only about $1.1\%$
performance loss, as compared to PCA without whitening having a loss of
$3.6\%$. To demonstrate the performance of low-dimensional gVLAD descriptor
using PCA whitening, we further plot the mAP performance curve by varying
$\rho=2^{4}$ to $2^{16}$ in Figure 6. It can be seen that with only $32$
dimensions, the performance by the proposed descriptor can reach to
$mAP=0.737$, which already outperforms the original VLAD descriptor using
$1024$ visual words with $65,536$ dimension ($mAP=0.670$) as shown in Table 3.

We also test our proposed method using SIFT detectors and root SIFT [1]
descriptors, since most previous published work use SIFT. For fair comparison,
we implement VLAD with root SIFT descriptors, which have better performance
compared with 0.526 on Holidays dataset reported in [8]. Results as shown in
Table 2 demonstrate the superior performance of proposed approach over SIFT
descriptors as well. As noted, comparing Tabel 2 and Table 3, we observe in
general that using SURF descriptors outperforms SIFT based descriptors.

Figure 6: Dimension reduction on original gVLAD descriptor using PCA whitening. The original feature dimension is 65,536. After compressed to 128-D, the mAP decreases only about 1%. Dataset |  VLAD* | gVLAD
---|---|---
Holidays | 0.548 | 0.710
Rotated Holidays | 0.550 | 0.786
Table 2: Comparison of our proposed gVLAD with VLAD on benchmark datasets. Both VLAD and gVLAD use SIFT detectors and root SIFT descriptors. $256$ visual words are used. The feature dimension of VLAD is $256\times 128=32768$, and for gVLAD is $256\times 128\times 4=131072$. * denotes our implementation. Method | Dimension | mAP
---|---|---
VLAD ($K=256$)* [9] | 16,384 | 0.662
VLAD ($K=1024$)* [9] | 65,536 | 0.670
VLAD ($K=256$) + inter-norm | 16,384 | 0.716
VLAD ($K=256$) + Angle Binning | 65,536 | 0.735
\+ inter-norm | 65,536 | 0.809
\+ Voc Adaptation | 65,536 | 0.847
PCA | 128 | 0.811
PCA + whitening | 128 | 0.836
Table 3: mAP on the Rotated Holidays dataset comparing to start-of-art
results. Best performances are in bold. *VLAD result in this table are based
on our implementation. All results use SURF detector and SURF descriptor.

Full Size & Low Dimensional gVLAD Descriptors: We compared our proposed method
with several benchmark results in [9, 16, 13, 2] for both full size descriptor
and dimension reduced descriptor ($\rho=128$). Experiments are done using both
Holidays dataset and Oxford 5K dataset. Table 4 shows that the proposed
approach significantly outperforms the state-of-the-art performance by
approximately $16.6\%$ and $7\%$ on Holidays and Oxford 5K dataset
respectively. For low dimensional case, as shown in Table 5, our algorithm
outperforms the best state-of-art result by $15\%$ on both datasets.

Comparing Table 4 and 5, results also show that the proposed gVLAD descriptor
is quite powerful in the sense that even with PCA whitening and reduced
dimension, it can still achieve better result as compare to the best benchmark
results with full size descriptors. In addition, PCA whitening based dimension
reduction only results in small amount of performance decrease which is about
$2.76\%$ in average of both datasets, and $1.1\%$ in the best case (Rotated
Holidays).

Method | Dimension | Holidays | Oxford
---|---|---|---
BoW 20k-D [9] [16] | 20,000 | 0.452 | 0.354
BoW 200k-D [9] [16] | 200,000 | 0.540 | 0.364
Improved Fisher [13] | 16,384 | 0.626 | 0.418
VLAD [8] | 8,192 | 0.526 | -
VLAD + SSR [9] | 16,384 | 0.598 | 0.378
Improved VLAD + SSR [2] | 32,768 | - | 0.532
VLAD + intra-norm [2] | 32,768 | 0.646 | 0.555
Ours | 65,536 | 0.812 | 0.626
Table 4: mAP performance by full size gVLAD descriptors as compared to state-of-the-art results on Holidays and Oxford. Existing approaches are based on SIFT descriptors, while the proposed gVLAD descriptor uses SURF detector and SURF descriptor. Best performances are in bold. Method | Holidays | Oxford
---|---|---
GIST [9] | 0.365 | -
BoW [9, 16] | 0.452 | 0.194
Improved Fisher [13] | 0.565 | 0.301
VLAD [8] | 0.510 | -
VLAD + SSR [9] | 0.557 | 0.287
Multivoc-BoW [6] | 0.567 | 0.413
Multivoc-VLAD [6] | 0.614 | -
VLAD + intra-norm [2] | 0.625 | 0.448
Ours | 0.779 | 0.600
Table 5: mAP performance by gVLAD low dimensional descriptors ($\rho=128$):
comparison with state-of-the-art on the Holidays and Oxford 5k benchmarks. The
existing approaches are based on SIFT descriptors, while the proposed gVLAD
descriptor uses SURF detector and descriptor. Best performances are in bold.

Performance on Large Scale Dataset: We scale the proposed algorithm to large
scale image dataset with millions of images, and test on both using full size
gVLAD and PCA dimension reduced 128-D descriptors. In total, $4$ large scale
datasets are used, including Holidays + Flickr 1M, Rotated Holidays + Flickr
1M, Oxford 105K, and Oxford 5K + Flickr 1M. As can be seen from Table 6, our
methods outperform all current state-of-the-art methods. For example, using
dimension reduced 128-D gVLAD descriptors, on Holidays + Flickr 1M dataset,
our algorithm outperforms the best result [2] reported in literature with a
significant gain of $22.8\%$. On Oxford 105K dataset, we are able to achieve
$11.6\%$ better result than [2].

Further, same with our previous observation in Table 5 as compared to Table 4,
Table 6 also shows performance only drops very slightly using the proposed PCA
whitening. This implies that the proposed gVLAD descriptor is quite powerful.
Also, being combined with proper dimension reduction schema, effective
representation with computational efficiency can be achieved.

| State of the Art | State of the Art | Ours | Ours | Ours
---|---|---|---|---|---
Dataset | Original Dimension | 128-D | Original Dimension | 128-D | Loss in PCA
Holidays |  0.646 [2] |  0.625 [2] | 0.812 | 0.779 | 0.033
Holidays + Flickr 1M | - |  0.378 [2] | - | 0.607 | -
Rotated Holidays | - | - | 0.847 | 0.836 | 0.011
Rotated Holidays + Flickr 1M | - | - | - | 0.654 | -
Oxford 5K |  0.555 [2] |  0.448 [2] | 0.626 | 0.600 | 0.026
Oxford 105K | - |  0.374 [2] | - | 0.490 | -
Oxford 5K + Flickr 1M | - | - | - | 0.438 | -
Paris 6K |  0.494 [15] | - | 0.631 | 0.592 | 0.039
Table 6: mAP Performance on large scale datasets: comparisons with benchmark
results. Best performances are in bold.

###  4.4 Time Complexity and Memory Footprint

Each image takes 512 bytes in memory after being converted to 128 dimensional
gVLAD feature vector by PCA compression. The largest dataset (Holidays +
Flickr 1M) in our experiment occupies $0.5$GB of RAM for keeping all features
in memory. To evaluate the time complexity of each step in the proposed gVLAD
computation, we conduct experiments on this dataset using a Ubuntu machine
with two Xeon X5675 CPUs at 3.07GHz, with 12 physical cores and 24 logical
cores in total. We rely on multi-threading whenever possible. Table 7
illustrates the average results on $10$ randomly selected queries. As shown,
our proposed technique takes approximately $100$ millisecond to compute gVLAD
representation, and $750$ millisecond to perform an end-to-end brute-force
retrieval over the entire inventory. Since our proposed descriptors can in
principle be used with other approximate distance matching or indexing schema,
better retrieval speed can be expected, which will be very useful in practical
applications.

Process | Mean $\pm$ std. (ms)
---|---
SURF detection & description |  373.5 $\pm$ 69.1
gVLAD computation |  71.7 $\pm$ 20.3
PCA compression |  28.0 $\pm$ 3.6
Nearest neighbor search |  266.7 $\pm$ 36.3
Table 7: Speed analysis based on 10 random query images from Holidays + Flickr
1M dataset.

##  5 Conclusion

We present gVLAD which is a novel extension of popular VLAD descriptor for
large scale image search. Our proposed descriptor extends VLAD by integrating
weak geometric cues in form of key point angles. A principled technique to
represent this information as membership function over angles is also
presented. The vocabulary adaptation and inter-Z-score normalization are also
proposed to improve the performance of the system. Extensive experiments are
conducted on existing publicly available benchmark datasets which demonstrate
the superior performance of our approach. Our future work focuses on exploring
efficient indexing strategies to avoid the brute-force matching of images. We
are also investigating other related low level information that can be further
integrated into gVLAD to make the representation more powerful.

## References

  * [1] R. Arandjelovic and A. Zisserman.  Three things everyone should know to improve object retrieval.  In CVPR, 2012, pages 2911–2918. IEEE, 2012.
  * [2] R. Arandjelovic and A. Zisserman.  All about vlad.  In CVPR 2013, 2013.
  * [3] H. Bay, T. Tuytelaars, and L. Van Gool.  Surf: Speeded up robust features.  In ECCV 2006, pages 404–417. Springer, 2006.
  * [4] O. Chum and J. Matas.  Large-scale discovery of spatially related images.  Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(2):371–377, 2010.
  * [5] O. Chum, M. Perdoch, and J. Matas.  Geometric min-hashing: Finding a (thick) needle in a haystack.  In CVPR 2009, pages 17–24. IEEE, 2009.
  * [6] H. Jégou and O. Chum.  Negative evidences and co-occurences in image retrieval: The benefit of pca and whitening.  In ECCV 2012, pages 774–787. Springer, 2012.
  * [7] H. Jegou, M. Douze, and C. Schmid.  Hamming embedding and weak geometric consistency for large scale image search.  In ECCV 2008, pages 304–317. Springer, 2008.
  * [8] H. Jégou, M. Douze, C. Schmid, and P. Pérez.  Aggregating local descriptors into a compact image representation.  In CVPR 2010, pages 3304–3311. IEEE, 2010.
  * [9] H. Jégou, F. Perronnin, M. Douze, C. Schmid, et al.  Aggregating local image descriptors into compact codes.  Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(9):1704–1716, 2012.
  * [10] S. Lazebnik, C. Schmid, and J. Ponce.  Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories.  In CVPR 2006, volume 2, pages 2169–2178. IEEE, 2006.
  * [11] D. G. Lowe.  Object recognition from local scale-invariant features.  In Computer Vision, 1999. Proceedings. Seventh IEEE International Conference on, volume 2, pages 1150–1157. IEEE, 1999.
  * [12] M. Perdoch, O. Chum, and J. Matas.  Efficient representation of local geometry for large scale object retrieval.  In CVPR 2009, pages 9–16. IEEE, 2009.
  * [13] F. Perronnin, Y. Liu, J. Sánchez, and H. Poirier.  Large-scale image retrieval with compressed fisher vectors.  In CVPR 2010, pages 3384–3391. IEEE, 2010.
  * [14] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman.  Object retrieval with large vocabularies and fast spatial matching.  In CVPR 2007, pages 1–8. IEEE, 2007.
  * [15] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman.  Lost in quantization: Improving particular object retrieval in large scale image databases.  In CVPR 2008, pages 1–8. IEEE, 2008.
  * [16] J. Sivic and A. Zisserman.  Video google: A text retrieval approach to object matching in videos.  In 9th IEEE International Conference on Computer Vision, pages 1470–1477. IEEE, 2003.
  * [17] Y. Zhang, Z. Jia, and T. Chen.  Image retrieval with geometry-preserving visual phrases.  In CVPR 2011, pages 809–816. IEEE, 2011.