# A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured
Sentiment Analysis

1st Qi Zhang  School of Computer Science and Technology
East China Normal University
Shanghai, China
<EMAIL_ADDRESS>2nd Jie Zhou  Jie Zhou is the corresponding author of
this paper. School of Computer Science
Fudan Univerisity
Shanghai, China
<EMAIL_ADDRESS>3rd Qin Chen  School of Computer Science and
Technology
East China Normal University
Shanghai, China
<EMAIL_ADDRESS>4th Qingchun Bai  Shanghai Open University
Shanghai, China
<EMAIL_ADDRESS>5th Jun Xiao  Shanghai Open University
Shanghai, China
<EMAIL_ADDRESS>6th Liang He  School of Computer Science and Technology
East China Normal University
Shanghai, China
<EMAIL_ADDRESS>
###### Abstract

Structured sentiment analysis, which aims to extract the complex semantic
structures such as holders, expressions, targets, and polarities, has obtained
widespread attention from both industry and academia. Unfortunately, the
existing structured sentiment analysis datasets refer to a few languages and
are relatively small, limiting neural network models’ performance. In this
paper, we focus on the cross-lingual structured sentiment analysis task, which
aims to transfer the knowledge from the source language to the target one.
Notably, we propose a Knowledge-Enhanced Adversarial Model (KEAM) with both
implicit distributed and explicit structural knowledge to enhance the cross-
lingual transfer. First, we design an adversarial embedding adapter for
learning an informative and robust representation by capturing implicit
semantic information from diverse multi-lingual embeddings adaptively. Then,
we propose a syntax GCN encoder to transfer the explicit semantic information
(e.g., universal dependency tree) among multiple languages. We conduct
experiments on five datasets and compare KEAM with both the supervised and
unsupervised methods. The extensive experimental results show that our KEAM
model outperforms all the unsupervised baselines in various metrics.

###### Index Terms:

cross-lingual, structured, sentiment analysis, adversarial, knowledge

##  I Introduction

Sentiment analysis is a fundamental task of natural language processing (NLP),
which aims to determine the emotion, sentiment, and opinions about a product,
attribute, target, or policy [1, 2]. Recently, more and more researchers focus
on fine-grained sentiment analysis task, such as aspect-based sentiment
analysis [3, 4], targeted sentiment analysis [5], and structured sentiment
analysis [6]. In this paper, we focus on the task of structured sentiment
analysis, which consists of four sub-tasks: 1) expression extraction, 2)
target extraction, 3) holder extraction, and 4) sentiment prediction. As shown
in Fig. 1, “The Sadc ministerial observer team” (holder $h$) expresses a
positive sentiment (polarity $p$) towards “President Mugabe” (target $t$) with
“congratulated” (expression $e$).

Figure 1: An example of structured sentiment analysis. We translate the
Spanish into English in the brackets for ease of understanding.

Unfortunately, the existing publicly available datasets for structured
sentiment analysis refer to a few languages [6] and are relatively small,
which limit the effectiveness of neural network models. The annotations for
structured sentiment analysis are time-consuming and laborious because they
are structured and require a rich label space. Furthermore, it is challenging
to extract complex semantic structures such as expressions, holders, targets,
and sentiments, especially for corpus written in under-annotated and under-
resourced languages. To address this problem, we focus on a cross-lingual
structured sentiment analysis task, which aims to translate the structural
sentiment knowledge from the source language (e.g., English) to the target
language (e.g., Spanish).

Previous works about structured sentiment analysis focus on modeling the
structure and sentiment representation in mono-lingual [6, 5]. As a specifical
case of fine-grained sentiment analysis task [3], neural-based pipeline and
joint algorithms are proposed for this tasks [7, 8]. They extracted the
targets, expressions, and sentiments by converting these tasks into a sequence
labeling task. Barnes et al. [6] proposed to model the relationships among the
subtasks by regarding this task as graph parsing. Despite their effectiveness,
these studies are insufficient to yield satisfactory performance for cross-
lingual structure sentiment analysis. The key reason is the typological
differences (e.g., syntactic dependencies) between the source and target
languages.

Recently, multi-lingual pre-trained models such as mBERT [9] and XLM [10] have
increasingly attracted attention in various polyglot tasks [11, 12]. Mulcaire
et al. [13] adopted multi-lingual contextualized word embedding for structure
extraction tasks, such as semantic role labeling, dependency parsing, and
named-entity recognition. However, there are many versions of multi-lingual
pre-trained models trained on different datasets with different training
strategies so that these models contain a lot of diverse semantic and syntax
information. For example, there are more than 170 cross-lingual pre-trained
models on the
website111https://huggingface.co/models?sort=downloads&search=multilingual.
Thus, the first question (Q1) we want to solve is “How to utilize these models
for cross-lingual structured sentiment analysis?”.

Furthermore, the existing studies have shown that the syntax structure (e.g.,
universal dependency tree, POS) is helpful for cross-lingual transfer [14,
15]. Subburathinam et al. [14] integrated syntax into event extraction and
relation extraction and obtained a large improvement. Also, De et al. [16]
released the language-universal dependency parsing across 83 languages. This
explicit syntax structure is similar among different languages since it is
labeled based on the linguists’ syntactic principles [14]. Also, the
dependency tree structure plays a significant role in structured sentiment
analysis since it can help model the relationships among the elements in the
text. Inspired by these findings, the second question (Q2) we want to solve is
“How to integrate the explicit structured knowledge (e.g., universal
dependency tree) into cross-lingual structured sentiment analysis models?”

To answer the above two questions, we propose a Knowledge-Enhanced Adversarial
Model (KEAM) to integrate both the implicit distribution and explicit
structure knowledge into cross-lingual structured sentiment analysis. For Q1,
we design an adversarial embedding adapter to obtain a robust and semantically
rich word embedding from multiple pre-trained cross-lingual embedding
adaptively. Particularly, a word-level attention mechanism is proposed to
capture the critical semantic information from multiple embeddings, and an
adversarial strategy is used to improve the robustness of the embedding. For
Q2, we construct a cross-lingual structure transfer learning method by
integrating language-universal syntax tree via a graph convolution network
(GCN) [17], namely Syntax-GCN encoder. Finally, we obtained the structured
representation used for expression extraction, target extraction, holder
extraction, and sentiment classification. In a word, to project the source
language and target language into the common semantic space, we propose to
share structural representations based on various pre-trained embeddings for
cross-lingual structured sentiment analysis.

We conduct extensive experiments on five datasets that refer to four
languages. We compare our KEAM model with four supervised and four
unsupervised baselines. The proposed model outperforms all the unsupervised
methods over five datasets in terms of token-level F1 and target-level F1. The
ablation studies show that both the adversarial embedding adapter and syntax-
GCN encoder can improve the performance effectively. Also, we run the transfer
experiments over 25 tasks to exploit what has been learned in one language to
improve generalization in another.

The main contributions of this paper are summarized as follows.

  * •

We propose a knowledge-enhanced adversarial model for cross-lingual structured
sentiment analysis by integrating both the implicit multiple embeddings and
explicit syntax information.

  * •

We design an adversarial embedding adapter to combine the multiple embeddings
adaptively using a word-level mechanism and an adversarial strategy. Moreover,
a syntax GCN is designed to integrate explicit structural knowledge.

  * •

A series of experiments on five datasets that refer to four languages show the
great advantage of our KEAM model by comparing with both supervised and
unsupervised baselines in terms of various metrics.

##  II Related Work

The key challenges for cross-lingual structured sentiment analysis tasks are
complex structural sentiment extraction and cross-lingual transfer. Thus, we
mainly review the most related works about fine-grained sentiment analysis and
cross-lingual structure extraction tasks.

###  II-A Fine-grained Sentiment Analysis

Fine-grained sentiment analysis plays an important role in sentiment analysis,
including targeted sentiment analysis (TSA) [5], aspect-based sentiment
analysis (ABSA) [3, 18, 19], and structured sentiment analysis [6, 20]. TSA
consists of two subtasks, target extraction, and polarity classification [5].
This task aims at inferring fine-grained sentiment polarity towards the
targets in the given text. Thus, detecting the targets’ sentiment based on the
identified target-related contexts in the text is the biggest challenge for
this task. In recent years, neural networks have been proven effective in
modeling the relationships between the target and its contexts. To take the
expression into account, ABSA is proposed to summarize users’ opinions towards
specific aspects in a sentence. ABSA not only predicts the aspect (target) and
its sentiment polarities but also extracts the terms of the opinion towards
the given aspect. He et al. [21] proposed a joint approach (IMN) via a multi-
task learning framework without the relationships between the target and
expression. Then, relation-aware collaborative learning (RACL) is designed to
consider the relatedness [22].

Moreover, the structured sentiment analysis task is proposed aims to extract
the holders, targets, expressions, and polarities w.r.t. the given targets
from the unstructured text. Zhang et al. [23] presented a transition-based
end-to-end method to extract the elements (e.g., holders, targets, and
expressions) with their relationships. However, the sentiment information of
the target is ignored by this work. Barnes et al. [6] regarded the structured
sentiment analysis as a dependent parsing problem to model the relatedness
among the holder, target, expression, and polarity. However, all these works
focus on monolingual structured sentiment analysis. Due to the complexity of
this task, the existing datasets refer to a few languages and are relatively
small, which largely limits the effectiveness of the neural network models. In
this paper, we focus on cross-lingual structured sentiment analysis tasks.

###  II-B Cross-lingual Structure Extraction

The goal of structure extraction (e.g., information extraction, event
extraction) is to extract the structural information (e.g., entities,
relations, and events) from the unstructured text. Recently, cross-lingual
transfer methods are proposed for structure extraction [14, 24]. Most previous
efforts working with cross-lingual structure transfer rely on parallel data
[25], language-specific characteristics [26], or machine translation [27],
which requires extra resources or tools. Feng et al. [28] and Xie et al. [29]
applied cross-lingual transfer to sequence labeling task without considering
the complex structures. Wang et al. [30] exploited common distribution
representation space across languages for relation extraction.

Recently, multi-lingual pre-trained models (e.g., mBERT [9] and XLM [10])
obtained great success for cross-lingual transfer [31]. Liu et al. [32] and
Mulcaire et al. [13] applied multi-lingual contextualized word embedding for
cross-lingual tasks, such as semantic role labeling, dependency parsing and
named entity recognition. However, all these researches utilized one kind of
word embedding one time. In fact, there are many multi-lingual pre-trained
models with various semantic information since there are trained on various
datasets with various optimization objectives. Thus, we propose an adversarial
embedding adapter to combine the multiple embeddings adaptively using a word-
level mechanism and an adversarial strategy. Inspired by the existing work
[31], we incorporate universal dependencies [33] for cross-lingual structured
sentiment analysis. Particularly, the syntax parsing tree is modeled to
construct multi-lingual structural representations via a GCN, which has been
successfully applied to several individual monolingual NLP tasks, including
relation extraction [17], event detection [34], ABSA [4].

Figure 2: The framework of our KEAM model. It consists of three parts:
adversarial embedding adapter, syntax GCN encoder, and decoder.

##  III Our KEAM Model

In this paper, we propose a knowledge-enhanced adversarial model for cross-
lingual structured sentiment analysis by integrating both the implicit
multiple embeddings and explicit syntax information. Fig. 2 shows the
framework of our KEAM model, which consists of three components, adversarial
embedding adaptor, syntax GCN encoder, and decoder. First, an adversarial
embedding adaptor is proposed to learn an informative and robust word
embedding from multiple cross-lingual pre-trained embeddings adaptively (See
Section III-B). Then, we design a syntax GCN encoder for learning the
structural representation across different languages (See Section III-C).
Finally, we perform elements extraction and sentiment classification tasks
based on the sentence representation learned by the above two parts (See
Section III-D).

###  III-A Task Definition

The formulate definition of cross-lingual structured sentiment analysis is
given as follows. Given a training corpus in a source language, denoted by
$\mathcal{D}_{s}=\\{x_{1},...,x_{|\mathcal{D}_{s}|}\\}$, where
$|\mathcal{D}_{s}|$ is the number of training samples in source language. For
each sentence $x=\\{w_{1},...,w_{i},...,w_{|x|}\\}\in\mathcal{D}_{s}$, the
labels $y$ is a set of opinion tuples $O=\\{o_{1},...,o_{|O|}\\}$. Each
opinion $o_{k}$ is a tuple $(h_{k},t_{k},e_{k},p_{k})$, where $h$ is a holder
who expresses a polarity $p$ towards a target $t$ through a sentiment
expression $e$. Note that $h_{k}=\\{w_{h^{s}_{k}},...,w_{h^{e}_{k}}\\}$,
$t_{k}=\\{w_{t^{s}_{k}},...,w_{t^{e}_{k}}\\}$,
$e_{k}=\\{w_{e^{s}_{k}},...,w_{e^{e}_{k}}\\}$ are a sub-sequence of the
sentence $x$, where ${h^{s}_{k}}$ (${h^{e}_{k}}$), ${t^{s}_{k}}$
(${t^{e}_{k}}$), ${e^{s}_{k}}$ (${e^{e}_{k}}$) are the start (end) index of
the holder, target and expression, respectively. $p_{k}\in\\{P,N\\}$ denotes
the positive, and negative sentiment polarities towards the given target.
Thus, cross-lingual structured sentiment analysis can be divided into four
substasks: 1) expression extraction (EE) extracts the expressions $e$ in the
sentence; 2) target extraction (TE) extracts the target $t$ towards the given
expression $e$; 3) holder extraction (HE) extracts the holder $h$ towards the
given expression $e$; 4) sentiment classification (SC) predicts the sentiment
polarity $p$ towards the given expression $e$. Finally, we aim to extract the
tuple for the given sentence in the unlabelled corpus $\mathcal{D}_{t}$ in the
target language.

###  III-B Adversarial Embedding Adapter

In this section, we aim to learn an informative and robust word embedding for
cross-lingual transfer. First, we design a word-level attention mechanism to
capture the vital distribution implicit semantic in the multiple embeddings
pre-trained on a different corpus with different training strategies and
tasks. Then, we adopt an adversarial training strategy to improve the
robustness of the embedding.

####  III-B1 Word-level Attention

Multi-lingual pre-trained language models, such as mBERT [9] and XLM [10] has
been widely used in different cross-lingual tasks and obtained great success.
In fact, the developer and researchers have released a large number of multi-
lingual pre-trained language models. There are more than 100 models available
on the website Hugging Face222https://huggingface.co/. These models have
different semantic information since they are trained on different large-scale
datasets using different objectives and settings. For example, mBERT-base-
cased and mBERT-base-uncased are trained on the top 104 languages with the
largest Wikipedia using a masked language modeling (MLM) objective based on
the lowercased or unlowercased text [9]. XLM-RoBERTa model is pre-trained on
the 100 languages with 2.5TB of filtered CommonCrawl data. Moreover, [35]
finetuned XLM-RoBERTa models on named-entity-recognition (NER) task with
multilingual datasets. However, most existing works use one of these models as
word embedding. To obtain a better word representation, we design a word-level
attention mechanism to combine the multiple embeddings adaptively.

Suppose we have $|\mathcal{M}|$ cross-lingual pre-trained models
$\mathcal{M}=\\{\mathcal{M}^{1},\mathcal{M}^{1},...,\mathcal{M}^{|\mathcal{M}|}\\}$,
we input the sentence $x$ into each of them and obtained the token-level
embeddings:

$E^{k}=\mathcal{M}^{k}(x)$ (1)

where $E_{k}=\\{e^{k}_{1},e^{k}_{2},...,e^{k}_{|x|}\\}$ is the vector
representations obtained by model $\mathcal{M}_{k}$ for sentence $x$ and
$e^{k}_{i}$ means the word $w_{i}$’s embedding. Note that the sub-words of a
word $x_{i}$ may be different for the different models since the tokenizers of
these models are not consistent. Thus, we use the average of the sub-words
embeddings in a word as the final word embedding.

Then, we design a word-level attention mechanism to combine the embedding sets
$\\{E^{1},E^{2},...,E^{|\mathcal{M}|}\\}$ into one embedding
$E=\\{e_{1},e_{2},...,e_{|x|}\\}$. We calculate the attention weight in word
level because the important dimensions for different words are different. For
example, the expression words focus on the sentiment information and the
target words focus on the entity information. Formally, the attention weight
is computed as:

$\displaystyle e_{i}=\sum_{k=1}^{|\mathcal{M}|}{\alpha^{k}_{i}\times
e^{k}_{i}}$ (2)
$\displaystyle\alpha^{k}_{i}=\rm{softmax}(v^{T}_{a}tanh(W_{a}e^{k}_{i}+b_{a}))$

where $v_{a}$, $W_{a}$, and $b_{a}$ are the trainable parameters.

####  III-B2 Adversarial Training

The typological and semantic gaps between the source and target language lead
to the instability of the cross-lingual transfer models. Thus, to improve the
robustness of the word embedding, we apply adversarial training [36, 37] to
input word embedding space for cross-lingual transfer. The existing studies
have shown that adversarial training is a novel regularization algorithm that
improves the robustness by adding small disturbance to inputs [38].

Let $r=\\{r_{1},r_{2},...,r_{|x|}\\}$ be a perturbation for sentence $x$,
where $r_{i}$ is the perturbation vector for the word $w_{i}$. We introduce
$\hat{e}=x_{+r}=\\{\hat{e}_{1},\hat{e}_{2},...,\hat{e}_{|x|}\\}$ that denotes
$x$ with additional small perturbations $r$, where $\hat{e}_{i}=e_{i}+r_{i}$.
To obtain worst-case perturbations $r_{adv}$, adversarial training aims to
minimize the log-likelihood,

$r^{\text{adv}}=\underset{r,\|r\|\leq\epsilon}{\operatorname{argmax}}\left\\{\ell\left(x_{+r},y\right)\right\\}$
(3)

where $\epsilon$ is a hyper-parameter that controls the norm of the
perturbation and $r^{adv}=\\{r_{1}^{adv},r_{2}^{adv},...,r_{|x|}^{adv}\\}$. It
is generally infeasible to exactly estimate $r^{adv}$ in Equation 3 for
sophisticated deep neural models. As a solution, an approximation method was
proposed by linearizing $(x,y)$ around $x$ [36]. The approximation method
induces the following non-iterative solution for calculating $r_{i}^{adv}$ for
all $i$.

${r}^{\text{adv}}_{i}=\frac{\epsilon{g}_{i}}{\|{g}\|_{2}},\quad{g}_{i}=\nabla_{{e}_{i}}\ell(x,y)$
(4)

where $g$ is the concatenation of $g_{i}$ for all $i$.

Then based on adversarial perturbation $r^{adv}$, the loss function for
adversarial training is computed,

$\mathcal{L}_{\text{adv}}=\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}}\ell\left(x_{+r^{\text{adv}}},y\right)$
(5)

where
$\hat{e}^{adv}=x_{+r^{adv}}=\\{\hat{e}^{adv}_{i},\hat{e}^{adv}_{2},...,\hat{e}^{adv}_{|x|}\\}$
where $\hat{e}^{adv}_{1}=e_{i}+r^{adv}_{i}$.

Finally, we jointly minimize objective functions $\mathcal{L}_{\text{task}}$
and $\mathcal{L}_{adv}$

$\mathcal{L}=\mathcal{L}_{\text{task}}+\lambda\mathcal{L}_{\text{adv }}$ (6)

where $\mathcal{L}_{\text{task}}$ is the loss function of the structured
sentiment analysis (e.g., $\mathcal{L}_{\text{exp}}$,
$\mathcal{L}_{\text{target}}$, $\mathcal{L}_{\text{holder}}$ and
$\mathcal{L}_{\text{polarity}}$) that will be discussed in Section III-D.
$\lambda$ is used to control the balance of two terms.

###  III-C Syntax GCN Encoder

Adversarial embedding adapter focuses on learning an informative and robust
distribution representation, while the explicit knowledge is ignored. Thus, to
learn a structural representation across multiple languages, we introduce a
syntax GCN encoder to integrate the dependency parsing tree into cross-lingual
structured sentiment analysis. The main reasons are summarized as follows.
First, the parsing tree structure plays an essential role in structured
sentiment analysis. As shown in Fig. 2, the holder “The Sadc ministerial
observer team” and target “President Mugabe” are both in a sub-tree.
Additionally, the distance between the expression and target (or holder) on
the tree is close to the sentence. All these show that the model can learn the
structural relationships among target, holder, and expression via neighbors in
the parsing trees. Second, the tree structures about two sentences that have
similar semantics are similar among multiple languages because they are
labelled based on the same linguists’ syntactic principles [14] while the word
order is different across languages. Thus, we propose a syntax GCN encoder to
model the explicit structural knowledge for cross-lingual transfer since GCN
[39] is natural to model the graph.

Inspired by [40], we construct a graph $G=(V,E)$ for each sentence $x$, where
$V=\\{v_{1},v_{2},...,v_{|x|}\\}$ is a set of nodes (words) and $E$ is a set
of relations in the dependency tree. Here, the number of nodes in $V$ is the
same as the number of words in the sentence. Based on the graphical dependency
parsing graph, we build an adjacency matrix $A\in\mathbb{R}^{|x|\times|x|}$,
where $A_{ij}$ is one if there is an edge between word $w_{i}$ and $w_{j}$ in
the dependency tree. The degree matrix of $G$ is $D$ where
$D_{ii}=\sum_{j}A_{ij}$. Self-loops for each node are considered by adding an
identity matrix to capture information about the current node itself.

The word embedding learned by the adversarial embedding adapter is input into
the syntax GCN encoder. Then, the the $l$-th graph convolution layer’s hidden
representation
${H}^{(l)}=\\{{h}_{1}^{(l)},{h}_{2}^{(l)},...,{h}_{|x|}^{(l)}\\}$ is learned
from its immediate neighbors’ representation at the $(l-1)$-th layer. Thus,
the hidden representation at the $l$-th layer is computed as:

${H}^{(l)}=\operatorname{ReLU}\left(D^{-\frac{1}{2}}{A}_{ij}D^{-\frac{1}{2}}{H}^{(l-1)}{W}^{(l)}\right)$
(7)

where $h^{(0)}=e$, and $W^{(l)}$, $b^{(l)}$ are the trainable parameter.
Finally, we obtain the structured representations
$H^{(L)}=\\{{h}_{1}^{(L)},{h}_{2}^{(L)},...,{h}_{|x|}^{(L)}\\}$ in an
universal common space from $L$-th layer, where ${h}_{i}^{(L)}$ the hidden
vector of word $w_{i}$ in sentence $x$.

###  III-D Decoder

Based on the informative and robust structural hidden representation $H^{L}$,
we adopt a simple decoder strategy for the four subtasks. First, we extract
the expressions by predicting their start and end indices. We regard the
expressions as the trigger words for each opinion. Then, we extract the
targets and holders and predict the sentiment polarity towards the given
expression.

##### Expression Extraction

To extract the expressions in the sentences, we predict whether the word is
the start or end indices of the expressions using a two binary classifiers.

$\displaystyle\boldsymbol{p}^{e^{s}}_{i}=\text{Sigmoid}(W^{e^{s}}h^{(L)}_{i}+b^{e^{s}})$
(8)
$\displaystyle\boldsymbol{p}^{e^{e}}_{i}=\text{Sigmoid}(W^{e^{e}}h^{(L)}_{i}+b^{e^{e}})$

where $\boldsymbol{p}^{e^{s}}_{i}$ and $\boldsymbol{p}^{e^{e}}_{i}$ are the
predicted expressions’ start and end probabilities of the word $w_{i}$,
$W^{e^{s}}$, $W^{e^{e}}$, $b^{e^{s}}$ and $b^{e^{e}}$ are the learnable
weights. Cross entropy (CE) is used as the loss function,

$\displaystyle\mathcal{L}_{\text{exp}}=\frac{1}{|x|}\sum_{i=1}^{|x|}{{\mathrm{CE}(y^{e^{s}}_{i},\boldsymbol{p}^{e^{s}}_{i})+\mathrm{CE}(y^{e^{e}}_{i},\boldsymbol{p}^{e^{e}}_{i})}}$

where $y^{e^{s}}_{i}$/$y^{e^{e}}_{i}$ is 1 if $i$ is the start/end index of
expression.

##### Target Extraction

To take the expressions information into account for target extraction, we
predict the start and end indices of the target based on the expression
representations.

$\displaystyle\boldsymbol{p}^{t^{s}_{k}}_{i}=\text{Sigmoid}(W^{t^{s}}[h^{(L)}_{i};h^{(L)}_{e^{s}_{k}};h^{(L)}_{e^{e}_{k}}]+b^{t^{s}})$
(9)
$\displaystyle\boldsymbol{p}^{t^{e}_{k}}_{i}=\text{Sigmoid}(W^{t^{e}}[h^{(L)}_{i};h^{(L)}_{e^{s}_{k}};h^{(L)}_{e^{e}_{k}}]+b^{t^{e}})$

where $\boldsymbol{p}^{t^{s}_{k}}_{i}$ and $\boldsymbol{p}^{t^{e}_{k}}_{i}$
are the predicted target $t_{k}$’s start and end probabilities of the word
$w_{i}$, $W^{t^{s}}$, $W^{t^{e}}$, $b^{t^{s}}$ and $b^{t^{e}}$ are the
learnable weights, $h^{(L)}_{e^{s}_{k}}$ and $h^{(L)}_{e^{e}_{k}}$ are the
vector representations of expression’s start and end words. $[a;b]$ is the
concatenation operation of $a$ and $b$. The loss function of target extraction
is,

$\displaystyle\mathcal{L}_{\text{target}}=\frac{1}{|O|\times|x|}\sum_{k=1}^{|O|}\sum_{i=1}^{|x|}{{\mathrm{CE}(y^{{t^{s}_{k}}}_{i},\boldsymbol{p}^{{t^{s}_{k}}}_{i})+\mathrm{CE}(y^{{t^{e}_{k}}}_{i},\boldsymbol{p}^{{t^{e}_{k}}}_{i})}}$
(10)

where $y^{t^{s}_{k}}_{i}$/$y^{t^{e}_{k}}_{i}$ is 1 if $i$ is the start/end
index of target towards the given expression $e_{k}$, $|O|$ is the number of
opinion tuples.

##### Holder Extraction

The same as target extraction, we predict probabilities of the holder’s start
and end indices, $\boldsymbol{p}^{h^{s}_{k}}_{i}$ and
$\boldsymbol{p}^{h^{e}_{k}}_{i}$, based on the expression-aware word
representations (like Equation 9). Then, we calculate the loss function for
holder extraction, $\mathcal{L}_{\text{holder}}$, just like Equation 10.

##### Sentiment Classification

Finally, we aim to predict the sentiment polarity $p_{k}$ with respect to the
given expression $e_{k}$. We use a max-pooling operation to obtain the
sentence representation $r^{s}=\text{Maxpooing}(H^{(L)})$ and concatenate it
with expression representations for polarity classification.

$\displaystyle\boldsymbol{p}^{p_{k}}=\text{Softmax}(W^{p}[r^{s};h^{(L)}_{e^{s}_{k}};h^{(L)}_{e^{e}_{k}}]+b^{p})$
(11)

Based on the sentiment probability distribution, we calculate the loss
function as follows,

$\displaystyle\mathcal{L}_{\text{polarity}}=\frac{1}{|O|}\sum_{k=1}^{|O|}{{\mathrm{CE}(y^{{p_{k}}},\boldsymbol{p}^{{p_{k}}})}}$

where $y^{p_{k}}=1$ if the sentiment polarity $p_{k}=P$ else $y^{p_{k}}=0$.

##  IV Experimental Setup

###  IV-A Datasets, Metrics and Implementation Details

##### Datasets

We evaluate our KEAM on five public benchmark datasets, namely
NoReC${}_{\text{Fine}}$, MultiB${}_{\text{EU}}$, MultiB${}_{\text{CA}}$, MPQA,
DS${}_{\text{Unis}}$, the same as [6]. NoReC${}_{\text{Fine}}$,
MultiB${}_{\text{EU}}$ and MultiB${}_{\text{CA}}$ are reviews in Norwegian,
Basque and Catalan, respectively. MPQA and DS${}_{\text{Unis}}$ are English
datasets. We remove the samples with neutral sentiment polarities for
transferring across languages since several datasets have no neutral examples.

TABLE I: The main results of our KEAM model and the strong baselines. The (supervised) methods trained on the target language are marked with *. The (unsupervised) methods without * are trained on source language and tested on the target language. Note that we report the results of the source language that obtained the best transfer performance for these unsupervised methods. The best results of supervised and unsupervised methods are marked with underline and bold. 4 Target | Methods | Token | Targeted F1
---|---|---|---
Holder F1 | Target F1 | Exp. F1 | F1
NoReC${}_{\text{Fine}}$ | Ovrelid et al. [41]* | 42.4 | 31.3 | 31.1 | -
IMN* | - | 35.9 | 48.7 | 18.0
RACL-BERT* | - | 47.2 | 56.3 | 30.3
RACL* | - | 45.6 | 55.4 | 20.1
Head-first* | 51.1 | 50.1 | 54.4 | 30.5
mBERT${}_{\text{base}}$ | 26.7 | 17.4 | 26.0 | 10.9
XLM-RoBERTa${}_{\text{base}}$ | 0.0 | 20.5 | 26.7 | 14.0
XLM-RoBERTa${}_{\text{large}}$ | 2.9 | 20.1 | 27.9 | 15.6
XLM-RoBERTa${}_{\text{large}}^{F}$ | 34.5 | 31.6 | 28.9 | 22.2
KEAM* | 55.7 | 41.8 | 30.0 | 29.1
KEAM | 41.9 | 39.8 | 30.7 | 25.2
MultiB${}_{\text{CA}}$ | Barnes et al. [42]* | 56.0 | 64.0 | 52.0 | -
IMN* | - | 56.3 | 60.9 | 32.5
RACL-BERT* | - | 67.5 | 70.3 | 52.4
RACL* | - | 65.4 | 67.6 | 49.1
Head-first* | 43.0 | 72.5 | 71.1 | 55.0
mBERT${}_{\text{base}}$ | 0.0 | 5.9 | 51.5 | 6.2
XLM-RoBERTa${}_{\text{base}}$ | 0.0 | 25.3 | 53.4 | 17.8
XLM-RoBERTa${}_{\text{large}}$ | 13.6 | 40.7 | 52.0 | 22.5
XLM-RoBERTa${}_{\text{large}}^{F}$ | 9.5 | 41.6 | 51.7 | 23.2
KEAM* | 64.0 | 67.8 | 58.5 | 38.7
KEAM | 32.7 | 48.1 | 54.9 | 26.7
MultiB${}_{\text{EU}}$ | Barnes et al. [42]* | 54.0 | 57.0 | 54.0 | -
IMN* | - | 48.2 | 65.2 | 39.5
RACL-BERT* | - | 59.9 | 72.6 | 56.8
RACL* | - | 55.4 | 70.7 | 48.2
Head-first* | 60.4 | 64.0 | 73.9 | 57.8
mBERT${}_{\text{base}}$ | 0.0 | 31.9 | 46.7 | 30.6
XLM-RoBERTa${}_{\text{base}}$ | 3.4 | 50.3 | 46.9 | 43.7
XLM-RoBERTa${}_{\text{large}}$ | 0.0 | 51.8 | 45.5 | 43.4
XLM-RoBERTa${}_{\text{large}}^{F}$ | 0.0 | 52.3 | 47.6 | 47.0
KEAM* | 64.6 | 65.3 | 62.2 | 52.7
KEAM | 3.8 | 58.6 | 50.4 | 48.0
MPQA | IMN* | - | 24.3 | 29.6 | 1.2
RACL-BERT* | - | 20.0 | 31.2 | 17.8
RACL* | - | 32.6 | 37.8 | 11.8
Head-first* | 43.8 | 51.0 | 48.1 | 33.5
mBERT${}_{\text{base}}$ | 0.4 | 2.3 | 9.7 | 1.1
XLM-RoBERTa${}_{\text{base}}$ | 4.6 | 3.4 | 9.6 | 2.1
XLM-RoBERTa${}_{\text{large}}$ | 4.5 | 4.8 | 10.6 | 1.9
XLM-RoBERTa${}_{\text{large}}^{F}$ | 10.5 | 5.0 | 40.7 | 2.2
KEAM* | 34.2 | 45.6 | 37.8 | 15.2
KEAM | 12.3 | 5.7 | 12.8 | 2.7
DS${}_{\text{Unis}}$ | IMN* | - | 33.0 | 27.4 | 17.9
RACL-BERT* | - | 44.6 | 38.2 | 27.3
RACL* | - | 39.3 | 40.2 | 22.8
Head-first* | 28.0 | 39.9 | 40.3 | 26.7
mBERT${}_{\text{base}}$ | 1.6 | 26.1 | 16.2 | 8.0
XLM-RoBERTa${}_{\text{base}}$ | 3.0 | 23.6 | 17.7 | 6.3
XLM-RoBERTa${}_{\text{large}}$ | 3.4 | 27.8 | 17.6 | 10.4
XLM-RoBERTa${}_{\text{large}}^{F}$ | 4.5 | 28.8 | 17.4 | 9.4
KEAM* | 31.2 | 44.7 | 36.4 | 21.5
KEAM | 8.1 | 30.4 | 18.5 | 12.4
4 |  |  |  |  |
TABLE II: The results of ablation studies. AEA: Adversarial Embedding Adapter. SGCNE: Syntax-GCN Encoder. The average score on the target language are reported. 4 Target | Methods | Token | Targeted F1
---|---|---|---
Holder F1 | Target F1 | Exp. F1 | F1
NoReC${}_{\text{Fine}}$ | KEAM | 16.6 | 32.3 | 28.0 | 16.8
\- AEA | 14.3 | 29.2 | 26.2 | 16.3
\- SGCNE | 13.4 | 30.3 | 27.1 | 14.1
MultiB${}_{\text{CA}}$ | KEAM | 12.9 | 43.0 | 49.4 | 14.3
\- AEA | 11.5 | 37.8 | 48.3 | 11.7
\- SGCNE | 10.7 | 42.1 | 47.1 | 13.9
MultiB${}_{\text{EU}}$ | KEAM | 5.2 | 46.8 | 45.8 | 17.0
\- AEA | 3.9 | 39.0 | 45.6 | 14.9
\- SGCNE | 4.4 | 42.5 | 41.9 | 15.5
MPQA | KEAM | 10.4 | 7.6 | 12.1 | 2.3
\- AEA | 8.7 | 7.2 | 12.0 | 1.7
\- SGCNE | 9.1 | 7.3 | 11.5 | 2.0
DS${}_{\text{Unis}}$ | KEAM | 2.0 | 26.97 | 18.5 | 11.8
\- AEA | 0.8 | 20.0 | 17.1 | 9.8
\- SGCNE | 1.6 | 23.6 | 18.1 | 10.5
4 |  |  |  |  |

##### Metrics

Following the previous works [6], we adopt four metrics from token and target
levels. The token-level metrics are holder F1 (Holder F1), target F1 (Target),
and expression F1 (Exp. F1). For target-level metrics, we utilize targeted F1,
which is commonly used in targeted sentiment analysis [21]. Notably, a correct
prediction requires that both the target boundary and its sentiment polarity
be correct.

##### Implementation Details

Adam optimizer is utilized with the learning rates of 1e-5. The max sequence
length is set as 128. The dropout is 0.1. We implement our KEAP model based on
the transformers framework 333https://huggingface.co/transformers/. We use
Stanza 444https://github.com/stanfordnlp/stanza tool to obtain the universal
dependencies tree. The default number of GCN layers is 3 in our experiments.
We train our proposed model for 50 epochs, run it five times with different
random seeds and report the mean. The reported test results are based on the
model that performs best regarding targeted F1 on the development set.

###  IV-B Baselines

To evaluate the effectiveness of our KEAM model, we compare our approach with
the state-of-the-art baselines from two views, supervised and unsupervised
(transfer) methods.

The following typical supervised baselines are selected, 1) IMN [8] performs
token-level classification with document-level classification. 2) RACL [22]
utilizes a is a relation propagation mechanism via a multi-task learning
strategy for the subtasks of aspect-based sentiment analysis. 3) RACL-BERT
[22] is an extension of RACL that replaces embedding with BERT-Large. 4) Head-
first [6] translates the structured sentiment analysis into a dependency
parsing problem and obtains the state-of-the-art results. We also compare with
previously reported extraction results from Barnes et al. [42] and Ovrelid. et
al. [41].

For the unsupervised (transfer) methods, four typical multi-lingual pre-
trained language models [9, 10] are utilized as the baselines. These models
have different semantic information since they are trained on different large-
scale datasets using different objectives and settings. First,
mBERT${}_{\text{base}}$ is trained on the top 104 languages with the largest
Wikipedia using a masked language modeling (MLM) objective [9]. XLM-
RoBERTa${}_{\text{base}}$ and XLM-RoBERTa${}_{\text{large}}$ models are pre-
trained on the 100 languages with 2.5TB of filtered CommonCrawl data.
Moreover, Ushio et al. [35] finetuned XLM-RoBERTa${}_{\text{large}}$ model on
named-entity-recognition (NER) task with multilingual datasets, namely XLM-
RoBERTa${}_{\text{large}}^{F}$.

##  V Experimental Results

###  V-A Main Results

To verify the effectiveness of our KEAM model, we evaluate it on a cross-
lingual transfer setting that trains on source language and tests on the
target language without labelled data. We compare our model with supervised
and unsupervised baselines (See Table I). Note that we report the results on
the source dataset that obtains the best performance on the target language
for the unsupervised transfer methods (without *).

From this table, we obtain the following observations. First, our transfer
model (KEAM) performs better than the unsupervised baselines significantly for
cross-lingual structured sentiment analysis. To be specific, our model obtains
the best performance overall the datasets in terms of all the metrics.
Moreover, KEAM improves more than three points on three datasets in terms of
targeted F1. All these indicate that our informative structural representation
can help model transfer structural sentiment across languages. Moreover, the
performance of various multi-lingual embeddings is very different due to
various multi-lingual embeddings having different semantic information.
Second, the performance of KEAM is comparable with the supervised methods
(e.g., KEAM*, IMN) in some cases. For example, the proposed model performs
better than IMN over NOReC${}_{\text{Fine}}$ and MultiB${}_{\text{CA}}$
regarding targeted F1. Third, our model also performs well on the supervised
tasks (KEAM*). We train our model on the target language with a supervised
method. We observe that KEAM* performs better or is comparable with the
supervised baselines in most cases. Fourth, regrading holder F1, KEAM (KEAM*)
outperforms better than the unsupervised (supervised) methods in most cases.
The frequency of holders is relatively low, which makes it hard to extract the
holders in the text. We incorporate syntax information via a GCN model to a
structural representation, which helps KEAM identify the holders.

###  V-B Ablation Studies

Furthermore, to investigate the effectiveness of each component consisting of
our KEAM, we do the ablation test over five datasets (See Table II). The
average score over all the source datasets except the target dataset is
reported. Specifically, we remove the adversarial embedding adapter (-AEA) and
syntax-GCN encoder (-SGCNE) from the KEAM respectively. For -AEA, we simply
use the concatenation of multiple multi-lingual word embeddings without
adversarial training.

The results demonstrate that both adversarial embedding adapter and syntax-GCN
encoder are important for this task. First, our adversarial embedding adapter
can capture the diversity of essential features from various multi-lingual
embeddings that contain different semantic information. It learns an
informative embedding via an attention mechanism and improves the robustness
via an adversarial training strategy. The final word embeddings can improve
the performance of cross-lingual transfer well. Second, the structural
representations learned by the syntax GCN encoder can further improve the
model effect. It is because the dependency tree is vital for structured
sentiment analysis, and the tree structures across languages are similar.

###  V-C The Effectiveness of Transfer Learning

In this section, we basically try to exploit what has been learned in one
language to improve generalization in another. Table III shows the results of
transfer learning across languages in terms of targeted F1. The findings are
summarized as follows. 1) The source language will influence the performance
largely. For example, transferring knowledge from MultiB${}_{\text{CA}}$
obtains outstanding performance for MultiB${}_{\text{EU}}$ while having a
limited effect for MPQA. 2) Transferring from the target language may not
obtain the best scores. For instance, though both MPQA and
DS${}_{\text{Unis}}$ are English datasets, transferring from
NoReC${}_{\text{Fine}}$ obtains the best performance for them. 3) It is
obvious that training on the target dataset (supervised approaches) obtains
the best results since the train and test datasets are in the same
distribution.

TABLE III: The performance of transfer learning in terms of targeted F1. 4 | NoReC${}_{\text{Fine}}$ | MultiB${}_{\text{CA}}$ | MultiB${}_{\text{EU}}$ | MPQA | DS${}_{\text{Unis}}$
---|---|---|---|---|---
NoReC${}_{\text{Fine}}$ | 29.1 | 6.5 | 7.7 | 3.7 | 16.8
MultiB${}_{\text{CA}}$ | 25.2 | 38.7 | 48.0 | 1.7 | 12.4
MultiB${}_{\text{EU}}$ | 24.5 | 26.7 | 52.7 | 1.1 | 10.9
MPQA | 6.5 | 11.6 | 5.0 | 15.2 | 8.9
DS${}_{\text{Unis}}$ | 11.2 | 5.9 | 7.2 | 2.7 | 21.5
4 |  |  |  |  |

##  VI Conclusions and Future Work

In this paper, we propose KEAM for cross-lingual structured sentiment
analysis. Mainly, we first design an adversarial embedding adaptor for
learning an informative and robust embedding. Then, a syntax GCN encoder is
introduced to learn structural representation based on the parsing tree. We
compare our model with both supervised and unsupervised baselines on five
datasets with four languages. The experimental results show the great
advantage of our model for cross-lingual transfer. We also do the ablation
studies to investigate the effectiveness of each part in our model. To
investigate the effectiveness of cross-lingual transfer, we run 25 transfer
tasks across five datasets.

In future work, we would like to explore more multi-lingual pre-trained
models. Also, exploring the performance of our model for other structure
extraction tasks is interesting.

## Acknowledgment

The authors wish to thank the reviewers for their helpful comments and
suggestions. This research is funded by the Science and Technology Commission
of Shanghai Municipality (No. 19511120200 and 21511100402) and by Shanghai Key
Laboratory of Multidimensional Information Processing, East China Normal
University, No. 2020KEY001. This research is also funded by the National Key
Research and Development Program of China (No. 2021ZD0114002), the National
Nature Science Foundation of China (No. 61906045), and Shanghai Science and
Technology Innovation Action Plan International Cooperation project “Research
on international multi language online learning platform and key technologies
(No.20510780100)”. The computation is performed in ECNU Multi-functional
Platform for Innovation (001).

## References

  * [1] R. K. Bakshi, N. Kaur, R. Kaur, and G. Kaur, “Opinion mining and sentiment analysis,” in _2016 3rd international conference on computing for sustainable global development (INDIACom)_. IEEE, 2016, pp. 452–455.
  * [2] J. Zhou, J. Tian, R. Wang, Y. Wu, W. Xiao, and L. He, “Sentix: A sentiment-aware pre-trained model for cross-domain sentiment analysis,” in _Proceedings of the 28th International Conference on Computational Linguistics_ , 2020, pp. 568–579.
  * [3] M. Pontiki, D. Galanis, H. Papageorgiou, I. Androutsopoulos, S. Manandhar, M. Al-Smadi, M. Al-Ayyoub, Y. Zhao, B. Qin, O. De Clercq _et al._ , “Semeval-2016 task 5: Aspect based sentiment analysis,” in _International workshop on semantic evaluation_ , 2016, pp. 19–30.
  * [4] J. Zhou, J. X. Huang, Q. V. Hu, and L. He, “Sk-gcn: modeling syntax and knowledge via graph convolutional network for aspect-level sentiment classification,” _Knowledge-Based Systems_ , vol. 205, p. 106292, 2020.
  * [5] M. Mitchell, J. Aguilar, T. Wilson, and B. Van Durme, “Open domain targeted sentiment,” in _Proceedings of EMNLP_ , 2013, pp. 1643–1654.
  * [6] J. Barnes, R. Kurtz, S. Oepen, L. Øvrelid, and E. Velldal, “Structured sentiment analysis as dependency graph parsing,” in _Proceedings of ACL_ , C. Zong, F. Xia, W. Li, and R. Navigli, Eds., 2021, pp. 3387–3402.
  * [7] W. Wang, S. J. Pan, D. Dahlmeier, and X. Xiao, “Recursive neural conditional random fields for aspect-based sentiment analysis,” in _Proceedings of EMNLP_ , 2016, pp. 616–626.
  * [8] R. He, W. S. Lee, H. T. Ng, and D. Dahlmeier, “An interactive multi-task learning network for end-to-end aspect-based sentiment analysis,” in _Proceedings of ACL_ , 2019, pp. 504–515.
  * [9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of NAACL_ , 2019, pp. 4171–4186.
  * [10] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in _Proceedings of ACL_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds., 2020, pp. 8440–8451.
  * [11] W. Ahmad, H. Li, K.-W. Chang, and Y. Mehdad, “Syntax-augmented multilingual bert for cross-lingual transfer,” in _ACL_ , 2021.
  * [12] D. Xu, J. Li, M. Zhu, M. Zhang, and G. Zhou, “XLPT-AMR: Cross-lingual pre-training via multi-task learning for zero-shot AMR parsing and text generation,” in _Proceedings of ACL_ , 2021, pp. 896–907.
  * [13] P. Mulcaire, J. Kasai, and N. A. Smith, “Polyglot contextual representations improve crosslingual transfer,” in _Proceedings of NAACL_ , 2019, pp. 3912–3918.
  * [14] A. Subburathinam, D. Lu, H. Ji, J. May, S.-F. Chang, A. Sil, and C. Voss, “Cross-lingual structure transfer for relation and event extraction,” in _Proceedings of EMNLP-IJCNLP_ , 2019, pp. 313–325.
  * [15] Y. Lin, Z. Liu, and M. Sun, “Neural relation extraction with multi-lingual attention,” in _Proceedings of ACL_ , 2017, pp. 34–43.
  * [16] M.-C. de Marneffe, C. D. Manning, J. Nivre, and D. Zeman, “Universal dependencies,” _Computational linguistics_ , vol. 47, no. 2, pp. 255–308, 2021.
  * [17] Y. Zhang, P. Qi, and C. D. Manning, “Graph convolution over pruned dependency trees improves relation extraction,” in _Proceedings of EMNLP_ , 2018, pp. 2205–2215.
  * [18] J. Zhou, Q. Chen, J. X. Huang, Q. V. Hu, and L. He, “Position-aware hierarchical transfer model for aspect-level sentiment classification,” _Information Sciences_ , vol. 513, pp. 1–16, 2020.
  * [19] J. Zhou, J. X. Huang, Q. Chen, Q. V. Hu, T. Wang, and L. He, “Deep learning for aspect-level sentiment classification: survey, vision, and challenges,” _IEEE access_ , vol. 7, pp. 78 454–78 483, 2019.
  * [20] J. Wiebe, T. Wilson, and C. Cardie, “Annotating expressions of opinions and emotions in language,” _Language resources and evaluation_ , vol. 39, no. 2, pp. 165–210, 2005.
  * [21] R. He, W. S. Lee, H. T. Ng, and D. Dahlmeier, “An interactive multi-task learning network for end-to-end aspect-based sentiment analysis,” in _Proceedings of ACL_ , 2019, pp. 504–515.
  * [22] Z. Chen and T. Qian, “Relation-aware collaborative learning for unified aspect-based sentiment analysis,” in _Proceedings of ACL_ , 2020, pp. 3685–3694.
  * [23] M. Zhang, Q. Wang, and G. Fu, “End-to-end neural opinion extraction with a transition-based model,” _Information Systems_ , vol. 80, pp. 56–63, 2019\.
  * [24] D. Lu, A. Subburathinam, H. Ji, J. May, S.-F. Chang, A. Sil, and C. Voss, “Cross-lingual structure transfer for zero-resource event extraction,” in _Proceedings of LREC_ , 2020, pp. 1976–1981.
  * [25] L. Qian, H. Hui, Y. Hu, G. Zhou, and Q. Zhu, “Bilingual active learning for relation classification via pseudo parallel corpora,” in _Proceedings of ACL_ , 2014, pp. 582–592.
  * [26] X. Feng, L. Huang, D. Tang, H. Ji, B. Qin, and T. Liu, “A language-independent neural network for event detection,” in _Proceedings of ACL_ , Aug. 2016, pp. 66–71.
  * [27] B. Zou, Z. Xu, Y. Hong, and G. Zhou, “Adversarial feature adaptation for cross-lingual relation classification,” in _Proceedings of COLING_ , 2018, pp. 437–448.
  * [28] X. Feng, X. Feng, B. Qin, Z. Feng, and T. Liu, “Improving low resource named entity recognition using cross-lingual knowledge transfer.” in _IJCAI_ , vol. 1, 2018, pp. 4071–4077.
  * [29] J. Xie, Z. Yang, G. Neubig, N. A. Smith, and J. Carbonell, “Neural cross-lingual named entity recognition with minimal resources,” _arXiv preprint arXiv:1808.09861_ , 2018.
  * [30] X. Wang, X. Han, Y. Lin, Z. Liu, and M. Sun, “Adversarial multi-lingual neural relation extraction,” in _Proceedings of COLING_ , 2018, pp. 1156–1166.
  * [31] P. Mulcaire, S. Swayamdipta, and N. A. Smith, “Polyglot semantic role labeling,” in _Proceedings of ACL_ , 2018, pp. 667–672.
  * [32] N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith, “Linguistic knowledge and transferability of contextual representations,” in _Proceedings of NAACL_ , 2019, pp. 1073–1094.
  * [33] O. Prazák and M. Konopík, “Cross-lingual srl based upon universal dependencies.” in _RANLP_ , 2017, pp. 592–600.
  * [34] T. H. Nguyen and R. Grishman, “Graph convolutional networks with argument-aware pooling for event detection,” in _Proceedings of AAAI_ , 2018\.
  * [35] A. Ushio and J. Camacho-Collados, “T-NER: An all-round python library for transformer-based named entity recognition,” in _Proceedings of EACL_ , 2021, pp. 53–62.
  * [36] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _Proceedings of ICLR_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [37] M. Sato, J. Suzuki, H. Shindo, and Y. Matsumoto, “Interpretable adversarial perturbation in input embedding space for text,” in _Proceedings of IJCAI_ , J. Lang, Ed., 2018, pp. 4323–4330.
  * [38] T. Miyato, A. M. Dai, and I. J. Goodfellow, “Adversarial training methods for semi-supervised text classification,” in _Proceedings of ICLR_. OpenReview.net, 2017.
  * [39] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _In Proceedings of ICLR_ , 2017.
  * [40] L. Yao, C. Mao, and Y. Luo, “Graph convolutional networks for text classification,” in _Proceedings of AAAI_ , vol. 33, no. 01, 2019, pp. 7370–7377.
  * [41] L. Øvrelid, P. Mæhlum, J. Barnes, and E. Velldal, “A fine-grained sentiment dataset for Norwegian,” in _Proceedings of the 12th Language Resources and Evaluation Conference_. Marseille, France: European Language Resources Association, May 2020, pp. 5025–5033. [Online]. Available: https://aclanthology.org/2020.lrec-1.618
  * [42] J. Barnes, P. Lambert, and T. Badia, “Multibooked: A corpus of basque and catalan hotel reviews annotated for aspect-level sentiment classification,” _arXiv preprint arXiv:1803.08614_ , 2018.


# Measures of Complexity for Large Scale Image Datasets

Ameet Annasaheb Rahane § University of California, Berkeley
Berkeley, CA
<EMAIL_ADDRESS>Anbumani Subramanian  Intel
Bangalore, India
<EMAIL_ADDRESS>
###### Abstract

Large scale image datasets are a growing trend in the field of machine
learning. However, it is hard to quantitatively understand or specify how
various datasets compare to each other - i.e., if one dataset is more complex
or harder to “learn” with respect to a deep-learning based network. In this
work, we build a series of relatively computationally simple methods to
measure the complexity of a dataset. Furthermore, we present an approach to
demonstrate visualizations of high dimensional data, in order to assist with
visual comparison of datasets. We present our analysis using four datasets
from the autonomous driving research community - Cityscapes, IDD, BDD and
Vistas. Using entropy based metrics, we present a rank-order complexity of
these datasets, which we compare with an established rank-order with respect
to deep learning.

###### Index Terms:

Machine learning, deep learning, manifold learning, intrinsic dimensionality,
learning theory

§§footnotetext: This work was done as an intern at Intel.

##  I Introduction

With an ever-increasing use and need for large image datasets for machine
learning, it is important to see how datasets differ in terms of complexity
and distribution. When comparing large scale image datasets, the common
practise is to evaluate metrics like total number of images, number of images
in each class, or class distributions in the dataset. However, all these are
metrics defined by humans and do not really provide any insights about the
underlying distribution of data.

Our objective in this work is to enable the comparison of the _unconditioned_
distribution of data between any two datasets. In other words – our interest
in the analysis of a comparative complexity of data between different image
datasets and to analyze whether this aspect of complexity impacts the ability
of a neural network to learn from that dataset. Towards this, we utilize
concepts from statistics, topology, and information theory – to capture the
essence of data dimensionality without any knowledge of underlying data labels
or classes in the dataset. We focus on three main factors and relate these
metrics to a network learning scheme. Following are the main contributions of
our work:

  1. (i)

A study on the distribution of entropy for a given dataset

  2. (ii)

Estimation of the intrinsic dimensionality of a given dataset and thus learn
about the dimensions of underlying data manifold

  3. (iii)

Use of Uniform Manifold Approximation and Projection (UMAP) [6] to transform a
given dataset, and thus visually compare the transformations

In this work, our focus is on the datasets widely used in autonomous driving
community, where semantic segmentation of images is a problem of high
interest. Towards this problem, a number of new image datasets have been
released in the recent years. These datasets are targeted for research in
object detection and semantic segmentation of images from a vehicle on roads.
A good performance on semantic segmentation on image datasets informs the
feasibility of building perception algorithms, a critical blocks for self-
driving, autonomous vehicles in the future. Therefore, analyzing the kind of
complexity in datasets offers a glimpse of how “hard” it is to learn from a
dataset compared to others and hence provides insights into building
generalizable agents for the problem of perception in self-driving. Our
objective is to build, test and demonstrate a comparison of complexity and
distribution of data in large volume, high dimensional image datasets.

##  II Datasets

In this work, we use four well known datasets in the field of autonomous
driving: Cityscapes (CS) [1], India Driving Dataset (IDD) [4], Berkeley Deep
Drive (BDD) [2], and Mapillary Vistas (Vistas) [3]. We chose these datasets
specifically to study and compare if the nature of datasets have any influence
on data complexity and distribution. The datasets like CS, BDD, or Vistas
contain images from structured driving conditions while IDD was designed with
images from unstructured driving conditions and hence our interest here.

##  III Methods

###  III-A Shannon Entropy

In our work, we measure entropy in three different ways. The standard Shannon
entropy of a grayscale image is defined as:

$H=-\sum_{i=0}^{n-1}p_{i}\log p_{i}$ (1)

where $n$ is the number of gray levels and $p_{i}$ is the probability of a
pixel having a given gray level $i$.

###  III-B GLCM

The gray level co-occurence matrix (GLCM) of an image is another commonly used
entropy measure. GLCM is a histogram of co-occuring grayscale values at a
given offset over an image. GLCM helps to characterize the texture in an
image, beyond the image properties like energy and homogeneity. Here, the GLCM
entropy is calculated as:

$H_{g}=-\sum_{i=0}^{n-1}\sum_{j=1}^{n-1}p(i,j)\log p(i,j)$ (2)

where $n$ is the number of gray levels and $p(i,j)$ is the probability of two
pixels having intensities $i$ and $j$, separated by the specified offset.

###  III-C Delentropy

Delentropy is a measure based on a computable probability density function
called deldensity [7]. This density distribution is capable of capturing the
underlying structure using spatial image and pixel co-occurrence. This is
possible because the scalar image pixel values are non-locally related to the
entire gradient vector field, which means that the usage of gradient vectors
in the calculation of this entropy allows for global image features to be
considered in the calculation of delentropy. This is in contrast to the pixel
level Shannon entropy, which entirely relies on individual pixel values.
Delentropy is defined similar to Equation 1 but uses with the new (deldensity)
distribution which can capture more non-local information. [7]

###  III-D Intrinsic Dimensionality Estimation

Intrinsic Dimensionality (ID) represents the minimal representation of the
underlying manifold possible for a dataset without losing any information. In
other words, it is the smallest dimension required for a deep learning model
to exactly represent a dataset. If $m$ is the intrinsic dimensionality of a
dataset, it implies that a $m$ dimensional manifold is required to represent
the data in the dataset and that any representation less than $m$ cannot cover
the data without any loss of information.

####  III-D1 Estimator

The maximum likelihood estimator of intrinsic dimensionality is defined in
[5].

Consider a dataset with $n$ images and let $X_{i}$ represent the individual
sample (image) in the dataset. Let $X_{1},\cdots,X_{n}\in\mathbb{R}^{p}$ be
i.i.d. observations with an embedding of a lower dimensional sample. Let each
sample $X_{i}$ be mapped to a point $x_{i}$ in the manifold. Specifically,
$X_{i}=g(Y_{i})$, where $Y_{i}$ are sampled from an unknown smooth density $f$
on $\mathbb{R}^{m}$, with some $m\leq p$, and $g(.)$ is a continuous and
sufficiently smooth mapping function. $m$ is the intrinsic dimensionality of
the dataset.

In other words, we pick a point $x$, assuming $f(x)$ is approximately constant
in a sphere of radius $R$ around $x$. Then, we can treat the observations as a
homogeneous Poisson process in $S_{x}(R)$. Using the derivation from [5] and
following the notations, the maximum likelihood estimation for $m$, given $k$
nearest neighbors, is

$\hat{m}_{k}(x)=\Bigg{[}\dfrac{1}{k-1}\sum_{j=1}^{k-1}\log\dfrac{T_{k}(x)}{T_{j}(x)}\Bigg{]}^{-1}$
(3)

where $T(x)$ is the average distance from each point to its $k$-th nearest
neighbor. This is an estimator of the intrinsic dimensionality which is shown
to balance bias and variance better than all other existing solutions [5].

###  III-E UMAP: Uniform Manifold Approximation and Projection

We use UMAP [6], a topology preserving dimensionality reduction technique to
create visual representation of the embedding of a dataset, and then
qualitatively see how this embedding relates to other metrics (discussed here)
and also relate to available performance comparison of deep learning methods.

####  III-E1 Assumptions

The use of uniform manifold approximation and projection requires a few
assumptions. UMAP assumes that data must be uniformly distributed on a
Riemannian manifold, the manifold is locally connected and the Riemannian
metric is locally constant or can be approximated as such.

####  III-E2 Embedding

UMAP provides a manifold learning technique for dimension reduction, with
foundations in Riemannian geometry and algebraic topology. The first phase
consists of constructing a fuzzy topological representation by using
simplices. Then, we optimize our embedding (by using stochastic gradient
descent) to have as close a fuzzy topological representation as possible
(measured by cross entropy). This enables UMAP to construct a topological
representation of the higher dimensional data in a lower dimensional space.
[6]

####  III-E3 Method

In our work, we define a vector space of image as follows. We define a mapping
from each image to a vector in a high dimensional space and define some
distance function between any two points in that space. The most simple
version of this is the space of images with $n$ pixels each, pointing to a
vector in $\mathbb{R}^{n}$ (flattening the image into a vector). However, we
can create a more intuitive representation of the image space as a vector
space, using the gray level co-occurence matrix (GLCM). Using this feature
space, we can find the dimensionality reduction dependent on the topology of
texture based features. Using this space, we can assume that the conditions
listed above are met.

Using additional parameters like number of nearest neighbors in UMAP, we show
how local and global structures change/shift differently in different datasets
- i.e., when the nearest number of neighbors to a point are low, it implies
that more local structure is used to create the embedding, and when the number
of nearest neighbors to a point are high, higher order structure in the
dataset is used to create the embedding. [6]

###  III-F Computation

We used the implementation of ID from [8] and performed all our experiments on
an Intel Xeon™ processor. The entire computation for intrinsic dimensionality
was completed well within 60 seconds for Cityscapes, IDD and BDD datasets.
This computation increased to over an hour for Vistas, which contains lot more
images than other datasets.

##  IV Results

###  IV-A Entropy

For Cityscapes, IDD, BDD, and Vistas, we used standard implementations of the
methods described in sections III-A, III-B, III-C to calculate Shannon
Entropy, GLCM Entropy, and delentropy for every image in each dataset. In
Table III, we present the average entropy of each dataset (for each type of
entropy). figure Figure1, 2, and 3 show the entropy distribution in each
dataset (fitted to a normal distribution).

Figure 1: Shannon entropy distributions fit to normal distributions of all
four datasets. Notably, as we move left to right from Cityscapes to Vistas,
the distribution appears to become much tighter in nature. This can also be
seen as the standard deviation becomes smaller from Cityscapes to BDD. This
implies that BDD and Vistas have fewer lower entropy images than CS and IDD.
As such, when a network learns these datasets, the first few layers of the
network will have a harder time learning CS and Vistas.

Figure 2: GLCM Entropy distributions fit to normal distributions of all four
datasets - Cityscapes, IDD, BDD and Vistas. Contrary to Figure1, the texture
level entropy of each dataset does not exhibit a shift in notable standard
deviation.

Figure 3: Delentropy distributions fit to normal distributions of all four
datasets. Delentropy seems to be normally distributed. This could be a feature
of the deldensity calculation, but this points out that global features do not
exhibit a severe shift in entropy between the given datasets.

###  IV-B Intrinsic Dimensionality Estimation

Using the implementation from [8], we calculate the maximum likelihood
estimation of intrinsic dimensionality for each dataset and these results are
shown in Table I. From this table, we can infer the geometric complexity of
our datasets, and also get some interesting insights into the representations
of the given datasets.

TABLE I: Intrinsic dimensionalities of the selected datasets.

Dataset | Intrinsic Dimensionality (ID)
---|---
Cityscapes | $11.4$
IDD | $11.9$
BDD | $13.7$
Vistas | $30.1^{*}$

$*$: Vistas had a high estimation variance possibly due to the large dataset
size, and computational restrictions.

###  IV-C UMAP: Uniform Manifold Approximation and Projection

We use UMAP to help visualize the entire datasets in our work. We find an
approximation to the topology of GLCM features of each of our datasets, and
then use it to embed the four datasets in $\mathbb{R}^{2}$.

In Figure4, we show a plot of each two dimensional embedding. Alongside each
axis is the distribution of each dimension of the embedding. Overall, it can
be seen as a plot of the distribution of a two-dimensional embedding of the
dataset, with the marginals on each axis. We have shown plots of four
variations with the number of neighbors $k=2,20,100,500$, to show the change
in datasets, considering local to global focused embeddings.

##  V Observations

###  V-A Entropy

Of the four datasets used here, Vistas is considered most difficult to learn
for semantic segmentation algorithms. From the entropy plots, we observe
possible supporting evidence in a quantitative manner - Vistas has the highest
mean entropy across all entropies, and more so for GLCM based entropy implying
that Vistas has relatively more complex texture features than the other three
datasets.

Based on the entropy distributions, we formulate a rank order of complexity
(lowest to highest) for the four datasets. This rank order is supported by the
plots in Figure1, 2, and 3 adn the tables in Table II and III.

TABLE II: Mean (and standard deviation) of entropies of the datasets. Dataset | Shannon pixel entropy | GLCM (texture) entropy | Delentropy
---|---|---|---
Cityscapes | $6.85(0.29)$ | $6.32(1.25)$ | $4.35(0.35)$
IDD | $7.17(0.38)$ | $7.8(1.27)$ | $4.38(0.55)$
BDD | $7.38(0.4)$ | $7.79(1.05)$ | $4.3(0.68)$
Vistas | $7.39(0.31)$ | $8.12(1.13)$ | $5.01,(0.62)$
TABLE III: Rank order of datasets with respect to average entropies. Shannon pixel entropy | GLCM (texture) Shannon entropy | Delentropy
---|---|---
Cityscapes | Cityscapes | BDD
IDD | BDD | Cityscapes
BDD | IDD | IDD
Vistas | Vistas | Vistas

BDD has a higher pixel entropy, although its texture and more global structure
are lower than those of IDD. For a deep learning network based semantic
segmentation method, a higher pixel entropy implies that the first few layers
of the network which interpret pixel distributions, will have difficulty in
learning and could manifest as issues like convergence of bias weights, time
taken to converge. But for the same network, in case of IDD, the latter layers
would have a difficulty in learning. Approximately, high Shannon pixel entropy
would imply that a given network would have a harder time learning the first
few layers, as these layers are used to interpret pixel level behavior.
Alternatively, high delentropy and GLCM texture entropy would mean that the
network would have a harder time learning in the middle or later layers, as
these layers are used to learn higher order features.

From our analysis, using an entropy-centric view, we observe that the
Cityscapes dataset tends to be the lowest in complexity, aside from
delentropy. Delentropy represents a higher order structure of the images. We
see that Cityscapes, BDD, and IDD are of similar complexity with respect to
delentropy. This observation can be correlated by the fact that these three
datasets were created out of a similar data capture setup - dashcam-style
images with view of roads – with differences in geographies and times of
capture. Whereas, the Vistas dataset is created with images from both dashcam-
style images and mobile-phone images. This capture setup leads to a totally
diverse collection of images, with any scene that looks like a road scene.
Therefore, the images in Vistas dataset contain lot more texture patterns -
i.e., higher order features, This leads to an increased difference between the
entropy values, except for Shannon entropy.

###  V-B Intrinsic Dimensionality

The significance of Cityscapes and IDD with similar complexity (in Table III)
can be understood as follows: these datasets have images with similar
perspective view of the road scene - possibly due to the similarity in data
capture setup and camera placements on the vehicle. Moreover, Cityscapes and
IDD lack variations in weather conditions, unlike BDD or Vistas. Considering
these factors, it is intuitive to understand the similar pixel based entropy
of these two datasets. The difference in entropy is completely dependent on
the nature of data in a dataset - in this case, as different landscapes, road
patterns, and driving patterns. The object (class) labels of IDD and BDD are
compatible with Cityscapes, and so have no impact in the analysis.

###  V-C UMAP

UMAP embedding helps to visualize the projection of each of our datasets. From
our visualization plots in Fig. 4, we observe that “difficult” datasets appear
to be more densely distributed. The three datasets - Cityscapes, IDD, and BDD
share a lot of similarities in distribution, with minor variations which is
easier to observe at more nubmer of nearet neighbors, $k$ (= $100,500$) (see
Figure4).

Our observations are consistent with the premise that distribution of data in
these datasets have an approximately complexity order, as given in Table I:
Cityscapes, IDD, BDD, Vistas.

###  V-D Accuracy of deep learning networks

From the performance metric in [4], we can observe that deep learning semantic
segmentation algorithms already have an order of complexity as follows:
Cityscapes, IDD, BDD, Vistas. This order matches the rank-order described in
our analysis here, and help conclude that pixelwise Shannon entropy,
delentropy, intrinsic dimensionality, and deep learning results are related.
Given the similarity of entropy distributions, a correlation between
delentropy and GLCM feature based entropy apppears to exist. Using the UMAP
specific density plots for network comparison, it is difficult to infer the
complexity of a dataset given the higher density of points in the embedding,
although it does seem as though a relation exists. These require a more deeper
study in future.

##  VI Conclusion

In this work, we demonstrated that well known information theoretic measures
like pixelwise Shannon entropy, GLCM feature Shannon entropy, delentropy, and
intrinsic dimensionality all correlate with known results of performance
metrics in semantic segmentation networks trained on the same datasets.
Therefore, we present these metrics to further study relationships between
information theoretic claims we can make about large scale image datasets and
deep learning performance. This could lead lead to providing guarantees about
the performance of deep learning on a given dataset or a data generation
technique. Additionally, we present a dimensionality reduction embedding in
the context of large image datasets to show the density of data clusters as a
measure of complexity.

##  VII Acknowledgements

Ameet Rahane was supported by the Intel AI Student Ambassadorship program
towards this publication.

## References

  * [1] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [2] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, T. Darrell, “BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling,” arXiv preprint $1805.04687$ , 2018
  * [3] G. Neuhold, T. Ollmann, S. Rota Bulò, and P. Kontschieder. “The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes.” In International Conference on Computer Vision (ICCV), 2017.
  * [4] G. Varma, A. Subramanian, A. Namboodiri, M. Chandraker, C. Jawahar, “IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments” In IEEE Winter Conf. on Applications of Computer Vision (WACV), 2019
  * [5] E. Levina, P. Bickel, “Maximum Likelihood Estimation of Intrinsic Dimension”, In Advances in Neural Information Processing Systems $17$, 2005.
  * [6] L. McInnes, J. Healy, J. Melville, “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction”, unpublished, 2018.
  * [7] K. Larkin, “Reflections on Shannon Information: In search of a natural information-entropy for images”, arXiv preprint $1609.01117$, 2016
  * [8] M. Cherti, “Implementation of ’Maximum Likelihood Estimation of Intrinsic Dimension”’, Github Repository, Retreived January 9, 2020, from https://gist.github.com/mehdidc/8a0bb21a31c43b0cbbdd31d75929b5e4/, May 2016.

(a) Cityscapes

(b) IDD

(c) BDD

(d) Vistas

Neighbors: $2$, Min. Distance: $.1$

Neighbors: $20$, Min. Distance: $.1$

Neighbors: $100$, Min. Distance: $.1$

Neighbors: $500$, Min. Distance: $.1$

Figure 4: Two dimensional projection of each dataset using UMAP (Sec. V-C).
The plots show embedding (with each data point being a coordinate), with the
distribution of each dimension on each axis. Many patterns can be be observed,
when the embeddeings change from local to global.