∎

11institutetext: A. Gharahighehi 22institutetext: C. Vens 33institutetext:
Itec, imec research group at KU Leuven, Kortrijk, Belgium 44institutetext: KU
Leuven, Campus KULAK, Department of Public Health and Primary Care, Kortrijk,
Belgium
44email: {alireza.gharahighehi<EMAIL_ADDRESS>
# Diversification in Session-based News Recommender Systems

Alireza Gharahighehi  Celine Vens

(Received: date / Accepted: date)

###### Abstract

Recommender systems are widely applied in digital platforms such as news
websites to personalize services based on user preferences. In news websites
most of users are anonymous and the only available data is sequences of items
in anonymous sessions. Due to this, typical collaborative filtering methods,
which are highly applied in many applications, are not effective in news
recommendations. In this context, session-based recommenders are able to
recommend next items given the sequence of previous items in the active
session. Neighborhood-based session-based recommenders has been shown to be
highly effective compared to more sophisticated approaches. In this study we
propose scenarios to make these session-based recommender systems diversity-
aware and to address the filter bubble phenomenon. The filter bubble
phenomenon is a common concern in news recommendation systems and it occurs
when the system narrows the information and deprives users of diverse
information. The results of applying the proposed scenarios show that these
diversification scenarios improve the diversity measures in these session-
based recommender systems based on four news datasets.

###### Keywords:

Session-based recommender system news recommendation diversity filter bubble
phenomenon

††journal: Personal and Ubiquitous Computing

##  1 Introduction

Nowadays recommender systems (RSs) are applied in almost every digital
platform. These platforms try to adapt their services based on user needs in
order to increase user satisfaction. RSs infer user needs and preferences
using the previous interactions and activities of the user in the platform. In
news aggregator websites, users are usually anonymous and therefore their
profiles and long-term interaction histories are not available. In this
situation the only available information is the sequence of interactions in
the active session of the (anonymous) user. Moreover, the news domain is
highly dynamic and the set of available news articles for recommendation
changes rapidly. Therefore, a news RS should focus on these characteristics to
capture recent trends and anonymous users’ short-term preferences
jugovac2018streamingrec ; karimi2018news ; gharahighehi2019news .

Session-based Recommender Systems (SBRSs) are applied, when the user’s long-
term history is not available and the item set is highly dynamic. SBRSs are
meant to recommend the next items given the sequence of visited items in the
active session of an anonymous user. SBRSs use the collaborative and
sequential information from previous sessions of anonymous users to rank and
recommend candidate items for an active session. These methods are applied in
many applications such as news recommendation gabriel2019contextual , music
recommendation and next basket prediction in e-commerce jannach2017recurrent .

RSs are mostly designed to generate accurate recommendations based on previous
interactions. As they are primarily optimized based on predictive accuracy
they can narrow the scope of users’ recommendations and tighten the filter
bubble around the user. The filter bubble phenomenon occurs when users are
exposed to similar topics and content and consequently are isolated from
diverse viewpoints and content nguyen2014exploring . Pariser pariser2011filter
describes a filter bubble as: The new generation of Internet filters … are
prediction engines, constantly creating and refining a theory of who you are
and what you’ll do and want next. Together, these engines create a unique
universe of information for each of us—what I’ve come to call a filter
bubble—which fundamentally alters the way we encounter ideas and information.

In news aggregator websites, in addition to the filter bubble phenomenon,
focusing only on accuracy can form eco-chambers and boost polarization,
radicalization, and fragmentation among users helberger2019democratic . To
mitigate these issues, diversity should be considered in the recommendation
lists to avoid recommending redundant items to the users and also to broaden
users’ horizons.

In this paper we propose diversification approaches for four state-of-art
neighborhood-based SBRSs using news article metadata. To the best of our
knowledge, most current SBRS methods only focus on providing accurate
predictions, ignoring diversity of recommendation lists. In particular, we
propose two simple yet effective methods to manipulate the candidate item
selection or the neighbor selection, resulting in more diverse recommendation
lists. We also study their combined effect. Our main question of interest is
investigating the accuracy/diversity trade-off, i.e., quantifying the accuracy
loss as a cost for the introduced diversity. For this purpose, we use an
evaluation measure that takes both goals into account, apart from the standard
accuracy or diversity measures. This is an extension of our previous study
gharahighehi2020news  where we introduced diversity in the session-based k
nearest neighbor (SKNN) method jannach2017recurrent , which is a neighborhood-
based SBRS. Our extension in this paper is fourfold:

  * •

First, instead of one model (SKNN), we generalize the diversification
approaches to several neighborhood-based SBRSs, namely, SKNN
jannach2017recurrent , vector multiplication SKNN (VSKNN)
ludewig2019performance , STAN garg2019sequence  and VSTAN ludewig2020empirical
.

  * •

Second, we provide a more comprehensive overview of related SBRS studies and
diversification approaches.

  * •

Third, we compare the results of the proposed approaches with the maximal
marginal relevance (MMR) re-ranking approach.

  * •

Finally, we extend the evaluation procedure by adding one more news dataset
and an additional evaluation measure (topics coverage) to assess the filter
bubble issue.

In the following, related studies about SBRSs and diversity are presented in
Section 2. Next, in Section 3, the proposed scenarios to diversify
recommendations of neighborhood-based SBRSs are explained. In Section 4, four
news datasets are described and the experimental setup in designing and
testing the proposed scenarios are discussed. Next, the obtained results of
applying these proposed scenarios are presented and discussed in Section 5.
Finally, we conclude and outline some future research directions in Section 6.

##  2 Related work

###  2.1 Session-based recommender systems

SBRSs are studied and applied in many applications such as news, music and
e-commerce ludewig2020empirical . In these applications usually the item set
is highly dynamic and an anonymous user can receive services without having an
explicit profile. Recent state-of-art SBRSs can be categorized to
neighborhood- and neural network-based methods.

Neighborhood-based SBRSs form neighborhoods based on the training sessions.
Item-based k nearest neighbors (IKNN) linden2003amazon  is a naive method that
has been traditionally considered as an SBRS baseline where the last item of
the active session is used to find neighbors. A generalization of IKNN is
session-based KNN (SKNN) jannach2017recurrent  that uses all of the items in
the active session to find neighbors. Ludewig et al. ludewig2019performance
proposed an extension of SKNN that gives more weights to recent events in the
session (Vector Multiplication SKNN). Garg et al. garg2019sequence  proposed a
sequence- and time-aware neighborhood SBRS (STAN) that considers recency and
position of the items in SKNN. Recently, Ludewig et al. ludewig2020empirical
proposed VSTAN, which is an extension of STAN where they add the sequential
information and weighting scheme of Vector Multiplication SKNN (VSKNN) to
STAN. We discuss SKNN, VSKNN, STAN and VSTAN in more detail in Section 3.1.

Neural network-based SBRSs have mostly used the recurrent neural network (RNN)
structure. GRU4REC hidasi2015session  is an RNN-based SBRS that uses Gated
Recurrent Units (GRU) to generate recommendations for an active session. There
are extensions of GRU4REC where the loss function and sampling approach have
been changed to enhance the performance hidasi2018recurrent ;
hidasi2016parallel . Li et al. li2017neural  proposed another RNN-based SBRS
that uses attention mechanism to predict the next item of an active session.
Their proposed model (NARM) captures the main purpose of the session to
generate recommendations. Recently, Wu et al. wu2019session  modeled the
current and the global preference of the session with a graph neural network
(SR-GNN) SBRS. In this model the sequences of items in the sessions are
represented as graph structured data.

Numerous previous studies on SBRSs ludewig2018evaluation ;
jannach2017recurrent ; ludewig2020empirical ; ludewig2019performance ;
kouki2020lab ; jugovac2018streamingrec  have shown that neighborhood-based
SBRSs surprisingly can outperform recent complex neural network-based SBRSs in
both accuracy and computational cost. More specifically, ludewig2020empirical
concluded that despite the recent surge of the newly proposed neural network-
based SBRSs, in most cases, they are unable to outperform much simpler methods
such as neighborhood-based SBRSs. For instance, they showed that VSTAN
performs better compared to the recently proposed SR-GNN w.r.t. accuracy,
while the training time of VSTAN is 10,000 times lower than SR-GNN.

###  2.2 Diversity in recommender systems

The concept of diversity was first introduced in the information retrieval
community. A diversified list is more likely to contain the user’s actual
search intent kaminskas2016diversity . In recommender systems diversification
is applied to provide a wider range of content and therefore to address the
filter bubble phenomenon. To measure the diversity of a recommendation list, a
common evaluation measure is the average pair-wise distance between items in
the ranked list smyth2001similarity . This measure is called intra-list
diversity (ILD) and a high value in ILD means the recommended list contains
items with a broad range of content. In recommender systems accuracy and
ranking play important roles. Vargas and Castells vargas2011rank  introduced a
rank and relevance sensitive intra-list diversity measure (RR-ILD) that shows
to what extent the recommender can diversify the list and preserve the
relevant items in the high ranks. It is worth mentioning that there are some
other measures related to diversity such as aggregate diversity, calibration
and fairness. Aggregate diversity or item coverage measures the portion of
available items in the catalog that are included in the recommendation lists
adomavicius2011maximizing . A calibrated recommendation list reflects the
different areas of interest of a user with their proportions in the user
history steck2018calibrated . Fairness measures the degree of parity of
recommendations for individuals or groups of users based on a sensitive factor
such as gender, ethnicity or age dwork2012fairness .

Generally there are two diversification approaches in recommendation systems:
re-ranking and diversity modeling. Re-ranking approaches such as
kelly2006enhancing ; vargas2011intent ; jambor2010optimizing ;
jugovac2017efficient  are post-processing methods that reorder the initial
ranked list generated by a baseline recommender. Jugovac et al.
jugovac2017efficient  proposed a re-ranking approach that can consider
multiple quality factors, such as accuracy and diversity, in its re-ranking
and adjusts the initial recommendation lists based on tendencies of the
individual users. While these methods are able to increase diversity, they
need additional post-processing steps that can be computationally demanding
when the initial list is long. On the other hand, diversity modeling methods
such as said2013user ; shi2012adaptive ; su2013set  adapt the main recommender
method to make it diversity-aware based on items metadata. Each of these
methods can only be applied on the specific recommender method (for instance
BPR) that is used as the main RS and therefore not applicable in other types
of RS such as SBRSs.

Although diversity has been vastly studied in user-based recommendation
systems, it has received very limited attention in SBRSs. In
gharahighehi2021personalizing  we proposed a strategy for SKNN to decide
between diversification and hybridization for music recommendations. We showed
that by hybridizing, i.e. showing more similar content, for the focused
sessions and diversifying for the broader sessions, the overall performance in
accuracy and diversity could be enhanced in the context of music
recommendation. While this idea of personalizing diversification is valid for
music recommendation, it should not be applied in any news recommender system.
In order to avoid possible undesirable side effects of algorithmic filtering
such as filter bubbles, fragmentation and radicalization
helberger2019democratic , a user should be exposed to a diverse range of ideas
and content. Therefore, a news recommender should broaden the range of
information even for focused sessions. For this reason, in the current
article, we show how the diversification approaches are able to extend the
range of news topics in recommendation lists, by maximizing the overall RR-
ILD. Furthermore, unlike music datasets, news datasets normally contain
several sources of information such as text, summary, named entities and tags,
which can be represented using article embeddings and exploiting when
constructing neighborhoods. To fully investigate and compare the potential of
neighborhood-based methods to increase diversity, we apply the diversity
enhancing techniques also to three more recent methods, besides SKNN, that
were discussed before (VSKNN, STAN and VSTAN).

##  3 Methodology

In this section we explain how we make neighborhood-based SBRSs diversity-
aware. We first explain the background related to these SBRSs, namely, SKNN
jannach2017recurrent , VSKNN ludewig2019performance , STAN garg2019sequence
and VSTAN ludewig2020empirical  in Section 3.1 and then propose our approaches
to make them diversity-aware in Section 3.2. As mentioned above, these SBRSs
are very promising in terms of accuracy and computational cost
ludewig2018evaluation ; jannach2017recurrent ; jugovac2018streamingrec ;
ludewig2020empirical ; ludewig2019performance ; kouki2020lab , compared to
more complex types of SBRS.

###  3.1 Background

SKNN jannach2017recurrent  is a memory-based SBRS that forms a neighborhood to
provide recommendations. It uses the items in the active session to select the
nearest neighbor sessions and to predict the next items in the active session.
To predict the score of a candidate item, SKNN uses the similarity of the item
set in the neighbor sessions that include the candidate item, with the item
set in the active session. This score is calculated using Eq. 1:

$\hat{r}_{{}_{SKNN}}(i,s)=\sum_{n\in N_{s}}w_{s,n}\times 1_{n}(i)$ (1)

where $\hat{r}_{{}_{SKNN}}(i,s)$ is the predicted score for active session $s$
and candidate item $i$, $w_{s,n}$ is the similarity between session $s$ and
session $n$, $1_{n}(i)$ is an indicator function that verifies whether item
$i$ exists in session $n$ and $N_{s}$ is the set of neighbor sessions for
session $s$. To calculate the similarity between two sessions one can use
cosine distance measure:

$w_{s,n}=\dfrac{\vv{s}\cdot\vv{n}}{\lVert\vv{s}\rVert\times\lVert\vv{n}\rVert}$
(2)

where $\vv{s}$ and $\vv{n}$ are the binary vector representations for session
$s$ and $n$ over the item set in training sessions. SKNN does not consider the
order of items in sessions and time in its predictions.

VSKNN ludewig2019performance  is an extension of SKNN that puts more focus on
the recent items in the active session. In this method, instead of binary
values, real values are assigned to the items of the session based on their
order. The last item of the active session gets the value 1 and the values of
the less recent items in the vector are decayed based on a decay function. The
choice of the decay function is a hyperparameter. Ludewig et al.
ludewig2019performance  proposed linear, logarithmic, quadratic and inverse
decay functions. For instance, using the inverse decay function, if the active
session contains three items then the last item in the session gets the value
one, the next one gets the value $\frac{1}{2}$ and the fist item of the
session gets the value $\frac{1}{3}$ in the session vector. After constructing
this vector with real values using one of these decay functions, $w_{s,n}$ is
defined as the similarity between the binary vector of the neighbor session
and this real-valued vector of the active session.

STAN garg2019sequence  is another extension of SKNN that considers three
additional components in SKNN to calculate relevance scores: (1) position of
items in the active session, (2) recency of the neighbor session and (3)
position of the candidate item in the neighbor session. They proposed three
decay functions to include these three components in SKNN. To focus more on
the more recent items in the active session (the first additional component),
real values are calculated for items in the active session using the following
exponential decay function:

$s_{i}=exp(\dfrac{p(s,i)-l(s)}{\lambda_{1}})$ (3)

where $i$ is an item in the active session $s$, $p(s,i)$ is the position of
item $i$ in session $s$, $l(s)$ is the position of the last item of session
$s$ and $\lambda_{1}$ is a hyperparameter (e.g. if $i$ is the third item of a
session with length 4, $p(s,i)=3$ and $l(s)=4$). The similarity of this real-
valued vector $\vv{s}$ and the binary vector of neighbor session $n$ can be
calculated using Eq. 2. This is similar to VSKNN but in STAN the exponential
decay function (Eq. 5) is used to calculate the real-values in vector
$\vv{s}$.

To give a higher weight to the more recent neighbor session w.r.t. active
session (the second additional component), they used the following decay
function:

$w_{t}(s,n)=exp(-\dfrac{t(s)-t(n)}{\lambda_{2}})$ (4)

where $t(s)$ and $t(n)$ are the timestamps of the last event/item of the
active session $s$ and the neighbor session $n$ ($t(s)>t(n)$) and
$\lambda_{2}$ is a hyperparameter. Finally to consider the position of the
candidate item in the neighbor session (the third additional component), the
following decay function is proposed:

$w_{n}(i,n)=exp(-\dfrac{\lvert p(i,n)-p(i^{\ast},n)\rvert}{\lambda_{3}})\times
I_{n}(i)\times I_{n\cap s}(i^{\ast})$ (5)

where $i^{\ast}$ is the most recent item that occurred in the active session
$s$ and that also occurred in the neighbor session $n$ and $\lambda_{3}$ is a
hyperparameter. The final predicted relevance score in STAN is:

$\hat{r}_{{}_{STAN}}(i,s)=\sum_{n\in N_{s}}w_{s,n}\times w_{t}(s,n)\times
w_{n}(i,n)\times 1_{n}(i)$ (6)

VSTAN ludewig2020empirical  is the most recent extension of SKNN that has two
additional components compared to STAN: (1) position of the last item in the
active session that also exists in the neighbor session (Eq. 7) and (2) an
idf111Inverse Document Frequency weighting scheme (Eq.8) that gives lower
weights to frequent candidate items:

$\displaystyle w_{s}(n,s)=exp(\dfrac{p(i^{\ast},s)-l(s)}{\lambda_{4}})\times
I_{n\cap s}(i^{\ast})$ (7) $\displaystyle idf_{i}=log(\dfrac{\lvert
S\rvert}{\lvert\\{s\in S:i\in s\\}\rvert})\times\lambda_{idf}$ (8)

where $w_{s}(n,s)$ is the position weight of the last item ($i^{\ast}$) in
active session $s$ that also exists in neighbor session $n$, $idf_{i}$ is the
$idf$ weight of candidate item $i$, $S$ is the set of training sessions and
$\lambda_{4}$ and $\lambda_{idf}$ are hyperparameters. The relevance score
using VSTAN is calculated by:

$\hat{r}_{{}_{VSTAN}}(i,s)=\sum_{n\in N_{s}}w_{s,n}\times w_{t}(n,s)\times
w_{n}(i,n)\times w_{s}(n,s)\times idf_{i}\times 1_{n}(i)$ (9)

###  3.2 Diversification approaches

To make these neighborhood-based SBRSs (SKNN, VSKNN, STAN and VSTAN)
diversity-aware we propose three diversification approaches: (1) diverse
neighbor, (2) diverse candidate item and (3) their combination. We need a
content representation for news articles and a distance measure to apply these
approaches. We use article embeddings, which are described in the next
section, as the content representation, and cosine similarity measure in this
study.

#### Diverse neighbor:

In this approach we consider higher weights for the neighbor sessions that
have more content diversity. Diversity of a neighbor session is the average
content dissimilarity of pairs of items in the session. Neighbors with higher
average content dissimilarity are more probable to bring more diverse
recommendations. The diversity of a neighbor session can be calculated as
following:

$d_{n}=\dfrac{\sum_{i\in n}\sum_{j\in n\texttt{\char
92\relax}\\{i\\}}dist_{c}(i,j)}{|n|(|n|-1)}\\\ $ (10)

#### Diverse candidate item:

In this approach we consider a higher diversity weight for a candidate item
with higher average dissimilarity with items of the active session. This
weight is calculated using the embedding of the candidate item and the
embeddings of the articles in the active session based on the following
equation:

$d_{i,s}=\dfrac{\sum_{j\in s}dist(i,j)}{|s|}$ (11)

where $d_{i,s}$ is the diversity weight of candidate item $i$ and active
session $s$ and $dist(i,j)$ is the dissimilarity between embeddings of item
$i$ and $j$. These two proposed diversification approaches can be
simultaneously applied to enhance diversity.

To make SKNN diversity-aware we apply these two introduced diversity weights:

$\hat{r}_{{}_{d-SKNN}}(i,s)=d_{i,s}\sum_{n\in N_{s}}w_{s,n}\times d_{n}\times
1_{n}(i)\\\ $ (12)

where $d_{n}$ is the diversity of session $n$ and $d_{i,s}$ is the average
content dissimilarity of item $i$ and the items in session $s$. Similar
weights ($d_{n}$ and $d_{i,s}$) are added to VSKNN, STAN and VSTAN to make
them diversity-aware.

To compare the results of the proposed approaches with a diversification
baseline, we use the Maximal Marginal Relevance (MMR) re-ranking approach
kaminskas2016diversity . In this approach multiple performance criteria (here
accuracy and diversity) are used to re-rank items of an initial recommendation
list.

##  4 Dataset and Experimental Setup

We evaluate three diversity boosting approaches, namely, diverse neighbor,
diverse candidate item and the combination of both with the original methods.
We use four news datasets, namely Adressa 222http://reclab.idi.ntnu.no/dataset
gulla2017adressa , Globo.com 333https://www.kaggle.com/gspmoreira/news-portal-
user-interactions-by-globocom gabriel2019contextual , Kwestie and
Roularta444The pre-processed version of the datasets are available upon
request. The last two datasets were obtained from Roularta Media Group, a
Belgian multimedia group., which are described in Table 1, to evaluate the
performance of these approaches. To calculate content dissimilarity, as
explained in the previous section, we should form content representations for
news articles. The CNN-based deep neural network approach proposed by
gabriel2019contextual  is used to generate article embeddings for news
articles of these datasets based on article title, summary, full text and
tags.

Table 1: Datasets descriptions | Roularta | Kwestie | Globo.com | Adressa
---|---|---|---|---
# Sessions | 324,505 | 297,342 | 1,048,389 | 5,607,303
# Interactions | 5,529,007 | 922,680 | 2,986,226 | 8,894,565
# Items | 29,641 | 22,465 | 45,559 | 27,805
Timespan | 26 days | 28 days | 16 days | 27 days
Language | Dutch and French | Dutch | Portuguese | Norwegian

We compare performance of the SBRSs explained in the previous section (SKNN,
VSKNN, STAN and VSTAN) with the proposed diversification approaches, diverse
candidate item (_I), diverse neighbor (_D), the combined approaches (_ID) and
the MMR re-ranking approach (_Re) based on five performance measures, namely
precision (P@k), recall (R@k), expected intra-list diversity (ILD@k), rank and
relevance sensitive expected intra-list diversity (RR-ILD@k) vargas2011rank
and covered topics (CT@k). P@k and R@k are standard information retrieval
accuracy measures that evaluate the model in predicting the relevant items in
the top k recommendations. ILD@k is the average content dissimilarity between
pairs of items in top k recommendation and RR-ILD@k is another diversity
measure, that considers the ranks and relevance of top k recommendation in
calculating diversity. CT@k is the average number of unique topics/keywords in
recommendation lists. CT@k is used to evaluate the merits of the
diversification approaches in expanding the range of recommended content to
the users by enhancing the number of news topics in the recommendation lists
and consequently addressing the filter bubble issue. A recommendation list
with more unique topics is less probable to tighten the filter bubble around
the user.

To form the train and test sets, we use the approach by ludewig2018evaluation
. In this approach, the datasets are split into five partitions with the same
duration. The sessions in the last day of each partition are considered as
test sessions, and the sessions from the other days of the partition as
training sessions. In these test sessions the last two items are regarded as
the test items. The accuracy measures (P@k and R@k) are calculated based on
the ability of the model in predicting these test items in the test sessions.
The reported performance in the next section is the average performance of the
model over these five partitions.

As mentioned in the previous section, there are some hyperparameters to be
tuned. We consider the last day of the first training set as the validation
set and tune the hyperparameters based on P@10. The final values of the tuned
hyperparameters are reported in Table 2. In order to decide which
diversification approach should be applied, one can select the approach based
on the performance in the validation set. In our experiments, we select the
approach that yields the highest RR-ILD@k in the validation set, as it
provides a trade-off between accuracy and diversity. The selected approach for
each pair of method-dataset is indicated in Table 2.

Table 2: The final tuned values for the hyperparameters of the neighborhood-
based SBRSs

Method |  Hyper-parameters | Range | Adressa | Globo | Kwestie | Roularta
---|---|---|---|---|---|---
SKNN |  sample size | [500,3000] | 2500 | 700 | 500 | 500
# Neighbors | [50,500] | 300 | 500 | 300 | 200
approach | [_I,_D,_ID] | _D | _ID | _I | _ID
VSKNN |  sample size | [500,3000] | 2500 | 500 | 2500 | 500
# Neighbors | [50,500] | 300 | 100 | 300 | 50
# Weighting | [inverse,linear,quad,log] | inverse | log | log | quad
approach | [_I,_D,_ID] | _D | _ID | _I | _ID
STAN |  sample size | [500,3000] | 1000 | 500 | 2500 | 1000
# Neighbors | [50,500] | 100 | 50 | 100 | 50
$\lambda_{spw}$ | [0.1,5] | 0.80 | 1.15 | 5.00 | 2.55
$\lambda_{snh}$ | [1,5] | 1.29 | 1.00 | 4.42 | 1.86
$\lambda_{inh}$ | [0.5,5] | 5.00 | 2.75 | 5.00 | 4.68
|  approach | [_I,_D,_ID] | _D | _I | _I | _ID
VSTAN |  sample size | [500,3000] | 1000 | 700 | 2500 | 500
# Neighbors | [50,500] | 100 | 100 | 300 | 50
$\lambda_{spw}$ | [0.1,5] | 1.50 | 1.85 | 3.95 | 3.60
$\lambda_{snh}$ | [1,5] | 1.57 | 3.57 | 2.71 | 2.14
$\lambda_{inh}$ | [0.5,5] | 3.39 | 2.11 | 4.68 | 3.71
$\lambda_{ipw}$ | [0.1,5] | 3.95 | 1.50 | 3.25 | 3.25
$\lambda_{idf}$ | [0.1,5] | 0.10 | 4.3 | 1.50 | 0.10
|  approach | [_I,_D,_ID] | _D | _I | _I | _ID

##  5 Result and discussion

The results of applying the proposed approaches555The source code is available
at https://github.com/alirezagharahi/d_SBRS for Adressa dataset w.r.t. P@10,
R@10, ILD@10, RR-ILD@10 and CT@10 are reported in Table 3. As is shown in
Table 3, while SKNN is the simplest form of neighborhood-based SBRSs, it has
the best accuracy and diversity among these methods. Among the proposed
approaches, the _ID approach has the best performance in ILD@10 but it
deteriorates the accuracy significantly. In SKNN and VSKNN the re-ranking
approach performs better w.r.t. RR-ILD@10 compared to the proposed approaches.
On the other hand, in STAN and VSTAN, the _D approach has the best RR-ILD@10
and therefore can provide a better trade-off between diversity and accuracy.
It also yields the two overall highest RR-ILD@10 values among all optimized
neighborhood based models.

Table 3: Results of diversity-aware neighborhood-based SBRSs for Adressa. Bold values highlight the best performing method and italic values indicate improvement w.r.t. the base method. Methods | P@10 | R@10 | ILD@10 | RR-ILD@10 | CT@10
---|---|---|---|---|---
SKNN | 0.0788 | 0.3942 | 0.1849 | 0.0198 | 19.28
SKNN_I | 0.0654 | 0.3272 | 0.2139 | 0.0179 | 19.99
SKNN_D | 0.0786 | 0.3929 | 0.1927 | 0.0204 | 19.62
SKNN_ID | 0.0631 | 0.3153 | 0.2214 | 0.0177 | 20.14
SKNN_Re | 0.0800 | 0.4001 | 0.2139 | 0.0224 | 20.67
VSKNN | 0.0780 | 0.3898 | 0.1848 | 0.0196 | 19.26
VSKNN_I | 0.0540 | 0.2699 | 0.2052 | 0.0143 | 19.77
VSKNN_D | 0.0768 | 0.3840 | 0.1920 | 0.0202 | 19.79
VSKNN_ID | 0.0523 | 0.2614 | 0.2114 | 0.0140 | 19.99
VSKNN_Re | 0.0760 | 0.3800 | 0.2134 | 0.0219 | 20.93
STAN | 0.0784 | 0.3920 | 0.1838 | 0.0196 | 19.91
STAN_I | 0.0572 | 0.2858 | 0.2050 | 0.0151 | 20.53
STAN_D | 0.0751 | 0.3755 | 0.2244 | 0.0230 | 21.01
STAN_ID | 0.0497 | 0.2483 | 0.2405 | 0.0153 | 21.18
STAN_Re | 0.0778 | 0.3889 | 0.1907 | 0.0200 | 20.26
VSTAN | 0.0784 | 0.3920 | 0.1835 | 0.0195 | 19.43
VSTAN_I | 0.0570 | 0.2851 | 0.2059 | 0.0153 | 20.38
VSTAN_D | 0.0745 | 0.3723 | 0.2265 | 0.0231 | 21.01
VSTAN_ID | 0.0489 | 0.2444 | 0.2447 | 0.0156 | 20.87
VSTAN_Re | 0.0782 | 0.3911 | 0.1890 | 0.0199 | 19.79

For the Globo dataset, the results of applying the proposed approaches are
shown in Table 4. According to this table, all the proposed approaches enhance
diversity and number of unique topics in recommendations with the cost of
reduced accuracy. In neighborhood-based SBRSs, STAN has the best performance
in all measures. The _ID approach has the best RR-ILD@10 and CT@10 in SKNN and
VSKNN, but the _I approach has better RR-ILD@10 in STAN and VSTAN among the
other approaches. The re-ranking approach can not outperform the best
performing proposed approach in RR-ILD@10 and therefore is less effective.

Table 4: Results of diversity-aware neighborhood-based SBRSs for Globo. Bold values highlight the best performing method and italic values indicate improvement w.r.t. the base method. Methods | P@10 | R@10 | ILD@10 | RR-ILD@10 | CT@10
---|---|---|---|---|---
SKNN | 0.0444 | 0.2218 | 0.3382 | 0.0162 | 7.04
SKNN_I | 0.0417 | 0.2085 | 0.3613 | 0.0191 | 7.59
SKNN_D | 0.0441 | 0.2206 | 0.3466 | 0.0166 | 7.28
SKNN_ID | 0.0408 | 0.2039 | 0.3668 | 0.0192 | 7.79
SKNN_Re | 0.0422 | 0.2108 | 0.3699 | 0.0173 | 7.75
VSKNN | 0.0461 | 0.2304 | 0.3383 | 0.0170 | 7.12
VSKNN_I | 0.0450 | 0.2250 | 0.3561 | 0.0215 | 7.54
VSKNN_D | 0.0460 | 0.2302 | 0.3463 | 0.0175 | 7.32
VSKNN_ID | 0.0442 | 0.2210 | 0.3634 | 0.0218 | 7.73
VSKNN_Re | 0.0422 | 0.2110 | 0.3696 | 0.0176 | 7.79
STAN | 0.0468 | 0.2341 | 0.3414 | 0.0174 | 7.15
STAN_I | 0.0454 | 0.2271 | 0.3602 | 0.0218 | 7.73
STAN_D | 0.0432 | 0.2161 | 0.3619 | 0.0176 | 7.63
STAN_ID | 0.0413 | 0.2066 | 0.3727 | 0.0206 | 8.10
STAN_Re | 0.0457 | 0.2286 | 0.3551 | 0.0178 | 7.46
VSTAN | 0.0448 | 0.2241 | 0.3382 | 0.0165 | 7.06
VSTAN_I | 0.0438 | 0.2189 | 0.3632 | 0.0207 | 7.87
VSTAN_D | 0.0408 | 0.2041 | 0.3585 | 0.0166 | 7.57
VSTAN_ID | 0.0378 | 0.1891 | 0.3754 | 0.0193 | 8.20
VSTAN_Re | 0.0448 | 0.2241 | 0.3414 | 0.0166 | 7.17

Table 5 shows the performance of the proposed methods for the Kwestie dataset.
As is shown in this table,

STAN has the best accuracy, diversity and RR-ILD@10, while VSTAN has the best
topic coverage compared to the other baselines. All the proposed
diversification approaches can improve diversity, RR-ILD@10 and topic
coverage. In SKNN and VSKNN the _I approach and the re-ranking approach have
almost same performance in RR-ILD@10. In STAN and VSTAN the _I approach can
effectively improve diversity while slightly reducing accuracy in all methods
and therefore has the best RR-ILD@10 among the diversification approaches.

Table 5: Results of diversity-aware neighborhood-based SBRS for Kwestie. Bold values highlight the best performing method and italic values indicate improvement w.r.t. the base method. Methods | P@10 | R@10 | ILD@10 | RR-ILD@10 | CT@10
---|---|---|---|---|---
SKNN | 0.0347 | 0.1735 | 0.1693 | 0.0069 | 3.71
SKNN_I | 0.0237 | 0.1185 | 0.2978 | 0.0091 | 5.40
SKNN_D | 0.0309 | 0.1543 | 0.2083 | 0.0077 | 4.07
SKNN_ID | 0.0195 | 0.0976 | 0.3223 | 0.0090 | 5.84
SKNN_Re | 0.0329 | 0.1643 | 0.2308 | 0.0090 | 4.34
VSKNN | 0.0353 | 0.1763 | 0.1685 | 0.0066 | 3.69
VSKNN_I | 0.0252 | 0.1260 | 0.2683 | 0.0087 | 5.03
VSKNN_D | 0.0312 | 0.1561 | 0.2092 | 0.0077 | 4.07
VSKNN_ID | 0.0214 | 0.1070 | 0.2913 | 0.0086 | 5.34
VSKNN_Re | 0.0321 | 0.1605 | 0.2291 | 0.0088 | 4.30
STAN | 0.0415 | 0.2075 | 0.1752 | 0.0084 | 3.80
STAN_I | 0.0341 | 0.1703 | 0.2679 | 0.0113 | 5.13
STAN_D | 0.0355 | 0.1776 | 0.2340 | 0.0093 | 4.45
STAN_ID | 0.0270 | 0.1352 | 0.3019 | 0.0111 | 5.74
STAN_Re | 0.0406 | 0.2029 | 0.1876 | 0.0085 | 3.97
VSTAN | 0.0370 | 0.1850 | 0.1817 | 0.0077 | 4.09
VSTAN_I | 0.0277 | 0.1385 | 0.2834 | 0.0106 | 5.64
VSTAN_D | 0.0327 | 0.1635 | 0.2361 | 0.0092 | 4.67
VSTAN_ID | 0.0219 | 0.1094 | 0.3061 | 0.0096 | 6.10
VSTAN_Re | 0.0370 | 0.1849 | 0.1861 | 0.0077 | 4.15

For the Roularta dataset, the results of applying the proposed approaches are
reported in Table 6. According to this table, STAN has the best accuracy,
diversity and RR-ILD@10 and VSTAN provides recommendations that cover more
unique topics compared to the other neighborhood-based baselines. All the
proposed approaches improve diversity, RR-ILD@10 and topic coverage. For VSTAN
the _I approach is the most effective way to diversify the recommendations,
while in the other neighborhood-based approaches the _ID approach is the most
effective approach among the proposed diversification approaches.

Table 6: Results of diversity-aware neighborhood-based SBRS for Roularta. Bold values highlight the best performing method and italic values indicate improvement w.r.t. the base method. Methods | P@10 | R@10 | ILD@10 | RR-ILD@10 | CT@10
---|---|---|---|---|---
SKNN | 0.0351 | 0.1754 | 0.1633 | 0.0063 | 4.34
SKNN_I | 0.0279 | 0.1395 | 0.2463 | 0.0078 | 5.16
SKNN_D | 0.0348 | 0.1742 | 0.1808 | 0.0067 | 4.60
SKNN_ID | 0.0244 | 0.1220 | 0.2679 | 0.0079 | 5.42
SKNN_Re | 0.0344 | 0.1720 | 0.1965 | 0.0072 | 4.66
VSKNN | 0.0340 | 0.1701 | 0.1626 | 0.0065 | 4.37
VSKNN_I | 0.0283 | 0.1416 | 0.2300 | 0.0088 | 5.04
VSKNN_D | 0.0298 | 0.1491 | 0.2106 | 0.0073 | 4.95
VSKNN_ID | 0.0257 | 0.1286 | 0.2522 | 0.0090 | 5.28
VSKNN_Re | 0.0333 | 0.1667 | 0.2021 | 0.0076 | 4.84
STAN | 0.0462 | 0.2311 | 0.1660 | 0.0085 | 4.37
STAN_I | 0.0419 | 0.2093 | 0.2147 | 0.0118 | 4.93
STAN_D | 0.0445 | 0.2225 | 0.1793 | 0.0088 | 4.56
STAN_ID | 0.0394 | 0.1972 | 0.2339 | 0.0119 | 5.25
STAN_re | 0.0459 | 0.2297 | 0.1701 | 0.0086 | 4.40
VSTAN | 0.0408 | 0.2041 | 0.1653 | 0.0076 | 4.46
VSTAN_I | 0.0335 | 0.1674 | 0.2365 | 0.0111 | 5.44
VSTAN_D | 0.0385 | 0.1926 | 0.1910 | 0.0083 | 4.84
VSTAN_ID | 0.0281 | 0.1403 | 0.2543 | 0.0099 | 5.74
VSTAN_re | 0.0448 | 0.2239 | 0.1688 | 0.0083 | 4.44

To summarize, the recommendation lists have more diversity when more diverse
neighbors are selected and when candidate items that have more dissimilarities
with the corresponding active sessions are recommended. In the neighborhood-
based SBRSs, the nearest neighbors convey the collaborative information and,
according to the results, using more diverse collaborative information gives a
better trade-off between diversity and accuracy, i.e., it has always higher
RR-ILD@10 compared to the original methods. The diverse candidate item
approach is based on content-based information and in all methods and datasets
it can bring more diverse recommendations. While all the proposed approaches
are able to enhance ILD@10, their capabilities should be evaluated based on
RR-ILD@10 as it shows to what extent the trade-off between diversity and
accuracy is achieved. These two diversity metrics may have different behavior
as they measure the diversity of recommendation lists differently (e.g. in
Table 5 the best approach based on ILD@10 is _ID which is not the case for RR-
ILD@10). Almost in all cases, at least one of the proposed diversification
approaches can enhance RR-ILD@10 compared to the original neighborhood-based
methods and also can outperform the re-ranking approach. The choice of the
best and the most effective approach depends on the SBRS method and the
dataset. This is in line with the results of ludewig2018evaluation ;
ludewig2020empirical  where the choice of best performing SBRS depends on the
dataset. Important to note is that the best performing diversification
approaches in RR-ILD@10 for all methods and datasets are in line with the
approaches selected in the validation phase (See Table 2).

To investigate how the proposed approaches address the filter bubble
phenomenon, the recommended lists generated by the diversity-aware approaches
and the original models are compared w.r.t. the average number of unique
topics (CT@10) that they recommend to the users. All the proposed approaches
enhance CT@10 in all datasets, which indicates that the diversification
approaches broaden the user reading experience and offer a wider range of
content by recommending articles from more diverse topics and therefore
mitigate the filter bubble issue. The topics in these datasets refer to the
main themes (e.g. entertainment, political news, health, …) that form the
content of these news websites. In the Globo, Kwestie and Roularta datasets
each article is connected to one topic and therefore the maximum value for
CT@10 in these datasets is 10. In the Adressa dataset multiple topics can be
assigned to each article and therefore CT@10 can be higher than 10. By
covering more topics in recommendation lists, users are exposed to news from
different themes and therefore have a more diverse reading experience. As is
shown in Table 7, in all datasets the Pearson correlations between diversity
(ILD@10) and CT@10 of the methods are strong and statistically significant
(p-values are less than 0.00001). Therefore, diversification can widen the
filter bubbles around the users by recommending news from more diverse topics.

Table 7: Pearson correlation between diversity (ILD@10) and topic coverage (CT@10) | Roularta | Kwestie | Globo.com | Adressa
---|---|---|---|---
R | 0.9594 | 0.9541 | 0.9251 | 0.8411
P-value | $<.00001$ | $<.00001$ | $<.00001$ | $<.00001$

The proposed diversification approaches have some limitations. Users’
perception of diversity may differ from the diversity that we aim to enhance
in this study ekstrand2014user . A user study should be performed to verify
whether users perceive the enhanced diversity in the recommendations lists.
Moreover, in this study we only focused on content diversity but ideally a
diversity-aware news recommender should also recommend diverse news w.r.t.
sentiment and polarity.

##  6 Conclusion

The main contribution of this study is to make neighborhood-based session-
based recommender systems (SBRSs) diversity-aware. In news aggregator
websites, focusing only on predictive performance of the recommender, can
tighten the filter bubbles around the users and can intensify polarization and
fragmentation among them. Diversification is a way to address these issues in
news recommenders. We proposed scenarios to diversify the recommendation lists
generated by these SBRSs. According to the results, all the scenarios improve
diversity in all news datasets. The choice of the most effective approach
depends on the method and the dataset. In all cases we can find a trade-off
between diversity and accuracy. The decision as to which diversification
approach should be applied can be seen as a hyperparameter and selected based
on the validation set. The diversification approach which has the highest RR-
ILD@10 in the validation phase should be applied on the test set. Moreover,
diversification addresses the filter bubble phenomenon by increasing the
number of unique news topics in the recommendation lists.

For future extension, we will assess the possibility of enhancing the
diversity of model-based SBRSs such as GRU4REC hidasi2018recurrent  and
CHAMELEON gabriel2019contextual . In the loss functions of these model-based
methods, regularization terms that penalize similar contents should be
applied. Moreover, we will apply the proposed scenarios on other domains such
as music and e-commerce recommenders. In these domains there are other types
of metadata such as lyrics, genres, artists, item descriptions or a hierarchy
of item categories that should be used to diversify recommendations. Finally,
we will investigate how the performance of neighborhood-based SBRSs can be
enhanced in other performance criteria such as fairness GHARAHIGHEHI2021102663
, coverage hyper_news_multi  and serendipity.

###### Acknowledgements.

This work was executed within the imec.icon project NewsButler, a research
project bringing together academic researchers (KU Leuven, VUB) and industry
partners (Roularta Media Group, Bothrs and ML6). The NewsButler project is co-
financed by imec and receives project support from Flanders Innovation &
Entrepreneurship (project nr. HBC.2017.0628). The authors also acknowledge
support from the Flemish Government (AI Research Program). “This is a post-
peer-review, pre-copy edit version of an article published in Personal and
Ubiquitous Computing. The final authenticated version is available online at:
http://dx.doi.org/10.1007/s00779-021-01606-4”.

## Conflict of interest

The authors declare that they have no conflict of interest.

## References

  * (1) Adomavicius, G., Kwon, Y.: Maximizing aggregate recommendation diversity: A graph-theoretic approach.  In: Proc. of the 1st International Workshop on Novelty and Diversity in Recommender Systems (DiveRS 2011), pp. 3–10. Citeseer (2011)
  * (2) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness.  In: Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214–226 (2012)
  * (3) Ekstrand, M.D., Harper, F.M., Willemsen, M.C., Konstan, J.A.: User perception of differences in recommender algorithms.  In: Proceedings of the 8th ACM Conference on Recommender systems, pp. 161–168 (2014)
  * (4) Gabriel De Souza, P.M., Jannach, D., Da Cunha, A.M.: Contextual hybrid session-based news recommendation with recurrent neural networks.  IEEE Access 7, 169185–169203 (2019)
  * (5) Garg, D., Gupta, P., Malhotra, P., Vig, L., Shroff, G.: Sequence and time aware neighborhood for session-based recommendations: Stan.  In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1069–1072 (2019)
  * (6) Gharahighehi, A., Vens, C.: Extended bayesian personalized ranking based on consumption behavior.  In: Postproceedings of the 31st Benelux Conference on Artificial Intelligence (BNAIC 2019) and the 28th Belgian Dutch Conference on Machine Learning (Benelearn 2019). Springer (2020)
  * (7) Gharahighehi, A., Vens, C.: Making session-based news recommenders diversity-aware.  In: OHARS’20: Workshop on Online Misinformation- and Harm-Aware Recommender Systems (2020)
  * (8) Gharahighehi, A., Vens, C.: Personalizing diversity versus accuracy in session-based recommender systems.  SN Computer Science 2(1), 1–12 (2021)
  * (9) Gharahighehi, A., Vens, C., Pliakos, K.: Multi-stakeholder news recommendation using hypergraph learning.  In: ECML PKDD 2020 Workshops, pp. 531–535. Springer International Publishing, Cham (2020)
  * (10) Gharahighehi, A., Vens, C., Pliakos, K.: Fair multi-stakeholder news recommender system with hypergraph ranking.  Information Processing & Management 58(5), 102663 (2021).  DOI https://doi.org/10.1016/j.ipm.2021.102663
  * (11) Gulla, J.A., Zhang, L., Liu, P., Özgöbek, Ö., Su, X.: The adressa dataset for news recommendation.  In: Proceedings of the international conference on web intelligence, pp. 1042–1048 (2017)
  * (12) Helberger, N.: On the democratic role of news recommenders.  Digital Journalism 7(8), 993–1012 (2019)
  * (13) Hidasi, B., Karatzoglou, A.: Recurrent neural networks with top-k gains for session-based recommendations.  In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 843–852 (2018)
  * (14) Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks.  arXiv preprint arXiv:1511.06939 (2015)
  * (15) Hidasi, B., Quadrana, M., Karatzoglou, A., Tikk, D.: Parallel recurrent neural network architectures for feature-rich session-based recommendations.  In: Proceedings of the 10th ACM conference on recommender systems, pp. 241–248. ACM (2016)
  * (16) Jambor, T., Wang, J.: Optimizing multiple objectives in collaborative filtering.  In: Proceedings of the fourth ACM conference on Recommender systems, pp. 55–62 (2010)
  * (17) Jannach, D., Ludewig, M.: When recurrent neural networks meet the neighborhood for session-based recommendation.  In: Proceedings of the Eleventh ACM Conference on Recommender Systems, pp. 306–310 (2017)
  * (18) Jugovac, M., Jannach, D., Karimi, M.: Streamingrec: a framework for benchmarking stream-based news recommenders.  In: Proceedings of the 12th ACM Conference on Recommender Systems, pp. 269–273 (2018)
  * (19) Jugovac, M., Jannach, D., Lerche, L.: Efficient optimization of multiple recommendation quality factors according to individual user tendencies.  Expert Systems with Applications 81, 321–331 (2017)
  * (20) Kaminskas, M., Bridge, D.: Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems.  ACM Transactions on Interactive Intelligent Systems (TiiS) 7(1), 1–42 (2016)
  * (21) Karimi, M., Jannach, D., Jugovac, M.: News recommender systems–survey and roads ahead.  Information Processing & Management 54(6), 1203–1227 (2018)
  * (22) Kelly, J.P., Bridge, D.: Enhancing the diversity of conversational collaborative recommendations: a comparison.  Artificial Intelligence Review 25(1-2), 79–95 (2006)
  * (23) Kouki, P., Fountalis, I., Vasiloglou, N., Cui, X., Liberty, E., Al Jadda, K.: From the lab to production: A case study of session-based recommendations in the home-improvement domain.  In: Fourteenth ACM Conference on Recommender Systems, pp. 140–149 (2020)
  * (24) Li, J., Ren, P., Chen, Z., Ren, Z., Lian, T., Ma, J.: Neural attentive session-based recommendation.  In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1419–1428 (2017)
  * (25) Linden, G., Smith, B., York, J.: Amazon. com recommendations: Item-to-item collaborative filtering.  IEEE Internet computing (1), 76–80 (2003)
  * (26) Ludewig, M., Jannach, D.: Evaluation of session-based recommendation algorithms.  User Modeling and User-Adapted Interaction 28(4-5), 331–390 (2018)
  * (27) Ludewig, M., Mauro, N., Latifi, S., Jannach, D.: Performance comparison of neural and non-neural approaches to session-based recommendation.  In: Proceedings of the 13th ACM Conference on Recommender Systems, pp. 462–466 (2019)
  * (28) Ludewig, M., Mauro, N., Latifi, S., Jannach, D.: Empirical analysis of session-based recommendation algorithms.  User Modeling and User-Adapted Interaction pp. 1–33 (2020)
  * (29) Nguyen, T.T., Hui, P.M., Harper, F.M., Terveen, L., Konstan, J.A.: Exploring the filter bubble: the effect of using recommender systems on content diversity.  In: Proceedings of the 23rd international conference on World wide web, pp. 677–686 (2014)
  * (30) Pariser, E.: The filter bubble: What the Internet is hiding from you.  Penguin UK (2011)
  * (31) Said, A., Fields, B., Jain, B.J., Albayrak, S.: User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm.  In: Proceedings of the 2013 conference on Computer supported cooperative work, pp. 1399–1408 (2013)
  * (32) Shi, Y., Zhao, X., Wang, J., Larson, M., Hanjalic, A.: Adaptive diversification of recommendation results via latent factor portfolio.  In: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pp. 175–184 (2012)
  * (33) Smyth, B., McClave, P.: Similarity vs. diversity.  In: International conference on case-based reasoning, pp. 347–361. Springer (2001)
  * (34) Steck, H.: Calibrated recommendations.  In: Proceedings of the 12th ACM conference on recommender systems, pp. 154–162 (2018)
  * (35) Su, R., Yin, L., Chen, K., Yu, Y.: Set-oriented personalized ranking for diversified top-n recommendation.  In: Proceedings of the 7th ACM conference on Recommender systems, pp. 415–418 (2013)
  * (36) Vargas, S., Castells, P.: Rank and relevance in novelty and diversity metrics for recommender systems.  In: Proceedings of the fifth ACM conference on Recommender systems, pp. 109–116 (2011)
  * (37) Vargas, S., Castells, P., Vallet, D.: Intent-oriented diversity in recommender systems.  In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 1211–1212 (2011)
  * (38) Wu, S., Tang, Y., Zhu, Y., Wang, L., Xie, X., Tan, T.: Session-based recommendation with graph neural networks.  In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 346–353 (2019)


# Inducing Multilingual Text Analysis Tools Using Bidirectional Recurrent
Neural Networks

Othman Zennaki
CEA, LIST, LVIC
Gif-sur-Yvette, France
<EMAIL_ADDRESS>

&Nasredine Semmar
CEA, LIST, LVIC
Gif-sur-Yvette, France
<EMAIL_ADDRESS>

&Laurent Besacier
LIG, Univ. Grenoble-Alpes
Grenoble, France
<EMAIL_ADDRESS>

###### Abstract

This work focuses on the rapid development of linguistic annotation tools for
resource-poor languages. We experiment several cross-lingual annotation
projection methods using Recurrent Neural Networks (RNN) models. The
distinctive feature of our approach is that our multilingual word
representation requires only a parallel corpus between the source and target
language. More precisely, our method has the following characteristics: (a) it
does not use word alignment information, (b) it does not assume any knowledge
about foreign languages, which makes it applicable to a wide range of
resource-poor languages, (c) it provides truly multilingual taggers. We
investigate both uni- and bi-directional RNN models and propose a method to
include external information (for instance low level information from POS) in
the RNN to train higher level taggers (for instance, super sense taggers). We
demonstrate the validity and genericity of our model by using parallel corpora
(obtained by manual or automatic translation). Our experiments are conducted
to induce cross-lingual POS and super sense taggers.

##  1 Introduction

In order to minimize the need for annotated resources (produced through manual
annotation, or by manual check of automatic annotation), several research
works were interested in building Natural Language Processing (NLP) tools
based on unsupervised or semi-supervised approaches [Collins and Singer, 1999,
Klein, 2005, Goldberg, 2010]. For example, NLP tools based on cross-language
projection of linguistic annotations achieved good performances in the early
2000s [Yarowsky et al., 2001]. The key idea of annotation projection can be
summarized as follows: through word alignment in parallel text corpora, the
annotations are transferred from the source (resource-rich) language to the
target (under-resourced) language, and the resulting annotations are used for
supervised training in the target language. However, automatic word alignment
errors [Fraser and Marcu, 2007] limit the performance of these approaches.

Our work is built upon these previous contributions and observations. We
explore the possibility of using Recurrent Neural Networks (RNN) to build
multilingual NLP tools for resource-poor languages analysis. The major
difference with previous works is that we do not explicitly use word alignment
information. Our only assumption is that parallel sentences (source-target)
are available and that the source part is annotated. In other words, we try to
infer annotations in the target language from sentence-based alignments only.
While most NLP researches on RNN have focused on monolingual
tasks111Exceptions are the recent propositions on Neural Machine Translation
[Cho et al., 2014, Sutskever et al., 2014] and sequence labeling [Collobert et
al., 2011, Graves, 2012], this paper, however, considers the problem of
learning multilingual NLP tools using RNN.

11footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/

Contributions In this paper, we investigate the effectiveness of RNN
architectures — Simple RNN (SRNN) and Bidirectional RNN (BRNN) — for
multilingual sequence labeling tasks without using any word alignment
information. Two NLP tasks are considered: Part Of Speech (POS) tagging and
Super Sense (SST) tagging [Ciaramita and Altun, 2006]. Our RNN architectures
demonstrate very competitive results on unsupervised training for new target
languages. In addition, we show that the integration of POS information in RNN
models is useful to build multilingual coarse-grain semantic (Super Senses)
taggers. For this, a simple and efficient way to take into account low-level
linguistic information for more complex sequence labeling RNN is proposed.

Methodology For training our multilingual RNN models, we just need as input a
parallel (or multi-parallel) corpus between a resource-rich language and one
or many under-resourced languages. Such a parallel corpus can be manually
obtained (clean corpus) or automatically obtained (noisy corpus).

To show the potential of our approach, we investigate two sequence labeling
tasks: cross-language POS tagging and multilingual Super Sense Tagging (SST).
For the SST task, we measure the impact of the parallel corpus quality with
manual or automatic translations of the SemCor [Miller et al., 1993]
translated from English into Italian (manually and automatically) and French
(automatically).

Outline The remainder of the paper is organized as follows. Section 2 reviews
related work. Section 3 describes our cross-language annotation projection
approaches based on RNN. Section 4 presents the empirical study and associated
results. We finally conclude the paper in Section 5.

##  2 Related Work

Cross-lingual projection of linguistic annotations was pioneered by ?) who
created new monolingual resources by transferring annotations from resource-
rich languages onto resource-poor languages through the use of word
alignments. The resulting (noisy) annotations are used in conjunction with
robust learning algorithms to build cheap unsupervised NLP tools [Padó and
Lapata, 2009]. This approach has been successfully used to transfer several
linguistic annotations between languages (efficient learning of POS taggers
[Das and Petrov, 2011, Duong et al., 2013] and accurate projection of word
senses [Bentivogli et al., 2004]). Cross-lingual projection requires a
parallel corpus and word alignment between source and target languages. Many
automatic word alignment tools are available, such as GIZA++ which implements
IBM models [Och and Ney, 2000]. However, the noisy (non perfect) outputs of
these methods is a serious limitation for the annotation projection based on
word alignments [Fraser and Marcu, 2007].

To deal with this limitation, recent studies based on cross-lingual
representation learning methods have been proposed to avoid using such pre-
processed and noisy alignments for label projection. First, these approaches
learn language-independent features, across many different languages [Durrett
et al., 2012, Al-Rfou et al., 2013, Täckström et al., 2013, Luong et al.,
2015, Gouws and Søgaard, 2015, Gouws et al., 2015]. Then, the induced
representation space is used to train NLP tools by exploiting labeled data
from the source language and apply them in the target language. Cross-lingual
representation learning approaches have achieved good results in different NLP
applications such as cross-language SST and POS tagging [Gouws and Søgaard,
2015], cross-language named entity recognition [Täckström et al., 2012],
cross-lingual document classification and lexical translation task [Gouws et
al., 2015], cross language dependency parsing [Durrett et al., 2012, Täckström
et al., 2013] and cross-language semantic role labeling [Titov and Klementiev,
2012].

Our approach described in next section, is inspired by these works since we
also try to induce a common language-independent feature space (crosslingual
words embeddings). Unlike ?) and ?), who use bilingual lexicons, and unlike ?)
who use word alignments between the source and target languages222to train a
bilingual representation regardless of the task our common multilingual
representation is very agnostic. We use a simple (multilingual) vector
representation based on the occurrence of source and target words in a
parallel corpus and we let the RNN learn the best internal representations
(corresponding to the hidden layers) specific to the task (SST or POS
tagging).

In this work, we learn a cross-lingual POS tagger (multilingual POS tagger if
a multilingual parallel corpus is used) based on a recurrent neural network
(RNN) on the source labeled text and apply it to tag target language text. We
explore simple and bidirectional RNN architectures (SRNN and BRNN
respectively). Starting from the intuition that low-level linguistic
information is useful to learn more complex taggers, we also introduce three
new RNN variants to take into account external (POS) information in
multilingual SST.

##  3 Unsupervised Approach Overview

To avoid projecting label information from deterministic and error-prone word
alignments, we propose to represent the word alignment information
intrinsically in a recurrent neural network architecture. The idea consists in
implementing a recurrent neural network as a multilingual sequence labeling
tool (we investigate POS tagging and SST tagging). Before describing our
cross-lingual (multilingual if a multi-parallel corpus is used) neural network
tagger, we present the simple cross-lingual projection method, considered as
our baseline in this work.

###  3.1 Baseline Cross-lingual Annotation Projection

We use direct transfer as a baseline system which is similar to the method
described in [Yarowsky et al., 2001]. First we tag the source side of the
parallel corpus using the available supervised tagger. Next, we align words in
the parallel corpus to find out corresponding source and target words. Tags
are then projected to the (resource-poor) target language. The target language
tagger is trained using any machine learning approach (we use TNT tagger
[Brants, 2000] in our experiments).

Figure 1:  Overview of the proposed model architecture for inducing
multilingual RNN taggers.

###  3.2 Proposed Approach

We propose a method for learning multilingual sequence labeling tools based on
RNN, as it can be seen in Figure 1. In our approach, a parallel or multi-
parallel corpus between a resource-rich language and one or many under-
resourced languages is used to extract common (multilingual) and agnostic
words representations. These representations, which rely on sentence level
alignment only, are used with the source side of the parallel/multi-parallel
corpus to learn a neural network tagger in the source language. Since a common
representation of source and target words is chosen, this neural network
tagger is truly multilingual and can be also used to tag texts in target
language(s).

####  3.2.1 Common Words Representation

In our agnostic representation, we associate to each word (in source and
target vocabularies) a common vector representation, namely
${V_{wi},i=1,...,N}$, where $N$ is the number of parallel sentences (bi-
sentences in the parallel corpus). If $w$ appears in i-th bi-sentence of the
parallel corpus then $V_{wi}=1$.

The idea is that, in general, a source word and its target translation appear
together in the same bi-sentences and their vector representations are close.
We can then use the RNN tagger, initially trained on source side, to tag the
target side (because of our common vector representation). This simple
representation does not require multilingual word alignments and it lets the
RNN learns the optimal internal representation needed for the annotation task
(for instance, the hidden layers of the RNN can be considered as multi-lingual
embeddings of the words).

####  3.2.2 Recurrent Neural Networks

There are two major architectures of neural networks: Feedforward [Bengio et
al., 2003] and Recurrent Neural Networks (RNN) [Schmidhuber, 1992, Mikolov et
al., 2010]. ?) showed that language models based on recurrent architecture
achieve better performance than language models based on feedforward
architecture. This is due to the fact that recurrent neural networks do not
use a context of limited size. This property led us to use, in our
experiments, the Elman recurrent architecture [Elman, 1990], in which
recurrent connections occur at the hidden layer level.

We consider in this work two Elman RNN architectures (see Figure 2): Simple
RNN (SRNN) and Bidirectional RNN (BRNN). In addition, to be able to include
low-level linguistic information in our architecture designed for more complex
sequence labeling tasks, we propose three new RNN variants to take into
account external (POS) information for multilingual Super Sense Tagging (SST).

Figure 2:  High level schema of RNN used in our work.

A. Simple RNN

In the simple Elman RNN (SRNN), the recurrent connection is a loop at the
hidden layer level. This connection allows SRNN to use at the current time
step hidden layer’s states of previous time steps. In other words, the hidden
layer of SRNN represents all previous history and not just $n-1$ previous
inputs, thus the model can theoretically represent long context.

The architecture of the SRNN considered in this work is shown in Figure 2. In
this architecture, we have 4 layers: input layer, forward (also called
recurrent or context layer), compression hidden layer and output layer. All
neurons of the input layer are connected to every neuron of forward layer by
weight matrix $I_{F}$ and $R_{F}$, the weight matrix $H_{F}$ connects all
neurons of the forward layer to every neuron of compression layer and all
neurons of the compression layer are connected to every neuron of output layer
by weight matrix $O$.

The input layer consists of a vector $w(t)$ that represents the current word
$w_{t}$ in our common words representation (all input neurons corresponding to
current word $w_{t}$ are set to 0 except those that correspond to bi-sentences
containing $w_{t}$, which are set to 1), and of vector $f(t-1)$ that
represents output values in the forward layer from the previous time step. We
name $f(t)$ and $c(t)$ the current time step hidden layers (our preliminary
experiments have shown better performance using these two hidden layers
instead of one hidden layer), with variable sizes (usually 80-1024 neurons)
and sigmoid activation function. These hidden layers represent our common
language-independent feature space and inherently capture word alignment
information. The output layer $y(t)$, given the input $w(t)$ and $f(t-1)$ is
computed with the following steps :

$f(t)=\Sigma(w(t).I_{F}(t)+f(t-1).R_{F}(t))\\\ $ (1)
$c(t)=\Sigma(f(t).H_{F}(t))\\\ $ (2) $y(t)=\Gamma(c(t).O(t))\\\ $ (3)

$\Sigma$ and $\Gamma$ are the sigmoid and the softmax functions, respectively.
The softmax activation function is used to normalize the values of output
neurons to sum up to 1. After the network is trained, the output $y(t)$ is a
vector representing a probability distribution over the set of tags. The
current word $w_{t}$ (in input) is tagged with the most probable output tag.

For many sequence labeling tasks, it is beneficial to have access to future in
addition to the past context. So, it can be argued that our SRNN is not
optimal for sequence labeling, since the network ignores future context and
tries to optimize the output prediction given the previous context only. This
SRNN is thus penalized compared with our baseline projection based on TNT
[Brants, 2000] which considers both left and right contexts. To overcome the
limitations of SRNN, a simple extension of the SRNN architecture — namely
Bidirectional recurrent neural network (BRNN) [Schuster and Paliwal, 1997] —
is used to ensure that context at previous and future time steps will be
considered.

B. Bidirectional RNN

An unfolded BRNN architecture is given in Figure 2. The basic idea of BRNN is
to present each training sequence forwards and backwards to two separate
recurrent hidden layers (forward and backward hidden layers) and then somehow
merge the results. This structure provides the compression and the output
layers with complete past and future context for every point in the input
sequence. Note that without the backward layer, this structure simplifies to a
SRNN.

C. RNN Variants

As mentioned in the introduction, we propose three new RNN variants to take
into account low level (POS) information in a higher level (SST) annotation
task. The question addressed here is: at which layer of the RNN this low level
information should be included to improve SST performance? As specified in
Figure 3, the POS information can be introduced either at input layer or at
forward layer (forward and backward layers for BRNN) or at compression layer.
In all these RNN variants, the POS of the current word is also represented
with a vector ($POS(t)$). Its dimension corresponds to the number of POS tags
in the tagset (universal tagset of ?) is used). We propose one hot vector
representation where only one value is set to 1 and corresponds to the index
of current tag (all other values are 0).

Figure 3:  SRNN variants with POS information at three levels: (a) input
layer, (b) forward layer, (c) compression layer.

####  3.2.3 Network Training

The first step in our approach is to train the neural network, given a
parallel corpus (training corpus), and a validation corpus (different from
train data) in the source language. In typical applications, the source
language is a resource-rich language (which already has an efficient tagger or
manually tagged resources). Our RNN models are trained by stochastic gradient
descent using usual back-propagation and back-propagation through time
algorithms [Rumelhart et al., 1985]. We learn our RNN models with an iterative
process on the tagged source side of the parallel corpus. After each epoch
(iteration) in training, validation data is used to compute per-token accuracy
of the model. After that, if the per-token accuracy increases, training
continues in the new epoch. Otherwise, the learning rate is halved at the
start of the new epoch. Eventually, if the per-token accuracy does not
increase anymore, training is stopped to prevent over-fitting. Generally,
convergence takes 5–10 epochs, starting with a learning rate ${\alpha}=0.1$.

The second step consists in using the trained model as a target language
tagger (using our common vector representation). It is important to note that
if we train on a multilingual parallel corpus with N languages ($N>2$), the
same trained model will be able to tag all the N languages.

Hence, our approach assumes that the word order in both source and target
languages are similar. In some languages such as English and French, word
order for contexts containing nouns could be reversed most of the time. For
example, the European Commission would be translated into la Commission
européenne. In order to deal with the word order constraints, we also combine
the RNN model with the cross-lingual projection model in our experiments.

###  3.3 Dealing with out-of-vocabulary words

For the words absent from in the initial parallel corpus, their vector
representation is a vector of zero values. Consequently, during testing, the
RNN model will use only the context information to tag the OOV words found in
the test corpus. To deal with these types of OOV words333words which do not
have a known vector representation, we use the CBOW model of [Mikolov et al.,
2013] to replace each OOV word by its closest known word in the current OOV
word context. Once the closest word is found, its common vector representation
is used (instead of the vector of zero values) at the input of the RNN.

###  3.4 Combining Simple Cross-lingual Projection and RNN Models

Since the simple cross-lingual projection model M1 and RNN model M2 use
different strategies for tagging (TNT is based on Markov models while RNN is a
neural network), we assume that these two models can be complementary. To keep
the benefits of each approach, we explore how to combine them with linear
interpolation. Formally, the probability to tag a given word w is computed as

$P_{M12}(t|w)=(\mu P_{M1}(t|w,C_{M1})+(1-\mu)P_{M2}(t|w,C_{M2}))$ (4)

where, $C_{M1}$ and $C_{M2}$ are the context of $w$ considered by M1 and M2
respectively. The relative importance of each model is adjusted through the
interpolation parameter $\mu$. The word $w$ is tagged with the most probable
tag, using the function $f$ described as

$f(w)=\arg\max_{t}(P_{M12}(t|w))$ (5)

##  4 Experiments

Our models are evaluated on two labeling tasks: Cross-language Part-Of-speech
(POS) tagging and Multilingual Super Sense Tagging (SST).

###  4.1 Multilingual POS Tagging

We applied our method to build RNN POS taggers for four target languages -
French, German, Greek and Spanish - with English as the source language.

In order to determine the effectiveness of our common words representation
described in section 3.2.1, we also investigated the use of state-of-the-art
bilingual word embeddings (using MultiVec Toolkit [Bérard et al., 2016]) as
input to our RNN.

####  4.1.1 Dataset

For French as a target language, we used a training set of $10,000$ parallel
sentences, a validation set of $1000$ English sentences, and a test set of
$1000$ French sentences, all extracted from the ARCADE II English-French
corpus [Veronis et al., 2008]. The test set is tagged with the French
TreeTagger [Schmid, 1995] and then manually checked.

For German, Greek and Spanish as a target language, we used training and
validation data extracted from the Europarl corpus [Koehn, 2005] which are a
subset of the training data used in [Das and Petrov, 2011, Duong et al.,
2013]. This choice allows us to compare our results with those of [Das and
Petrov, 2011, Duong et al., 2013, Gouws and Søgaard, 2015]. The train data set
contains $65,000$ bi-sentences ; a validation set of $10,000$ bi-sentences is
also available. For testing, we use the same test corpora as [Das and Petrov,
2011, Duong et al., 2013, Gouws and Søgaard, 2015] (bi-sentences from CoNLL
shared tasks on dependency parsing [Buchholz and Marsi, 2006]). The evaluation
metric (per-token accuracy) and the ?) universal tagset are used for
evaluation.

Model Lang. | French | German | Greek | Spanish
---|---|---|---|---
All words | OOV | All words | OOV | All words | OOV | All words | OOV
Simple Projection | 80.3 | 77.1 | 78.9 | 73.0 | 77.5 | 72.8 | 80.0 | 79.7
SRNN MultiVec | 75.0 | 65.4 | 70.3 | 68.8 | 71.1 | 65.4 | 73.4 | 62.4
SRNN | 78.5 | 70.0 | 76.1 | 76.4 | 75.7 | 70.7 | 78.8 | 72.6
BRNN | 80.6 | 70.9 | 77.5 | 76.6 | 77.2 | 71.0 | 80.5 | 73.1
BRNN - OOV | 81.4 | 77.8 | 77.6 | 77.8 | 77.9 | 75.3 | 80.6 | 74.7
Projection + SRNN | 84.5 | 78.8 | 81.5 | 77.0 | 78.3 | 74.6 | 83.6 | 81.2
Projection + BRNN | 85.2 | 79.0 | 81.9 | 77.1 | 79.2 | 75.0 | 84.4 | 81.7
Projection + BRNN - OOV | 85.6 | 80.4 | 82.1 | 78.7 | 79.9 | 78.5 | 84.4 | 81.9
(Das, 2011) | — | — | 82.8 | — | 82.5 | — | 84.2 | —
(Duong, 2013) | — | — | 85.4 | — | 80.4 | — | 83.3 | —
(Gouws, 2015a) | — | — | 84.8 | — | — | — | 82.6 | —
Table 1:  Token-level POS tagging accuracy for Simple Projection, SRNN using
MultiVec bilingual word embeddings as input, RNN444For RNN models, only one
(same) system is used to tag German, Greek and Spanish, Projection+RNN and
methods of Das & Petrov (2011), Duong et al (2013) and Gouws & Søgaard (2015).

For training, the English (source) sides of the training corpora (ARCADE II
and Europarl) and of the validation corpora are tagged with the English
TreeTagger toolkit. Using the matching provided by ?), we map the TreeTagger
and the CoNLL tagsets to the common Universal Tagset.

In order to build our baseline unsupervised tagger (based on a Simple Cross-
lingual Projection – see section 3.1), we also tag the target side of the
training corpus, with tags projected from English side through word-alignments
established by GIZA++. After tags projection, a target language POS tagger
based on TNT approach [Brants, 2000] is trained.

The combined model is built for each considered language using cross-
validation on the test corpus. First, the test corpus is split into 2 equal
parts and on each part, we estimate the interpolation parameter $\mu$
(Equation 4) which maximizes the per-token accuracy score. Then each part of
test corpus is tagged using the combined model tuned on the other part, and
vice versa (standard cross-validation procedure).

We trained MultiVec bilingual word embeddings on the parallel Europarl corpus
between English and each of the target languages considered.

####  4.1.2 Results and discussion

Table 1 reports the results obtained for the unsupervised POS tagging. We note
that the POS tagger based on bidirectional RNN (BRNN) has better performance
than simple RNN (SRNN), which means that both past and future contexts help
select the correct tag. Table 1 also shows the performance before and after
performing our procedure for handling OOVs in BRNNs. It is shown that after
replacing OOVs by the closest words using CBOW, the tagging accuracy
significantly increases.

As shown in the same table, our RNN models accuracy is close to that of the
simple projection tagger. It achieves comparable results to ?), ?) (who used
the full Europarl corpus while we use only a $65,000$ subset of it) and to ?)
(who used extra resources such as Wiktionary and Wikipedia). Interestingly,
RNN models learned using our common words representation (section 3.2.1) seem
to perform significantly better than RNN models using MultiVec bilingual word
embeddings.

It is also important to note that only one single SRNN and BRNN tagger applies
to German, Greek and Spanish; so this is a truly multilingual POS tagger!
Finally, as for several other NLP tasks such as language modelling or machine
translation (where standard and NN-based models are generally combined in
order to obtain optimal results), the combination of standard and RNN-based
approaches (Projection+_) seems necessary to further optimize POS tagging
accuracies.

###  4.2 Multilingual SST

In order to measure the impact of the parallel corpus quality on our method,
we also learn our SST models using the multilingual parallel corpus
MultiSemCor (MSC) which is the result of manual or automatic translation of
SemCor from English into Italian and French.

####  4.2.1 Dataset

SemCor The SemCor [Miller et al., 1993] is a subset of the Brown Corpus
[Kucera and Francis, 1979] labeled with the WordNet [Fellbaum, 1998] senses.

MultiSemCor The English-Italian MultiSemcor (MSC-IT-1) corpus is a manual
translation of the English SemCor to Italian [Bentivogli et al., 2004]. As we
already mentioned, we are also interested in measuring the impact of the
parallel corpus quality on our method. For this we use two translation
systems: (a) Google Translate to translate the English SemCor to Italian (MSC-
IT-2) and French (MSC-FR-2). (b) LIG machine translation system [Besacier et
al., 2012] to translate the English SemCor to French (MSC-FR-1).

Training corpus The SemCor was labeled with the WordNet synsets. However,
because we train models for SST, we convert SemCor synsets annotations to
super senses. We learn our models using the four different versions of MSC
(MSC-IT-1,2 - MSC-FR-1,2), with modified Semcor on source side.

Test Corpus To evaluate our models, we used the SemEval 2013 Task 12
(Multilingual Word Sense Disambiguation) [Navigli et al., 2013] test corpora,
which are available in 5 languages (English, French, German, Spanish and
Italian) and labeled with BabelNet [Navigli and Ponzetto, 2012] senses. We map
BabelNet senses to WordNet synsets, then WordNet synsets are mapped to super
senses.

####  4.2.2 SST Systems Evaluated

The goals of our SST experiments are twofold: first, to investigate the
effectiveness of using POS information to build multilingual super sense
tagger, secondly to measure the impact of the parallel corpus quality (manual
or automatic translation) on our RNN models (SRNN, BRNN and our proposed
variants). To summarize, we build four super sense taggers based on baseline
cross-lingual projection (see section 3.1) using four versions of MultiSemcor
(MSC-IT-1, MSC-IT-2, MSC-FR-1, MSC-FR-2) described above. Then we use the same
four versions to train our multilingual SST models based on SRNN and BRNN. For
learning our multilingual SST models based on RNN variants proposed in part
(C) of section 3.2.2, we also tag SemCor using TreeTagger (POS tagger proposed
by ?)).

####  4.2.3 Results and discussion

Our models are evaluated on SemEval 2013 Task 12 test corpora. Results are
directly comparable with those of systems which participated to this
evaluation campaign. We report two SemEval 2013 (unsupervised) system results
for comparison:

  * •

MFS Semeval 2013 : The most frequent sense is the baseline provided by SemEval
2013 for Task 12, this system is a strong baseline, which is obtained by using
an external resource (the WordNet most frequent sense).

  * •

GETALP : a fully unsupervised WSD system proposed by [Schwab et al., 2012]
based on Ant-Colony algorithm.

The DAEBAK! [Navigli and Lapata, 2010] and the UMCC-DLSI systems [Gutiérrez
Vázquez et al., 2011] have also participated to SemEval 2013 Task 12. However,
they use a supervised approach 555DAEBAK! and UMCC-DLSI for SST have obtained:
68.1% and 72.5% on Italian; 59.8% and 67.6 % on French.

Table 2 shows the results obtained by our RNN models and by two SemEval 2013
WSD systems. SRNN-POS-X and BRNN-POS-X refer to our RNN variants: In means
input layer, H1 means first hidden layer and H2 means second hidden layer. We
achieve the best performance on Italian using MSC-IT-1 clean corpus while
noisy training corpus degrades SST performance. The best results are obtained
with combination of simple projection and RNN which confirms (as for POS
tagging) that both approaches are complementary.

We also observe that the RNN approach seems more robust than simple projection
on noisy corpora. This is probably due to the fact that no word alignments are
required in our cross language RNN. Finally, BRNN-POS-H2-OOV achieves the best
performance, which shows that the integration of POS information in RNN models
and dealing with OOV words are useful to build efficient multilingual super
senses taggers. Finally, it is worth mentioning that integrating low level
(POS) information lately (last hidden layer) seems to be the best option in
our case.

Model | Italian | French
---|---|---
Baseline |  | MSC-IT-1 | MSC-IT-2 | MSC-FR-1 | MSC-FR-2
| trans man. | trans. auto | trans. auto | trans auto.
Simple Projection | 61.3 | 45.6 | 42.6 | 44.5
SST Based RNN | SRNN | 59.4 | 46.2 | 46.2 | 47.0
BRNN | 59.7 | 46.2 | 46.0 | 47.2
SRNN-POS-In | 61.0 | 47.0 | 46.5 | 47.3
SRNN-POS-H1 | 59.8 | 46.5 | 46.8 | 47.4
SRNN-POS-H2 | 63.1 | 48.7 | 47.7 | 49.8
BRNN-POS-In | 61.2 | 47.0 | 46.4 | 47.3
BRNN-POS-H1 | 60.1 | 46.5 | 46.8 | 47.5
BRNN-POS-H2 | 63.2 | 48.8 | 47.7 | 50
BRNN-POS-H2 - OOV | 64.6 | 49.5 | 48.4 | 50.7
Combination | Projection + SRNN | 62.0 | 46.7 | 46.5 | 47.4
Projection + BRNN | 62.2 | 46.8 | 46.4 | 47.5
Projection + SRNN-POS-In | 62.9 | 47.4 | 46.9 | 47.7
Projection + SRNN-POS-H1 | 62.5 | 47.0 | 47.1 | 48.0
Projection + SRNN-POS-H2 | 63.5 | 49.2 | 48.0 | 50.1
Projection + BRNN-POS-In | 62.9 | 47.5 | 46.9 | 47.8
Projection + BRNN-POS-H1 | 62.7 | 47.0 | 47.0 | 48.0
Projection + BRNN-POS-H2 | 63.6 | 49.3 | 48.0 | 50.3
Projection + BRNN-POS-H2 - OOV | 64.7 | 49.8 | 48.6 | 51.0
S-E | MFS Semeval 2013 | 60.7 | 52.4
GETALP [Schwab et al., 2012] | 40.2 | 34.6
Table 2: Super Sense Tagging (SST) accuracy for Simple Projection, RNN and
their combination.

##  5 Conclusion

In this paper, we have presented an approach based on recurrent neural
networks (RNN) to induce multilingual text analysis tools. We have studied
Simple and Bidirectional RNN architectures on multilingual POS and SST
tagging. We have also proposed new RNN variants in order to take into account
low level (POS) information in a super sense tagging task. Our approach has
the following advantages: (a) it uses a language-independent word
representation (based only on word co-occurrences in a parallel corpus), (b)
it provides truly multilingual taggers (1 tagger for N languages) (c) it can
be easily adapted to a new target language (when a small amount of supervised
data is available, a previous study [Zennaki et al., 2015a, Zennaki et al.,
2015b] has shown the effectiveness of our method in a weakly supervised
context).

Short term perspectives are to apply multi-task learning to build systems that
simultaneously perform syntactic and semantic analysis. Adding out-of-language
data to improve our RNN taggers is also possible (and interesting to
experiment) with our common (multilingual) vector representation.

## References

  * [Al-Rfou et al., 2013]  Rami Al-Rfou, Bryan Perozzi, and Steven Skiena.  2013\.  Polyglot: Distributed word representations for multilingual nlp.  CoNLL-2013, pages 183–192.
  * [Bengio et al., 2003]  Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin.  2003\.  A neural probabilistic language model.  Journal of Machine Learning Research, 3:1137–1155.
  * [Bentivogli et al., 2004]  Luisa Bentivogli, Pamela Forner, and Emanuele Pianta.  2004\.  Evaluating cross-language annotation transfer in the multisemcor corpus.  In COLING, page 364.
  * [Bérard et al., 2016]  Alexandre Bérard, Christophe Servan, Olivier Pietquin, and Laurent Besacier.  2016\.  Multivec: a multilingual and multilevel representation learning toolkit for nlp.  In The 10th edition of the Language Resources and Evaluation Conference (LREC 2016).
  * [Besacier et al., 2012]  Laurent Besacier, Benjamin Lecouteux, Marwen Azouzi, and Ngoc-Quang Luong.  2012\.  The lig english to french machine translation system for iwslt 2012.  In IWSLT, pages 102–108.
  * [Brants, 2000]  Thorsten Brants.  2000\.  Tnt: a statistical part-of-speech tagger.  In Proceedings of the sixth conference on Applied natural language processing, pages 224–231.
  * [Buchholz and Marsi, 2006]  Sabine Buchholz and Erwin Marsi.  2006\.  Conll-x shared task on multilingual dependency parsing.  In Proceedings of the Tenth CoNLL, pages 149–164.
  * [Cho et al., 2014]  Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.  2014\.  On the properties of neural machine translation: Encoder–decoder approaches.  Syntax, Semantics and Structure in Statistical Translation.
  * [Ciaramita and Altun, 2006]  Massimiliano Ciaramita and Yasemin Altun.  2006\.  Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger.  In Proceedings of the EMNLP-2006, pages 594–602.
  * [Collins and Singer, 1999]  Michael Collins and Yoram Singer.  1999\.  Unsupervised models for named entity classification.  In Proceedings of the joint SIGDAT conference on EMNLP and very large corpora, pages 100–110.
  * [Collobert et al., 2011]  Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  2011\.  Natural language processing from scratch.  The Journal of Machine Learning Research, 12:2493–2537.
  * [Das and Petrov, 2011]  Dipanjan Das and Slav Petrov.  2011\.  Unsupervised part-of-speech tagging with bilingual graph-based projections.  Proceedings of the 49th ACL, 1:600–609.
  * [Duong et al., 2013]  Long Duong, Paul Cook, Steven Bird, and Pavel Pecina.  2013\.  Simpler unsupervised pos tagging with bilingual projections.  In ACL (2), pages 634–639.
  * [Durrett et al., 2012]  Greg Durrett, Adam Pauls, and Dan Klein.  2012\.  Syntactic transfer using a bilingual lexicon.  In The Joint Conference on EMNLP and CoNLL, pages 1–11.
  * [Elman, 1990]  Jeffrey L Elman.  1990\.  Finding structure in time.  Cognitive science, 14(2):179–211.
  * [Fellbaum, 1998]  Christiane Fellbaum.  1998\.  WordNet.  Wiley Online Library.
  * [Fraser and Marcu, 2007]  Alexander Fraser and Daniel Marcu.  2007\.  Measuring word alignment quality for statistical machine translation.  Computational Linguistics, 33(3):293–303.
  * [Goldberg, 2010]  Andrew Brian Goldberg.  2010\.  New directions in semi-supervised learning.  Ph.D. thesis, University of Wisconsin–Madison.
  * [Gouws and Søgaard, 2015]  Stephan Gouws and Anders Søgaard.  2015\.  Simple task-specific bilingual word embeddings.  In NAACL-HLT, pages 1386–1390.
  * [Gouws et al., 2015]  Stephan Gouws, Yoshua Bengio, and Greg Corrado.  2015\.  Bilbowa: Fast bilingual distributed representations without word alignments.  ICML 2015.
  * [Graves, 2012]  Alex Graves.  2012\.  Supervised sequence labelling.  Springer.
  * [Gutiérrez Vázquez et al., 2011]  Yoan Gutiérrez Vázquez, Antonio Fernández Orquín, Andrés Montoyo Guijarro, Sonia Vázquez Pérez, et al.  2011\.  Enriching the integration of semantic resources based on wordnet.
  * [Klein, 2005]  Dan Klein.  2005\.  The unsupervised learning of natural language structure.  Ph.D. thesis, Stanford University.
  * [Koehn, 2005]  Philipp Koehn.  2005\.  Europarl: A parallel corpus for statistical machine translation.  In MT summit, volume 5, pages 79–86.
  * [Kucera and Francis, 1979]  H Kucera and W Francis.  1979\.  A standard corpus of present-day edited american english, for use with digital computers (revised and amplified from 1967 version).
  * [Luong et al., 2015]  Thang Luong, Hieu Pham, and Christopher D. Manning.  2015\.  Bilingual word representations with monolingual quality in mind.  In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151–159.
  * [Mikolov et al., 2010]  Tomáš Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur.  2010\.  Recurrent neural network based language model.  In INTERSPEECH 2010, pages 1045–1048.
  * [Mikolov et al., 2013]  Tomáš Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems.
  * [Miller et al., 1993]  George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker.  1993\.  A semantic concordance.  In Proceedings of the workshop on HLT, pages 303–308.
  * [Navigli and Lapata, 2010]  Roberto Navigli and Mirella Lapata.  2010\.  An experimental study of graph connectivity for unsupervised word sense disambiguation.  Pattern Analysis and Machine Intelligence, 32(4):678–692.
  * [Navigli and Ponzetto, 2012]  Roberto Navigli and Simone Paolo Ponzetto.  2012\.  Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network.  Artificial Intelligence, 193:217–250.
  * [Navigli et al., 2013]  Roberto Navigli, David Jurgens, and Daniele Vannella.  2013\.  Semeval-2013 : Multilingual word sense disambiguation.  In Second Joint Conference on Lexical and Computational Semantics, volume 2, pages 222–231.
  * [Och and Ney, 2000]  Franz Josef Och and Hermann Ney.  2000\.  Improved statistical alignment models.  In Proceedings of the 38th Annual Meeting on ACL, pages 440–447.
  * [Padó and Lapata, 2009]  Sebastian Padó and Mirella Lapata.  2009\.  Cross-lingual annotation projection for semantic roles.  Journal of Artificial Intelligence Research, 36(1):307–340.
  * [Petrov et al., 2012]  Slav Petrov, Dipanjan Das, and Ryan McDonald.  2012\.  A universal part-of-speech tagset.  In LREC’12.
  * [Rumelhart et al., 1985]  David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  1985\.  Learning internal representations by error propagation.  Technical report, DTIC Document.
  * [Schmid, 1995]  Helmut Schmid.  1995\.  Treetagger| a language independent part-of-speech tagger.  Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart, 43:28.
  * [Schmidhuber, 1992]  Jürgen Schmidhuber.  1992\.  A fixed size storage o (n3) time complexity learning algorithm for fully recurrent continually running networks.  Neural Computation, 4(2):243–248.
  * [Schuster and Paliwal, 1997]  Mike Schuster and Kuldip K Paliwal.  1997\.  Bidirectional recurrent neural networks.  Signal Processing, 45(11):2673–2681.
  * [Schwab et al., 2012]  Didier Schwab, Jérôme Goulian, Andon Tchechmedjiev, and Hervé Blanchon.  2012\.  Ant colony algorithm for the unsupervised word sense disambiguation of texts: Comparison and evaluation.  In COLING, pages 2389–2404.
  * [Sundermeyer et al., 2013]  Martin Sundermeyer, Ilya Oparin, J-L Gauvain, Ben Freiberg, Ralf Schluter, and Hermann Ney.  2013\.  Comparison of feedforward and recurrent neural network language models.  In ICASSP, pages 8430–8434. IEEE.
  * [Sutskever et al., 2014]  Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  2014\.  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems, pages 3104–3112.
  * [Täckström et al., 2012]  Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit.  2012\.  Cross-lingual word clusters for direct transfer of linguistic structure.  In Proceedings of the 2012 conference of the NAACL-HLT, pages 477–487.
  * [Täckström et al., 2013]  Oscar Täckström, Ryan McDonald, and Joakim Nivre.  2013\.  Target language adaptation of discriminative transfer parsers.
  * [Titov and Klementiev, 2012]  Ivan Titov and Alexandre Klementiev.  2012\.  Crosslingual induction of semantic roles.  In Proceedings of the 50th Annual Meeting of the ACL, volume 1, pages 647–656.
  * [Veronis et al., 2008]  J Veronis, O Hamon, C Ayache, R Belmouhoub, O Kraif, D Laurent, TMH Nguyen, N Semmar, F Stuck, and W Zaghouani.  2008\.  Arcade ii action de recherche concertée sur l’alignement de documents et son évaluation.
  * [Yarowsky et al., 2001]  David Yarowsky, Grace Ngai, and Richard Wicentowski.  2001\.  Inducing multilingual text analysis tools via robust projection across aligned corpora.  In Proceedings of the first international conference on Human language technology research, pages 1–8.
  * [Zennaki et al., 2015a]  Othman Zennaki, Nasredine Semmar, and Laurent Besacier.  2015a.  Unsupervised and lightly supervised part-of-speech tagging using recurrent neural networks.  In PACLIC 29.
  * [Zennaki et al., 2015b]  Othman Zennaki, Nasredine Semmar, and Laurent Besacier.  2015b.  Utilisation des réseaux de neurones récurrents pour la projection interlingue d’étiquettes morpho-syntaxiques à partir d’un corpus parallèle.  In TALN 2015.