# Classification Benchmarks for Under-resourced
Bengali Language based on Multichannel
Convolutional-LSTM Network

Md. Rezaul Karim  Fraunhofer FIT, Aachen, GermanyRWTH Aachen University,
Germany ,  Bharathi Raja Chakravarthi  Insight SFI Research Centre for Data
Analytics,
Data Science Institute, National University of Ireland Galway, Ireland ,  John
P. McCrae  Insight SFI Research Centre for Data Analytics,
Data Science Institute, National University of Ireland Galway, Ireland and
Michael Cochez  Department of Computer Science,
Vrije Universiteit Amsterdam, Netherlands

###### Abstract.

Exponential growths of social media and micro-blogging sites not only provide
platforms for empowering freedom of expressions and individual voices but also
enables people to express anti-social behaviour like online harassment,
cyberbullying, and hate speech. Numerous works have been proposed to utilize
these data for social and anti-social behaviours analysis, document
characterization, and sentiment analysis by predicting the contexts mostly for
highly resourced languages such as English. However, there are languages that
are under-resources, e.g., South Asian languages like Bengali, Tamil,
Assamese, Telugu that lack of computational resources for the natural language
processing (NLP)111List of abbreviations can be found at the end of this
paper. tasks. In this paper222This paper is under review in the Journal of
Natural Language Engineering., we provide several classification benchmarks
for Bengali, an under-resourced language. We prepared three datasets of
expressing hate, commonly used topics, and opinions for hate speech detection,
document classification, and sentiment analysis, respectively. We built the
largest Bengali word embedding models to date based on 250 million articles,
which we call BengFastText. We perform three different experiments, covering
document classification, sentiment analysis, and hate speech detection. We
incorporate word embeddings into a Multichannel Convolutional-LSTM (MConv-
LSTM) network for predicting different types of hate speech, document
classification, and sentiment analysis. Experiments demonstrate that
BengFastText can capture the semantics of words from respective contexts
correctly. Evaluations against several baseline embedding models, e.g.,
Word2Vec and GloVe yield up to 92.30%, 82.25%, and 90.45% F1-scores in case of
document classification, sentiment analysis, and hate speech detection,
respectively during 5-fold cross-validation tests.

Under-resource language, NLP, Deep learning, Sentiment analysis, Hate speech
detection, Word embedding, Word2Vec, FastText.

††copyright: none††conference: BLA ; BLA; BLA

##  1\. Introduction

In recent years, micro-blogging platforms and social networking sites have
grown exponentially, enabling their users to voice their opinions (Khan et
al., 2018). At the same time, they have also enabled anti-social behavior
(ElSherief et al., 2018), online harassment, cyberbullying, false political
and religious rumor, and hate speech activities (Ribeiro et al., 2018;
Kshirsagar et al., 2018). The anonymity and mobility offered by such media
have made the breeding and spread of hate speech (Zhang et al., 2018) –
eventually leading to hate crime in many aspects including religious,
political, geopolitical, personal, and gender abuse. This applies to every
human being regardless of languages, geographic locations or ethnicity. While
the Web began as an overwhelmingly English phenomenon, it now contains extant
text in hundreds of languages. In the real world, languages are becoming
extinct at an alarming rate and the world could lose more than half of its
linguistic diversity by the year 2100. Certain languages are low resource
because they are actually low-density languages based on the number of people
speaking in the world.

Languages are dynamic being spoken by communities whose lives are shaped by
the rapidly changing world. The number of living languages spoken in the world
is about 7,099, which is constantly in flux. About a third of these languages
are now endangered, often with less than 1,000 speakers remaining. Meanwhile,
only 23 languages account for more than half the world’s population. Bengali,
which is a rich language with a lot of diversity is spoken in Bangladesh, the
second most commonly spoken language in India, and the seventh most commonly
spoken language in the world with nearly 230 million speakers (200 million
native speakers) (Islam, 2009).

Sentiment analysis makes use of the polarity expressed in texts, where lexical
resources are often used to look up specific words as their presence can be
predictive features. Linguistic features, on the other hand, utilize syntactic
information like Part of Speech (PoS) and certain dependency relations as
features. Content-based classification, which is the process of grouping
documents into different classes or categories, is emerging due to the
continuous growth of digital data. Apart from these, sentiment analysis in
Bengali is progressively being considered as a non-trivial task, for which
previous approaches have attempted to detect the overall polarity of Bengali
texts. Hate speech, which is defined by the Encyclopedia of American
Constitution (Nockleby, 2000) is “any communication that disparages a person
or a group on the basis of some characteristic such as race, color, ethnicity,
gender, sexual orientation, nationality, religion, or other characteristics”,
is also rampant online, e.g., social media, newspapers, and books (Salminen et
al., 2018a). Like in other major languages like in English, the use of hate
speech in Bengali is pervasive and can have serious consequences, because
according to a Special Rapporteur to the UN Humans Rights Council, failure to
monitor and react to such hate speech in a timely manner can reinforce the
subordination of targeted minorities, making them not only vulnerable to
attack but also influencing majority populations and making them more
indifferent to various manifestations of such hatred (Izs´ak, 2015).

Figure 1. Examples of Bengali hate speech, either directed towards a specific
person or entity, or generalized towards a group

Thus, identifying such hate speech and making awareness in a virtual landscape
beyond the realms of traditional law enforcement is a non-trivial research
contribution. However, a vast amount of content published online does not
allow for manual analysis. For years, tech giants like Twitter, Facebook, and
YouTube have been combating the issue of hate speech and it has been estimated
that hundreds of millions of euros are being invested every year on counter-
measures including manpower (Izs´ak, 2015). However, they are still being
criticized for not doing enough (Olteanu et al., 2018). Efficient and
automated techniques for such useful researches for under-resourced Bengali
language is in scarce, which is largely because such measures require
significant manual efforts. However, reviewing online contents to identify
offensive materials and taking actions need manual intervention, which is not
only labor intensive and time consuming but also not scalable (Izs´ak, 2015).

To the end, computational linguistics has long been dominated by techniques
specifically geared towards English and other European languages in which
English language models are often used as baselines, despite the fact that it
is an outlier in a lot of linguistic aspects. Despite being often unreported
if the results hold up in other languages, the techniques are often considered
state-of-the-art for natural language understanding. The field now, has
unprecedented access to a great number of low-resource languages, readily
available to be studied, but needs to act quickly before political, social,
and economic pressures cause these languages to disappear from the real and/or
virtual worlds. In the absence of these resources, the lack of good
performance across various tasks feeds into a systemic information asymmetry
in the digital world. The quality of translation systems, the lack of social
media corpora, and the general absence of language-agnostic tools is going to
exacerbate the digital divide in coming times at the global level and is a
prominent barrier in the way of truly democratizing technology access in the
world.

Accurate identification of such hate speeches or sentiment analysis requires
not only an automated, scalable, and accurate methods but also computational
resources such as state-of-the-art machine learning (ML) approaches, word
embedding models and rich labelled datasets, which in turn needs natural
language processing (NLP) resources for the language in question is the core
contribution of this work. This is obvious, because the NLP resources for
Bengali are still scarce (Besacier et al., 2014; Chakravarthi et al., 2018).
Recent research efforts from both the NLP and ML communities have proven to be
very effective for well-resourced languages like English. Computational
linguistics today is largely dominated by supervised machine learning (ML),
which requires annotating data for low-resource languages that can be time
consuming, expensive and needs experts. They identified these research
problems as supervised learning tasks, which can be divided into two
categories. First, there are classic methods, which rely on manual feature
engineering such as Support Vector Machines (SVM), Naïve Bayes (NB), KNN,
Logistic Regression (LR), Decision Trees (DT), Random Forest (RF) and Gradient
Boosted Trees(GBT). Second, there are approaches which employs deep neural
networks (DNN) to learn multilayers of abstract features from raw texts. These
approaches are largely based on Convolutional (CNN) or Long Short-term Memory
(LSTM) networks.

These approaches are mostly deep learning (DL) based, referring to the depth
of the neural networks used. Nevertheless, despite the growth of studies and
existing ML approaches, under-resourced languages such as Bengali lacks
resources such as rich word embedding models and comparative evaluation on NLP
tasks. In this work, we built a Bengali word embedding model called
BengFastText and implemented a robust neural network architecture called
multichannel convolutional-LSTM network (MConv-LSTM) by combining both CNN and
LSTM networks with multiple channels in a single architecture to leverage
their benefits. The other key contributions of this paper can be summarized as
follows:

  * •

We built the largest Bengali word embedding model called BengFastText based on
250 million Bengali articles in an under-resourced setting. BengFastText is a
language model based on fastText (Grave et al., 2018), which is
computationally-efficient predictive model for learning word embeddings from
raw Bengali texts.

  * •

We prepared three public datasets of hate speech, sentiment analysis, and
document classification of Bengali texts, which are larger than any currently
available datasets by both quantity and subject coverage.

  * •

We trained several ML baseline models like SVM, KNN, LR, NB, DT, RF and GBT.
Further, we prepared embeddings based on Word2Vec and GloVe for a comparative
analysis between ML and MConv-LSTM model.

  * •

Our approach shows how the automatic feature selection using MConv-LSTM model
significantly improves the overhead of manual feature engineering.

The rest of the paper is structured as follows: Section 2 reviews related work
on hate speech and Bengali word embedding. Section 3 describes the data
collection and annotation process. Section 4 describes the process of Bengali
neural embedding, network construction, and training. Section 5 illustrates
experiment results including a comparative analysis with baseline models on
all datasets. Section 6 summarizes this research with potential limitations
and points some possible outlook before concluding the paper.

##  2\. Related Work

Although linguistic analyses are carried out extensively and well studied on
other major languages, only a few approaches have been explored for Bengali
(Islam, 2009), which is due to lack of systematic method of text collection,
annotated corpora, name dictionaries, morphological analyzer, and overall
research outlook. Existing work on Bengali NLP mostly focus on either document
categorization (Islam et al., 2017) using supervised learning techniques or
applying N-gram technique to categorize newspaper corpora (Mansur, 2006),
dataset preparation (Rahman and Kumar Dey, 2018) for aspect-based sentiment
analysis (Sumit et al., 2018), and word embeddings for document classification
(Ahmad and Amin, 2016).

Apart from these, some approaches focus on parts of speech (PoS) tagging
techniques for parsing texts for rich semantic processing by combining Maximum
Entropy, Conditional Random Field (CRF) and SVM classifiers using weighted
voting techniques (Ekbal et al., 2009) or based on bidirectional LSTM—CRF
based network (Alam, 2016). In these approaches, classic ML algorithms such as
SVM, KNN, NB, RF, and GBT have been used. These approaches, either lack
comprehensive linguist analysis or are carried out on a small-scale labeled
dataset with limited vocabulary, which diminishes the performance of
supervised learning algorithms. IN contrast, a growing body of work aims to
understand and detect hate speech by creating representations for content on
the web and then classifying these tweets or comments as hateful or not,
drawing insights along the way (ElSherief et al., 2018). Many efforts have
been made to classify hate speech using data scraped from online message
forums and popular social media including Twitter, YouTube, Facebook, and
Google, with their Perspective API (Salminen et al., 2018b).

A LR-based approach is proposed for one-to-four-character n-grams for
classifying tweets that are labeled as racist, sexist or neither (Waseem and
Hovy, 2016) and for distinguishing hateful and offensive but not hateful
tweets with L2 regularization (Alorainy et al., 2018). For the latter use
case, the authors applied word-level n-grams and various PoS, sentiment and
tweet-level metadata features. However, the accuracy of these solutions are
not satisfactory (Islam et al., 2017). Davidson et al. (Davidson et al., 2017)
used DNN approaches with two binary classifiers: one to predict the presence
of abusive speech more generally, and another one to discern the form of
abusive speech or building a classifier composed of two separate networks for
hate speech detection (Kshirsagar et al., 2018).

Existing classifiers are predominantly supervised (Wei et al., 2017): LR is
the most popular, while other algorithms, e.g. SVM, NB, and RF are also used.
Despite diverse types of features introduced, little is known about the
multimodality of different types of features in such a single classifier. Most
methods simply ‘use them all’ by concatenating all feature types into high-
dimensional, sparse feature vectors that are prone to overfitting, especially
on short texts e.g. tweets. Some applied an automated statistical feature
selection process to reduce and optimize the feature space, while others did
this manually.

Since the impact of feature selection is unknown, whether different types of
features are contributing to better classification accuracy remains
questionable. Hence, classic methods that have been investigated in the
literature employ a abundance of task-specific features. It is not clear what
their individual contributions could be (Fortuna and Nunes, 2018). These
approaches in comparison with DL-based approaches is rather incomparable
because of the efficiency of linear models at dealing with billions of such
texts proven less accurate and unscalable, which is probably another primary
reason. DL is applied to learn abstract feature representations used for the
text classification. The input can take various forms of feature encoding,
including any of those used in the classic methods. However, the key
difference is that the input features are not directly used for
classification. Instead, multilayer structures learn abstract representation
of the features, which is found more effective for learning, as typical manual
feature engineering is then shifted to the network topology, which is
carefully designed to automatically extract useful features from a simple
input feature representation (Fortuna and Nunes, 2018).

CNN, Gated Recurrent Unit (GRU) (Zhang et al., 2018), and LSTM networks are
the most popular architectures in the literature. In particular, CNN is an
effective arsenal to act as ‘feature extractors’, whereas LSTM is a type of
powerful recurrent type network for modeling orderly sequence learning
problems. We also observe the use of LSTM or GRU with pre-trained Word2Vec,
GloVe, and fastText embeddings to fed into a CNN with max pooling to produce
input vectors for a neural network (Chen and Yuan, 2018), because, in the
context of text classification, CNN extracts word or character combinations,
e.g., phrases, n-grams and LSTM learns a long-range word or character
dependencies in texts. In theory, the Conv-LSTM is a powerful architecture to
capture long-term dependencies between features extracted by CNN. In practice,
they are found to be more effective than structures solely based on CNN or
LSTM in tasks such as drug-drug interaction predictions (Karim et al., 2019),
network intrusion detection (Khan et al., 2019), and gesture/activity
recognition where the networks learn temporal evolution of different regions
between frames, and Named Entity Recognition (NER) (Xingjian et al., 2015)
where the class of a word sequence can depend on the class of its preceding
word sequence.

While each type of network has shown effectiveness for general purpose text
classification, few works have explored combining both structures into a
single network (Salminen et al., 2018a), except that using this combination,
and especially when trained using transfer learning achieved higher
classification accuracy than either neural network classifier alone (Izs´ak,
2015). In contrast, our approach relies on a word embedding similar to
fastText but with a strong focus on the under-resourced setting by reducing
the number of required parameters and length of training required, while still
yielding improved performance and resilience across related classification
tasks. We hypothesize that MConv-LSTM can also be effective not only for hate
speech detection but also for sentiment analysis and document classification
since this deep architecture can capture co-occurring word n-grams as useful
patterns for classification (Joulin et al., 2016b; Grave et al., 2018).
Moreover, we believe that our network will be able to learn flexible vector
representations that demonstrate associations between words typically used not
only for hate speech detection but also for document classification and
sentiment analysis.

##  3\. Datasets

One major limitation of state-of-the-art approaches is the lack of comparative
evaluation on publicly available data sets for the Bengali language. A large
majority of the existing works are evaluated on privately collected datasets,
often targeting different problems, and most of them contain only a small
Bengali text corpus. A careful analysis was carried out while differentiating
linguistic features, e.g. abusive language can be different from hate speech
but is often used to express hate as shown in examples in fig. 1. To best of
our knowledge, there are only a few works proposed on Bengali word embeddings.
The Polyglot project (Alrfou and Skiena, 2013) is one of the first attempts,
which in which the word embeddings are trained for more than 100 languages
using their corresponding Wikipedia dumps in which only 55K corpus from the
Bengali Wikipedia dump were used to train the model to generate the
embeddings. The second approach (Ahmad and Amin, 2016) built an embedding
model on 200K Bengali corpus from 13 news articles. Third, a neural lemmatizer
is proposed by (Chakrabarty and Garain, 2016) using Bengali word embeddings,
which is built on even a smaller dataset of 20K corpus.

The fourth one called fastText (Grave et al., 2018), which is trained on
common crawl and Wikipedia using CBOW with position-weights, in dimension 300,
with character n-grams of length 5, a window of size 5 and 10 negatives.
Eventually, fastText distributes the pre-trained word vectors for 157
languages. To further explore and apply more extensive linguist analysis on
Bengali with flexibly and efficiently, we not only collected the largest
corpus of Bengali raw texts from Wikipedia but also prepared three more
datasets: documents of contemporary Bengali topics, texts expressing hate
speech (political, religious, personal, geopolitical and gender abusive) and
public sentiments (about Cricket, products, movie, and politics).

###  3.1. Raw text corpus collection

For our corpus, Bengali articles were collected from numerous sources from
Bangladesh and India including a Bengali Wikipedia dump, Bengali news articles
(Daily Prothom Alo, Daily Jugontor, Daily Nayadiganta, Anandabazar Patrika,
Dainik Jugasankha, BBC, and Deutsche Welle), news dumps of TV channels (NTV,
ETV Bangla, ZEE News), books, blogs, sports portal, and social media (Twitter,
Facebook pages and groups, LinkedIn). We also categorized the raw articles for
ease preprocessing for later stage. Facebook pages (e.g. celebrities,
athletes, sports, and politicians) and newspaper sources were scrutinized
because composedly they have about 50 million followers and many opinions,
hate speech, and review texts come or spread out from there. All text samples
were collected from publicly available sources. Altogether, our raw text
corpus consists of 250 million articles.Then two linguists and three native
Bengali speakers annotated the samples from separate datasets for hate speech
detection, document classification, and sentiment analysis, which is described
in subsequent subsections333The prepared dataset, embedding model, and codes
will be made publicly available. to encourage future comparative evaluation.

(a) Hashtags normalization

(b) Stemming

Figure 2. Example of hashtags normalization and stemming

Figure 3 shows the frequency word clouds of potential creative hates expressed
in our datasets. Since the collected Bengali text corpus comes with various
dimensional, sparse and relatively noisy, before the word embedding step and
manual labeling, emoticon, digits (Bengali and English) special characters,
hash-tags and excess use of spaces (due to redundant use of spaces, tabs,
shift) were removed to give annotators an unbiased-text-only content to make a
decision based on criteria of the objective.

###  3.2. Data preprocessing

Since the obtained corpus is rather noisy, we perform an automated cleaning
for all datasets before giving it to the annotators. First, we remove HTML
markup, links, and image titles. From text, digits, special characters, hash-
tags and excessive white space are removed. Then the following further steps
are performed:

  * •

PoS tagging: an BLSTM-CRF based approach suggested in literature (Alam, 2016)
was applied.

  * •

Removal of proper nouns: various proper nouns and noun suffixes were
identified and replaced with tags to provide ambiguity.

  * •

Hashtags normalization: hastags in tweets, comments, and posts were normalized
using a dictionary-based look up444Based on
https://www.101languages.net/bengali/bengali-word-list/, which are often used
to compose words (e.g. fig. 2(a)) was used to split such hashtags.

  * •

Stemming: inflected words are reduced to their stem, base or root form in
order to reduce dimension and obtain more standard Bengali tokens (e.g. fig.
2(b))(English translation: Bangladeshe to Bangladesh, Pakistaner to Pakistan,
from village to village).

  * •

Stop word removal: commonly used Bengali stop words (provided by Stopwords
ISO555https://github.com/stopwords-iso/stopwords-bn) are removed.

  * •

Removing infrequent words: finally, we remove any tokens with a document
frequency less than 5.

###  3.3. Dataset for document classification

Despite several comprehensive textual datasets available for document
categorization, only a few labeled datasets exist for Bengali articles.
Sophisticated supervised learning approaches, therefore, were not a viable
option to address document classification problems. Eventhough, large-scale
corpora for other language are available online, it is rare for under-
resourced languages like Bengali. Hence we decided to build our own corpus. As
described above, the articles are collected into different categories such
that each belongs to any one of the following categories: state, economy,
entertainment, international, and sports as outlined in table 1.

Table 1. Statistics of document classification dataset. For each category, we show the number of documents, number of words, average number of sentences per document, and average number of words per sentence Category | #Doc | #Words | ASD | AWS
---|---|---|---|---
State | 242,860 | 57,019,465 | 18.50 | 13.356
Economy | 18,982 | 4,915,141 | 20.18 | 13.378
International | 32,203 | 7,096,111 | 18.47 | 12.493
Entertainment | 31,293 | 6,706,563 | 21.70 | 10.236
Sports | 50,888 | 12,397,415 | 22.80 | 11.069
Total | 376,226 | 88,134,695 | 20.33 | 12.10

###  3.4. Hate speech dataset preparation

To prepare the data we used a bootstrapping approach, which starts with an
initial search for specific types of texts, articles or tweets containing
common slurs and terms used pertaining to targeting characteristics. We
manually identify frequently occurring terms in the texts containing hate
speech and references to specific entities. From the raw texts, we further
annotated 100,000 statements, texts or articles, which directly or indirectly
express ‘hate speech’.

Table 2. Statistics of the hate speech detection dataset Type of hate | #Statement | #Words | AWS
---|---|---|---
Political | 7,182 | 132,867 | 18.50
Religious | 6,975 | 140,756 | 20.18
Gender abusive | 7,300 | 134,831 | 18.47
Geopolitical | 6,793 | 117,180 | 17.25
Personal | 6,750 | 146,475 | 21.70
Total | 35,000 | 672,109 | 19.22

However, we encountered numerous challenges e.g. since there exists different
types of hate in the regions, distinguishing hate speech from non-hate
offensive language is a challenging task, which is because sometimes they
overlap but are not the same (Zhang and Luo, 2018). Collected data samples are
annotated in a semi-automated way (after removing mostly similar statements).
First, we attempted to annotate them in an automated way, which is inspired by
the comparing communities technique using creative political slang (CSPlang)
(Hossain and Kautz, 2018). We prepare normalized frequency vectors of 175
abusive terms666https://github.com/rezacsedu/BengFastText across the raw
texts, which are commonly used to express hates in Bengali language. Then we
assign label ‘hate’ if at least one of these terms exists in the text,
otherwise, we put label ‘neutral’. This way, however, differentiating
’neutral’ and ‘non-hate’ was not possible.

(a) Frequently used terms in hate speech

(b) Frequently used terms in sentiments

(c) Frequently used topics of discussion

Figure 3. Frequency word clouds on potential hates, sentiments, and common
topics

The annotations are further validated and corrected by three experts (one
South Asian linguist and two native speakers) into one of three categories:
hate, non-hate and neutral. To reduce possible bias, each label was assigned
based on a majority voting on the annotator’s independent opinions.
Fortunately, certain types of hate were easy to identify and annotate based on
CSPlang that are nonstandard word for Bengali, which conveys a positive or
negative attitude towards a person, a group of people, or an issue that is the
subject of discussion in political discourse (Hossain and Kautz, 2018) and can
be easily annotated as a political hates. Finally, non-hate and neutral
statements were removed from the list and hates were further categorized into
political, personal, gender abusive, geopolitical, and religious hates. We
found that from the annotated text 3.5% was classified as hate speech, which
resulted in 35,000 statements labeled as hate. Figure 3(a) shows the most
frequently used terms expressing hates; whereas, some statistics can be found
in table 2.

###  3.5. Dataset for sentiment analysis

Our empirical study found that opinions, reviews, comment or reactions about
news or someone’s status updates are usually provided in Bengali. However,
research has observed that about 20% of express sentiments in English (Rahman
and Kumar Dey, 2018). Also, some comments were only emoticons or written in
non-native scripts. As English and emoticons are already investigated
elsewhere, we consider this outside of the scope of the current work.

Table 3. Statistics of the dataset for sentiment analysis Polarity | #Statement | #Words | AWS
---|---|---|---
Negative | 153,000 | 727,3200 | 40.16
Positive | 167,000 | 557,3550 | 30.65
Total | 320,000 | 12,846750 | 35.40

The annotation was done manually with the help of native Bengali speakers and
majority voting (with ambiguous polarity removed), which left 320,000 reviews
categorized either positive or negative. Figure 3(a) shows the most frequently
used terms expressing hate; whereas, related statistics can be found in table
3.

Finally, we follow a similar preprocessing steps for extracting most important
words, which is a statement-based observation that states that in a collection
of data, the frequency of a given word should be inversely proportional to its
rank in the corpus –that is most frequent word scores rank 1 in a dataset
should occur approximately twice as the second most frequent word and three
times as the third most frequent, and so on.

##  4\. Methods

To validate the usefulness of the dataset and the BengFastText model, we
perform several ML tasks, which are discussed including word embeddings,
network construction, training, and hyper-parameter optimization.

###  4.1. Neural word embeddings

Given a document, e.g., an article, a tweet, or a review of Bengali text, we
apply preprocessing to normalize the text (see section 3.2). The preprocess
reduces vocabulary size in the dataset due to the colloquial nature of the
texts and to some degree, addresses the sparsity in word-based feature
representations. We also tested by keeping word inflections, using
lemmatization instead of stemming, and lower document frequencies. Empirically
we found that using lemmatization to contribute to slightly better accuracy,
hence reported here subsequently. Each token from the preprocessed input is
embedded into a 300-dimensional real-valued vector, where each element is the
weight for the dimension for the token. For later classification tasks, we
constrain each sequence to 100 words, truncating long texts and pad shorter
ones with zero values.

However, one issue with this design specific choice forces us to consider 300
words per document for document classification otherwise for the majority of
the documents inputs to the convolutional layers will be padded with many
blank vectors, which causes the network to perform poorly. First, the Word2Vec
model is built based on skip-gram method since it is computationally faster
and efficient than CBOW for large-scale text corpus (Bian et al., 2014) and
used for Bengali sentiment analysis (Sumit et al., 2018). From a given a
sequence of words $(w_{1},w_{2},...,w_{n})\in\mathcal{C}$, the neural network
model aims to maximize the average log probability $L_{p}$ (see eq. 1)
according to the context within the fixed-size window, in which c represents a
context (Mikolov et al., 2013a).

(1) $L_{p}=\frac{1}{N}\sum_{n=1}^{N}\sum_{-c\leq j\leq c,j\neq 0}\log
p\left(w_{n+j}|w_{n}\right)$ (2)
$\log\sigma\left(v_{w_{O}}^{\prime\top}v_{w_{I}}\right)+\sum_{i=1}^{k}\mathbb{E}_{w_{i}}\sim_{P_{n}(w)}\left[\log\sigma\left(-v_{w_{i}}^{\prime\top}v_{w_{I}}\right)\right]$

To define $p\left(w_{n+j}|w_{n}\right)$, we use negative sampling by replacing
$\log p\left(w_{O}|w_{I}\right)$ with a function to discriminate target words
$(w_{o})$ from a noise distribution $P_{n}(w)$ drawing $k$ words from
$P_{n}(w)$ in eq. 2. Eventually, the embedding of a word $s$ occurring in
corpus $\mathcal{C}$ is the vector $v_{s}$ in eq. 2 derived by maximizing eq.
1.

Figure 4. t-SNE representation of the embeddings, where closer words are
semantically and statistically more related

Then we built BengFastText model based on GloVe for creating the embeddings
(Pennington et al., 2014). In which, instead of extracting the embeddings from
a neural network that is designed to perform a different task like predicting
neighboring words (i.e., CBOW) or predicting the focus word (i.e., skip-gram),
the embeddings are optimized directly, so that the dot product of two word
vectors equals the log of the number of times the two words will occur near
each other.

Finally, we employed fastText for creating the embeddings (Joulin et al.,
2016b) by the BengFastText model. Instead of learning vectors for words
directly, fastText represents each word as an n-gram of characters, which
helps capture the meaning of shorter words and allows the embeddings to
understand suffixes and prefixes. Once the word is represented using character
n-grams, a skip-gram model with negative sampling based on (Mikolov et al.,
2013b) is used to score similarity between a fixed target word $w_{t}$ and a
word within the context $w_{c}$ to learn the embeddings. This approach
intuitively sounds similar to Word2Vec and can be considered to be a CBOW
model with a sliding window over a word because no internal structure of the
word is taken into account. As long as the characters are within this window,
the order of the n-grams does not matter. The difference, however, is that the
similarity function is not a direct dot product between the word vectors,
rather it is a dot product between two words represented by sums of n-gram
vectors. Then a latent text embedding is derived as an average of the word
embeddings, and the text embedding is used to predict the label. The overall
objective function is to minimize the following loss function (Joulin et al.,
2016b; Grave et al., 2018):

(3) $-\frac{1}{N}\sum_{n=1}^{N}y_{n}\log\left(f\left(BAx_{n}\right)\right)$

where $x_{n}$ represents an n-gram feature, A represents the lookup table to
an average text embedding, and B converts the embedding to pre-softmax values
for each class, while a hierarchical softmax (Goodman, 2001) is applied for
the text classification (Joulin et al., 2016a) given the large number of
classes to minimize computational complexity.

###  4.2. Network construction

In CNN, convolutional filters are used to capture the local dependencies
between neighbor words. However, a fixed filter lengths makes CNN model hard
to learn overall dependencies of a whole sentence. Following steps are
employed to overcome this limitation:

  * •

We consider 3-channel CNN approach by setting a fixed filter size with
different sized kernels. This allows a text to be processed at different
n-grams at a time. The concatenated vector helps the model to learn how to
best integrate these interpretations altogether, which can be considered as a
representation vector constructed from every channel concatenating local
relationship values.

  * •

Then employ an LSTM layer, which is because an LSTM can preserve long-term
dependency among features. Hence, a feature vector constructed by LSTM can
carry overall dependencies of a whole sentence more efficiently (Karim et al.,
2019).

  * •

We concatenate both multichannel-CNN and LSTM layers to get a shared feature
representation: CNN layers capture local relationship, LSTM layers carry
overall relationship).

We expect the shared representation layer containing two vectors to model the
non-linearity more efficiently for our tasks, e.g., hate speech detection,
document classification, or sentiment analysis. Given a text expressing hates
(hopefully), the MConv-LSTM model either reuse the weight matrix from the pre-
trained BengFastText model or generates a word embedding representation by an
embedding layer. This means that we utilize two types of word embeddings to
train the models: i) we initialize the weights in the embedding layer randomly
and let our model learn the embeddings on the fly, ii) we use pre-trained word
embeddings from the GloVe, Word2Vec, and BengFastText models to set the
weights of our embedding layer as a custom embedding layer.

Figure 5. A schematic representation of the MConv-LSTM network, which starts
from taking an input into a n-dimensional embedding hyperspace and passing to
both CNN and LSTM layers before getting a concatenated vector representation
to fed through dense, dropout, and Softmax layers for predicting the hate
class

Using either option, we construct and train the MConv-LSTM network
architecture shown in fig. 5 in which the rectified linear unit (ReLU) is used
as the activation function for the hidden layers and Softmax in the output
layer. Once the word embedding layer maps each text as a ‘sequence’ into a
real vector domain, the embedding representation is fed into both LSTM and
multi-CNN layers towards getting a shared feature representation. To capture
both the local and global relationships, we extended the Conv-LSTM network
proposed in literature (Xingjian et al., 2015) in which each input
$\mathcal{X}_{1},\mathcal{X}_{2},...,\mathcal{X}_{t}$, cell outputs
$\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{t}$, hidden states
$\mathcal{H}_{1},\mathcal{H}_{2}....,\mathcal{H}_{t}$, and gates
$i_{t}$,$f_{t}$,$o_{t}$ of the network (Zhang and Luo, 2018). Conv-LSTM
determines the future state of a certain cell in the input hyperspace by the
inputs and past states of its local neighbors, which is achieved by using a
conv operator in the state-to-state and input-to-state transitions as
(Xingjian et al., 2015):

(4) $\displaystyle i_{t}$
$\displaystyle=\sigma\left(W_{xi}*\mathcal{X}_{t}+W_{hi}*\mathcal{H}_{t-1}+W_{ci}\circ\mathcal{C}_{t-1}+b_{i}\right)$
(5) $\displaystyle f_{t}$
$\displaystyle=\sigma\left(W_{xf}*\mathcal{X}_{t}+W_{hf}*\mathcal{H}_{t-1}+W_{cf}\circ\mathcal{C}_{t-1}+b_{f}\right)$
(6) $\displaystyle\mathcal{C}_{t}$
$\displaystyle=f_{t}\circ\mathcal{C}_{t-1}+i_{t}\circ\tanh\left(W_{xc}*\mathcal{X}_{t}+W_{hc}*\mathcal{H}_{t-1}+b_{c}\right)$
(7) $\displaystyle o_{t}$
$\displaystyle=\sigma\left(W_{xo}*\mathcal{X}_{t}+W_{ho}*\mathcal{H}_{t-1}+W_{co}\circ\mathcal{C}_{t}+b_{o}\right)$
(8) $\displaystyle\mathcal{H}_{t}$
$\displaystyle=o_{t}\circ\tanh\left(\mathcal{C}_{t}\right)$

where ‘*’ denotes the conv operator and ‘o’ is the entrywise multiplication of
two matrices of the same dimension. The second LSTM layer emits an output ‘H’,
which is then reshaped into a feature sequence and feed into fully connected
layers to make the prediction at the next step and as an input at the next
time step. Intuitively, an LSTM layer treats an input feature space of
100$\times$300 and it’s embedded feature vector’s dimension as timesteps and
outputs 100 hidden units per timestep. Once the embedding layer passes an
input feature space 100$\times$300 into three channels (i.e. a multi-1D
convolutional layer) with 100 filters each but with different kernel-size of
4, 6 and 8, respectively (i.e. 4-grams, 6-grams, and 8-grams of hate speech
text).

We pad the input such that the output has the same length as the original
input. Then the output of each convolutional layer is pass to three separate
dropout layers to regularize learning to avoid overfitting (Srivastava et al.,
2014). Intuitively, this can be thought of as randomly removing a word in
sentences and forcing the classification not to rely on any individual words.
This involves the input feature space into a 100$\times$100 representation,
which is then further down-sampled by three different 1D max pooling layers,
each having a pool size of 4 along the word dimension, each producing an
output of shape 25$\times$100\. Where each of the 25 dimensions can be
considered as an‘extracted feature’. Each max pooling layer follows
to‘flatten’ the output space by taking the highest value in each timestep
dimension. This produces a 1$\times$100 vector, which forces words that are
highly indicative of interest. Then these vectors as two information channels
(from CNN + LSTM) are then concatenated and fed into the neural network after
passing through another dropout layer. Finally, the concatenated
representation is passed to a fully connected Softmax to predict a probability
distribution over the classes for the specific tasks.

###  4.3. Network training

The first-order gradient-based AdaGrad optimizer is used to learn model
parameters with varying learning rates and a batch size of 128 to optimize the
cross-entropy loss (categorical cross-entropy in case of document
classification and hate speech detection and binary cross-entropy loss in case
of sentiment analysis) of the predicted label of the text P vs the actual
label of the text T.

(9) $\displaystyle L=\sum_{i,j,k}T_{i,j,k}\log
P_{i,j,k}+\left(1-T_{i,j,k}\right)\log\left(1-P_{i,j,k}\right)$

We perform hyperparameters optimization with random search and 5-fold cross-
validation. Further, we observed the performance by adding Gaussian noise
layers, followed by Conv and LSTM layers to improve model generalization and
reduce overfitting. Out of vocabulary (OOV), particularly in word embedding
models, is a known issue. The OOV cannot be tackled by existing language
models like Polyglot, Word2Vec, and GloVe. This potential issue with the pre-
trained Word2Vec model were addressed based on (Wei et al., 2017): i) randomly
setting the OOV word vector following a continuous uniform distribution, ii)
randomly selecting an in-vocabulary word from the pretrained model to use its
vector.

For the latter, surrounding-contexts are considered, given that a low-
magnitude vector may not be appropriate at the start of training (Bojanowski
et al., 2017). Both settings are found to be empirically helpful in our
experiments. However, the latter outperformed the former. Other options could
have been employed to get rid of OOV e.g. morphological classes (Müller and
Schütze, 2011) or character-word LSTM language models (Verwimp et al., 2017)
but are particularly applied to non-Bengali languages such as English and
Dutch languages.

##  5\. Experiments

In this section, we discuss the results of the experiments based on and
without the pre-trained BengFastText model. In addition, we provide a
comparative analysis between BengFastText and baseline embedding methods:
Word2Vec and GloVe.

###  5.1. Experiment setup

All the methods were implemented in
Python777https://github.com/rezacsedu/BengFastText and experiments are carried
out on a machine having a core i7 CPU, 32GB of RAM and Ubuntu 16.04 OS. The
software stack consisting of Scikit-learn and Keras with the TensorFlow GPU
backend, while the network training is performed on a Nvidia GTX 1050 GPU with
CUDA and cuDNN enabled to make the overall pipeline faster. Open source
implementations of Word2Vec888Word2Vec:
https://radimrehurek.com/gensim/models/word2vec.html, fastText999fastText:
https://github.com/Kyubyong/wordvectors, and GloVe101010GloVe:
http://nlp.stanford.edu/projects/glove/ are used to create the embeddings. For
each experiment, 80% of the data is used for the training with 5-fold cross-
validation and testing the trained model on 20% held-out data. Results based
on best hyperparameters produced through empirical random search are reported
here (results based on embedding model are marked with + in subsequent
tables).

We trained LR, SVM, KNN, NB, RF, and GBT as baseline models using character
n-grams and word uni-grams with TF-IDF weighting. Additionally, we report the
results based on Word2Vec and GloVe embeddings methods. We report our results
using macro-averaged precision, recall, and F1-score since the dataset is
imbalanced. In case of hate speech detection, standard precision, recall, and
F1-score are used. For the sentiment analysis, Matthias Correlation
Coefficient (MCC) and AUC score are measured the performance of the
classifier. Finally, we perform model averaging ensemble (MAE) of the top-3
models to report the final prediction.

Table 4. Document classification performance based on embedding methods Embedding method | Classifier | Precision | Recall | F1
---|---|---|---|---
BengFastText | LR | 0.754+3.8% | 0.763+3.9% | 0.756+3.85%
NB |  0.734-2.11% |  0.736-2.10% |  0.737-2.0%
SVM | 0.766+3.9% | 0.737+3.75% | 0.739+3.65%
KNN | 0.741+2.85% | 0.745+2.75% | 0.748+2.55%
GBT | 0.825+5.65% | 0.825+5.91% | 0.826+5.85%
RF | 0.813+5.7% | 0.815+6.3% | 0.822+6.24%
MConv-LSTM |  0.871+8.75% |  0.883+8.62% |  0.871+8.75%
| MAE |  0.883+9.45% |  0.886+9.25% |  0.892+9.45%
GloVe | LR | 0.725+1.7% | 0.731+1.5% | 0.723+1.7%
NB |  0.692-3.25% |  0.701-3.1% |  0.693-3.2%
SVM | 0.724+2.2% | 0.724+2.2% | 0.718+2.4%
KNN | 0.715+2.75% | 0.716+2.54% | 0.718+2.45%
GBT | 0.772+2.32% | 0.785+2.45% | 0.779+2.65%
RF | 0.792+4.15% | 0.792+4.65% | 0.795+4.75%
MConv-LSTM |  0.834+6.4% |  0.846+7.12% |  0.846+7.31%
| MAE |  0.859+7.4% |  0.867+7.2% |  0.864+7.5%
Word2Vec | LR | 0.746+2.4% | 0.752+2.6% | 0.749+2.7%
NB |  0.721-2.15% |  0.717-2.1% |  0.719-2.2%
SVM | 0.741+2.7% | 0.732+2.5% | 0.736+2.6%
KNN | 0.732+3.4% | 0.725+3.6% | 0.728+3.5%
GBT | 0.806+4.854% | 0.815+4.9% | 0.810+5.1%
RF | 0.792+5.6% | 0.781+5.4% | 0.786+5.5%
MConv-LSTM |  0.851+7.9% |  0.862+8.0% |  0.856+8.1%
| MAE |  0.865+8.5% |  0.876+9.1% |  0.870+8.8%

###  5.2. Analysis of document classification

As shown in fig. 6, classical models, specially KNN and NB, often confused
economics and international related topics and managed to classify them 45%
and 35% of cases correctly. This is because a document from both these
categories contains common words and topics discussing finance, development,
share market, intentional affairs, and politics. Further, as shown in table 1,
the MConv-LSTM model also confused between similar categories plus the sports.
However, for state-related topics, both classic and MConv-LSTM classifiers
perform quite well.

(a) Without pretrained word embeddings

(b) Using pretrained word embeddings

Figure 6. Confusion matrix of the MConv-LSTM model for document classification

Each model’s performance is then evaluated after initializing network’s
weights using the pre-trained BengFastText model. Using the latter approach,
the accuracy increased by 2.0 to 9.45 points weighted F1 (except NB, which
performed even worse with the word embeddings). As shown in table 4, among
standalone classifier, most significant boost was with MConv-LSTM by 8.75%,
which is about 12% improvement compared to the previous results. Overall, MAE
gives 9.45% boost, which is 1.3% better than the second best MConv-LSTM
showing robustness of our approach.

###  5.3. Analysis of hate speech detection

We evaluated the model with and without initializing network weight using
pretrained BengFastText model. Using the pretrained word embeddings, the
overall accuracy is increased by 2.15 to 7.27 points weighted F1 as shown in
table 5. Nevertheless, each classifier’s performance improved with the
pretrained word embeddings except for the NB (performs the worst with only
69.3% accuracy). Among tree-based classifiers, GBT performs the best giving as
much as 86.2% accuracy, which is the best among classic models as well.
Overall, MAE gives 7.27% boost, which is already 1.10% better than the second
best MConv-LSTM (that performs best as a standalone classifier giving 7.15%
improvement).

Table 5. Performance of hate speech detection based on embedding methods Embedding method | Classifier | Precision | Recall | F1
---|---|---|---|---
BengFastText | LR | 0.723+2.21% | 0.727+2.25% | 0.723+2.15%
NB | 0.691-1.05% | 0.694 -1.12% | 0.693 - 1.15%
SVM | 0.735+2.54% | 0.729+2.90% | 0.728+2.25%
KNN | 0.716+2.61% | 0.711+2.52% | 0.712+2.32%
GBT | 0.842+5.37% | 0.845+5.41% | 0.845+5.79%
RF | 0.861+6.25% | 0.857+4.91% | 0.862+5.32%
MConv-LSTM |  0.881+7.25% |  0.883+7.54% |  0.882+7.15%
| MAE |  0.894+7.45% |  0.896+7.65% |  0.891+7.27%
GloVe | LR | 0.718+1.54% | 0.715+1.65% | 0.716+1.40%
NB | 0.672-2.21% | 0.673 -2.12% | 0.681 - 2.15%
SVM | 0.724+2.92% | 0.726+2.75% | 0.726+2.65%
KNN | 0.715+2.7% | 0.716+2.75% | 0.714+2.53%
GBT | 0.823+4.772% | 0.825+5.11% | 0.824+5.17%
RF | 0.818+5.93% | 0.824+4.72% | 0.821+6.10%
MConv-LSTM |  0.827+6.12% |  0.822+6.12% |  0.824+6.11%
| MAE |  0.831+6.55% |  0.834+6.65% |  0.837+6.9%
Word2Vec | LR | 0.69+1.75% | 0.70+2.6% | 0.71+1.25%
NB | 0.65-2.5% | 0.66 -2.2% | 0.67 - 1.6%
SVM | 0.69+2.1% | 0.70+1.75% | 0.77+2.6%
KNN | 0.70+2.5% | 0.71+2.6% | 0.71+2.5%
GBT | 0.806+4.854% | 0.815+4.9% | 0.810+5.1%
RF | 0.74+5.6% | 0.75+4.7% | 0.74+5.3%
MConv-LSTM |  0.77+5.7% |  0.78+5.5% |  0.78+5.8%
| MAE |  0.79+6.45% |  0.80+6.25% |  0.79+6.7%

(a) Without pretrained word embeddings

(b) With pretrained word embeddings

Figure 7. ROC curves of cross-validated MConv-LSTM model(hate speech
detection)

This analysis can be further validated by calibrating the best performing
Conv-LSTM classifier against different embedding methods for which the output
probability of the classifier can be directly interpreted as a confidence
level in terms of ‘fraction of positives’ as shown in fig. 8. As seen the
Conv-LSTM classifier gave a probability value between 0.82 to 0.93, which
means 93% predictions belong to true positive predictions generated by the
BengFastText.

Figure 8. Calibrating classifiers on word embeddings using BengFastText
showing reliability curves

###  5.4. Performance of sentiment analysis

Table 6 summarizes the results of sentiment analysis, which again suggest that
each model performs better with the pre-trained word embeddings, measured by
the MCC showing strong positive relationships. This result suggests that the
predictions were strongly correlated with the ground truth, yielding a Pearson
product-moment correlation coefficient higher than 0.65 for all the
classifiers. This is because MConv-LSTM model learns better with the pre-
trained word embeddings, which why it gives about 5.49% boost compared to
without pre-trained word embeddings in terms of AUC score. Overall, the MEA
gives 7.85% improvement.

Table 6. Performance of sentiment analysis Embedding method | Classifier | MCC | AUC
---|---|---|---
BengFastText | LR | 0.652+2.9% | 0.76+2.8%
NB | 0.653 + 3.25% | 0.77 + 2.32%
SVM | 0.694+2.25% | 0.79+2.45%
KNN | 0.701+2.12% | 0.80+3.33%
GBT | 0.715+3.18% | 0.85+3.91%
RF | 0.727+3.67% | 0.86+4.26%
MConv-LSTM |  0.746+5.49% |  0.87+7.13%
| MAE |  0.758+7.85% |  0.88+7.42%
GloVe | LR | 0.646+2.9% | 0.75+1.75%
NB | 0.644 + 2.10% | 0.76 + 2.11%
SVM | 0.681+1.85% | 0.78+2.51%
KNN | 0.695+2.22% | 0.79+2.28%
GBT | 0.707+2.63% | 0.84+2.67%
RF | 0.719+2.45% | 0.85+3.17%
MConv-LSTM |  0.736+4.76% |  0.86+5.78%
| MAE |  0.734+6.62% |  0.87+5.29%
Word2Vec | LR | 0.634+2.6% | 0.73+2.7%
NB | 0.645 + 2.25% | 0.71 + 2.40%
SVM | 0.677+1.75% | 0.77+1.5%
KNN | 0.662+3.2% | 0.78+3.3%
GBT | 0.683+3.5% | 0.83+3.8%
RF | 0.692+4.5% | 0.82+4.3%
MConv-LSTM |  0.705+6.4% |  0.83+6.5%
| MAE |  0.716+8.1% |  0.85+7.4%

On the other hand, AUC score of the classifier found to be the highest is also
generated by the MConv-LSTM network, which is at least 2% better than the
second-best score by the RF classifier but the worst performance was recorded
by the LR classifier. The ROC curves generated by the MConv-LSTM model are
shown in fig. 7(a) and fig. 7(b). As seen AUC scores are consistent across
folds and This signifies that the predictions by the MConv-LSTM model are much
better than random guessing.

###  5.5. Effects of feature selection

Based on the results for each classification task, it is obvious that feature
selection can be a very powerful technique to improve the performance of
classic methods especially with NB, SVM, KNN, and tree-based approaches.
Contrarily, although LR is intrinsically simple, has low variance, and less
prone to over-fitting, feature selector based on this may be discriminating
features very aggressively. This forces the model to lose some useful
features, which results in lower performance. While investigating alternative
feature selection algorithms, e.g., recursive feature elimination or genetic
algorithms is beyond the scope of this research.

(a) SVM

(b) GBT

(c) RF

(d) Conv-LSTM

Figure 9. Learning curves with validation and training scores of top
classifiers

Another reason could be the size of the training data, which is comparatively
small, i.e., hate speech dataset. However, tree-based approaches such as RF
and GBT perform quite better compared to NB, SVM, and KNN, especially, NB
which even performs worse with pre-trained word embeddings. One potential
reason behind such worse performance of NB is that it’s conditional
independence assumption,which assumes features are independent of one another
when conditioned upon class labels, was rarely accurate in our case. Compared
to classic models, neural network based MConv-LSTM models seems to be much
better (about 3 to 4%) as it outperforms the best performing classic GBT model
in F1-score, which is probably because neural networks managed to capture
abstract features that couldn’t be modeled by classic models.

###  5.6. Effects of number of samples

To understand the effects of having more training samples, and to understand
if our classifiers suffer more from variance errors or bias errors, we
observed the learning curves of top-3 classifiers (i.e., RF, GBT, and MConv-
LSTM) and SVM (a linear model) for varying numbers of training samples in case
of document classification.

As shown in fig. 9, both the validation and training scores converge to a
value, which is too low with increasing size of the training set. Hence, SVM
didn’t benefit much from more training samples. However, since RF and GBT are
tree-based ensemble methods and MConv-LSTM model learn more complex concepts
producing lower bias, the training scores are much higher than the validation
scores for the maximum number of samples, i.e., adding more training samples
has increased model’s generalization.

###  5.7. Comparison with baseline embeddings

Our empirical study founds that almost every classifier performs poorly across
every supervised learning task that are based on the embeddings generated by
BengFastText method compared to Word2Vec and GloVe, as shown in table 4, table
3, and table 2. In particular, except for the sentiment analysis task in which
classifiers performance were consistent, whereas each classifier’s performance
were clearly boosted based on embeddings generated by the BengFastText model,
whereas. Further, an initial experiment based on Polyglot embedding method
shows that it failed to capture the semantic contexts from texts, which is
probably because Polyglot uses only 64 dimensions for the embeddings.

Although classifiers trained on embeddings from Word2Vec model work well
across tasks, the online scanning approach used by Word2Vec is sub-optimal
since the global statistical information regarding word co-occurrences does
not fully exploited (Pennington et al., 2014). GloVe, on the other hand,
performed better than that of Word2Vec since it produces a vector space with
meaningful substructure and performs consistently well in word analogy tasks
and outperforms related models on similarity tasks and named entity
recognition (Pennington et al., 2014).

Further, Word2Vec and GloVe both failed to provide any vector representation
for words that are not in the model dictionary, i.e., not resilient against
OOV. Hence, the approach used to tackle the OOV in our approach (i.e.,
randomly assigned word out of in-vocabulary words list from the pretrained
model by considering the surrounding-contexts) won’t give accurate embeddings
(Bojanowski et al., 2017). On the other hand, fastText performed much better
compared to other embedding methods since fastText works well with rare words.
Thus, even if a word was not seen during the training, it can be broken down
into n-grams to get it’s corresponding embeddings.

##  6\. Conclusion and outlook

In this paper, we provide three different classification benchmarks for under-
resourced Bengali language, which is based on word embeddings and a deep
neural network architecture called MConv-LSTM. The largest BengFastText word
embedding model, which is built on 250 million Bengali articles seems able to
sufficiently capture the semantics of words. To show the effectiveness of our
approach, we prepared three datasets of hate speech, sentiment analysis, and
commonly discussed topics. Then we experimented on three use cases: hate
speech detection, sentiment analysis, and text classification, using a method
that combines CNN and LSTM networks with dropout and pooling, which is found
to empirically improve classification accuracy across all use cases. We also
conducted comparative evaluations on all of these datasets and show that the
proposed method outperformed classic models across metrics and use cases.

Nevertheless, we provided the results based on other Bengali language
embedding methods like fastText and Polyglot. Our results show that approaches
based on BengFastText embeddings clearly outperformed Word2Vec and GloVe-based
ones. Classic methods that depend on pre-engineered features do not require
sophisticated feature engineering but using automatic feature selection
techniques on generic features e.g. n-grams can, in fact, produce better
results. As research suggests character-based features to be more effective
than word-based ones, taking this into account, we experienced better results
with MConv-LSTM using only word-based features, which is likely due to the
superiority in the network architecture. Results of different experiments
showed that feature selection with drop-out, Gaussian noise layer, and pooling
can significantly enhance the learning capabilities to MConv-LSTM towards
through better training and learning.

Research also found that misspellings of words often appeared in up to 15% of
web search queries (Piktus et al., 2019). Such misspellings words are also
responsible for the OOV (apart from other issues like abbreviations, slang,
etc.) in language models. Likewise, a recent language model called Misspelling
Oblivious Word Embeddings (MOWE) (Piktus et al., 2019) in the context of
Bengali language to generate word embeddings. MOWE, which is a combination of
fastText and a supervised learning technique tries to embed misspelled words
close to their correct context and variants. This makes MOWE resilient to
misspellings.

In the future, we intend to employ MOWE-based approach to generate the
embeddings and explore other possibilities such as i) building and training a
likely more robust neural network by stacking multiple conv and bidirectional-
LSTM layers, which are good for extracting hierarchical features, ii) using
Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al.,
2018) (based on deep bidirectional representations from unlabeled text by
jointly conditioning on both left and right context in all layers) for solving
similar problem even without using word embeddings, iii) integrating user-
centric features, e.g., frequency of a user detected for posting hate speech
and the user’s interaction with others, iv) studying and quantifying the
difference between hate speech, abusive or offensive language, and cyber-
bullying, v) further break sentiment polarities down into specific categories,
e.g., sentiment toward Cricket, electronics, food, or celebrities.

## Acronyms

Acronyms and their full forms used in this paper are as follows:

BERT:

Bidirectional Encoder Representations from Transformers

CNN:

Convolutional Neural Networks

CSPlang:

Creative Political Slang

CRF:

Conditional Random Field

DT:

Decision Trees

DL:

Deep Learning

GBT:

Gradient Boosted Trees

GRU:

Gated Recurrent Unit

LR:

Logistic Regression

LSTM:

Long Short-term Memory

MConv-LSTM:

Multichannel Convolutional-LSTM

ML:

Machine Learning

MCC:

Matthias Correlation Coefficient

MAE:

Model Averaging Ensemble

MOWE:

Misspelling Oblivious Word Embeddings

NER:

Named Entity Recognition

NB:

Naïve Bayes

NLP:

Natural Language Processing

OOV:

Out-of-vocabulary

PoS:

Part-of-speech

RF:

Random Forest

ReLU:

Rectified Linear Unit

SVM:

Support Vector Machines.

## References

  * (1)
  * Ahmad and Amin (2016) Adnan Ahmad and Mohammad Ruhul Amin. 2016.  Bengali word embeddings for solving document classification problem. In _19th IEEE Intl. Conf. on ICCIT_. 425–430.
  * Alam (2016) Chowdhury Alam. 2016\.  Bidirectional LSTMs—CRFs networks for bangla POS tagging. In _19th IEEE intl. Conf. on ICCIT_. 377–382.
  * Alorainy et al. (2018) Wafa Alorainy, Pete Burnap, Han Liu, and Matthew Williams. 2018.  Cyber Hate Classification:’Othering’Language And Paragraph Embedding.  _arXiv preprint arXiv:1801.07495_ (2018).
  * Alrfou and Skiena (2013) Rami Alrfou and Steven Skiena. 2013.  Polyglot: Distributed Word Representations for Multilingual NLP. In _Proc. of the 17th Intl. Conference on Computational Natural Language Learning_. Association for Computational Linguistics, 183–192.  http://aclweb.org/anthology/W13-3520
  * Besacier et al. (2014) Laurent Besacier, Etienne Barnard, Alexey Karpov, and Tanja Schultz. 2014.  Automatic speech recognition for under-resourced languages: A survey.  _Speech Communication_ 56 (2014), 85–100.
  * Bian et al. (2014) Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014.  Knowledge-powered deep learning for word embedding. In _Joint European conf. on ML-KDD_. Springer, 132–148.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics_ 5 (2017), 135–146.
  * Chakrabarty and Garain (2016) Abhisek Chakrabarty and Utpal Garain. 2016.  BenLem (A Bengali Lemmatizer) and Its Role in WSD.  _ACM Transactions on Asian and Low-Resource Language Information Processing_ 15, 3 (2016), 12.
  * Chakravarthi et al. (2018) Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. 2018\.  Improving Wordnets for Under-Resourced Languages Using Machine Translation. In _Proceedings of the 9th Global WordNet Conference_.
  * Chen and Yuan (2018) Long Chen and Fajie Yuan. 2018.  Jose, and Weinan Zhang. 2018. Improving negative sampling for word representation using self-embedded features. In _Proc. of the Eleventh ACM Intl. Conf. on Web Search and Data Mining. ACM_. 99–107.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  _arXiv preprint arXiv:1703.04009_ (2017).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Ekbal et al. (2009) Asif Ekbal, Md Hasanuzzaman, and Sivaji Bandyopadhyay. 2009\.  Voted approach for part of speech tagging in Bengali. In _Proc. of the 23rd Pacific Asia Conf. on Language, Information and Computation_ , Vol. 1.
  * ElSherief et al. (2018) Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth M. Belding. 2018\.  Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media. In _Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25-28, 2018._ 42–51.  https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17910
  * Fortuna and Nunes (2018) Paula Fortuna and Sérgio Nunes. 2018.  A Survey on Automatic Detection of Hate Speech in Text.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 85\.
  * Goodman (2001) Joshua Goodman. 2001\.  Classes for fast maximum entropy training.  _arXiv preprint cs/0108006_ (2001).
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018\.  Learning Word Vectors for 157 Languages. In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Hossain and Kautz (2018) Nabil Hossain and Henry Kautz. 2018.  Discovering Political Slang in Readers’ Comments.  _New York Times_ 16 (2018), 873–421.
  * Islam (2009) MS Islam. 2009.  Research on Bangla language processing in Bangladesh: progress & challenges. In _8th Intl. Language & Development Conf._ 23–25.
  * Islam et al. (2017) Md Islam, Fazla Elahi Md Jubayer, Syed Ikhtiar Ahmed, et al. 2017\.  A comparative study on different types of approaches to Bengali document categorization.  _arXiv preprint arXiv:1701.08694_ (2017).
  * Izs´ak (2015) R. Izs´ak. 2015\.  Hate speech and incitement to hatred against minorities in the media.  _UN Humans Rights Council, A/HRC/28/64_ (2015).
  * Joulin et al. (2016b) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016b.  FastText.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Joulin et al. (2016a) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016a.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_ (2016).
  * Karim et al. (2019) Md Rezaul Karim, Michael Cochez, Joao Bosco Jares, Mamtaz Uddin, Oya Beyan, and Stefan Decker. 2019\.  Drug-Drug Interaction Prediction Based on Knowledge Graph Embeddings and Convolutional-LSTM Network. In _Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics_. ACM, 113–123.
  * Khan et al. (2018) Muhammad Khan, Md Karim, and Yangwoo Kim. 2018.  A Two-Stage Big Data Analytics Framework with Real World Applications Using Spark Machine Learning and Long Short-Term Memory Network.  _Symmetry_ 10, 10 (2018), 485.
  * Khan et al. (2019) Muhammad Ashfaq Khan, Md Karim, Yangwoo Kim, et al. 2019\.  A Scalable and Hybrid Intrusion Detection System Based on the Convolutional-LSTM Network.  _Symmetry_ 11, 4 (2019), 583.
  * Kshirsagar et al. (2018) Rohan Kshirsagar, Tyus Cukuvac, Kathleen McKeown, and Susan McGregor. 2018.  Predictive Embeddings for Hate Speech Detection on Twitter.  _arXiv preprint arXiv:1809.10644_ (2018).
  * Mansur (2006) Munirul Mansur. 2006\.  _Analysis of n-gram based text categorization for Bangla in a newspaper corpus_.  Ph.D. Dissertation. BRAC University.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. 3111–3119.
  * Müller and Schütze (2011) Thomas Müller and Hinrich Schütze. 2011.  Improved modeling of out-of-vocabulary words using morphological classes. In _Proc of the 49th Annual Meeting of the ACL: Human Language Technologies: Vol. 2_. ACL, 524–528.
  * Nockleby (2000) John Nockleby. 2000\.  Hate speech.  _Encyclopedia of American constitution_ 3 (2000).
  * Olteanu et al. (2018) Alexandra Olteanu, Carlos Castillo, Jeremy Boy, and Kush R Varshney. 2018.  The Effect of Extremist Violence on Hateful Speech Online.  _arXiv preprint arXiv:1804.05704_ (2018).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Piktus et al. (2019) Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, and Fabrizio Silvestri. 2019.  Misspelling Oblivious Word Embeddings. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 3226–3234.
  * Rahman and Kumar Dey (2018) Md Atikur Rahman and Emon Kumar Dey. 2018.  Datasets for Aspect-Based Sentiment Analysis in Bangla and Its Baseline Evaluation.  _Data_ 3, 2 (2018), 15.
  * Ribeiro et al. (2018) Manoel Horta Ribeiro, Pedro H. Calais, Yuri A. Santos, Virgilio A. F. Almeida, and Wagner Meira Jr. 2018.  Characterizing and Detecting Hateful Users on Twitter. In _Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25-28, 2018._ 676–679.  https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17837
  * Salminen et al. (2018a) Joni Salminen, Hind Almerekhi, and Milenkovic. 2018a.  Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media.. In _ICWSM_. 330–339.
  * Salminen et al. (2018b) Joni Salminen, Hind Almerekhi, Milica Milenkovic, and Jung. 2018b.  Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media.. In _ICWSM_. 330–339.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ 15, 1 (2014), 1929–1958.
  * Sumit et al. (2018) Sakhawat Hosain Sumit, Md Zakir Hossan, Tareq Al Muntasir, and Tanvir Sourov. 2018.  Exploring Word Embedding For Bangla Sentiment Analysis. In _Intl. Conf. on Bangla Speech and Language Processing (ICBSLP)_ , Vol. 21. 22\.
  * Verwimp et al. (2017) Lyan Verwimp, Joris Pelemans, Patrick Wambacq, et al. 2017\.  Character-word LSTM language models.  _arXiv preprint arXiv:1704.02813_ (2017).
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In _Proc. of the NAACL student research workshop_. 88–93.
  * Wei et al. (2017) Xiaocong Wei, Hongfei Lin, Liang Yang, and Yuhai Yu. 2017\.  A convolution-lstm-based deep neural network for cross-domain MOOC forum post classification.  _Information_ 8, 3 (2017), 92.
  * Xingjian et al. (2015) SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015\.  Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In _Advances in neural information processing systems_. 802–810.
  * Zhang and Luo (2018) Ziqi Zhang and Lei Luo. 2018.  Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter.  _arXiv preprint arXiv:1803.03662_ (2018).
  * Zhang et al. (2018) Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018\.  Detecting Hate Speech on Twitter Using a Convolution-GRU Based Neural Network. In _ESWC_. Springer, 745–760.