SLCNN: Sentence-Level Convolutional Neural Network for Text
  Classification
Text classification is a fundamental task in natural language processing
(NLP). Several recent studies show the success of deep learning on text
processing. Convolutional neural network (CNN), as a popular deep learning
model, has shown remarkable success in the task of text classification. In this
paper, new baseline models have been studied for text classification using CNN.
In these models, documents are fed to the network as a three-dimensional tensor
representation to provide sentence-level analysis. Applying such a method
enables the models to take advantage of the positional information of the
sentences in the text. Besides, analysing adjacent sentences allows extracting
additional features. The proposed models have been compared with the
state-of-the-art models using several datasets. The results have shown that the
proposed models have better performance, particularly in the longer documents.
Ali Jarrahi, Ramin Mousa, Leila Safari
SLCNN: Sentence-Level Convolutional Neural Network for Text Classification

###### Abstract

Text classification is a fundamental task in natural language processing (NLP). Several recent studies show the success of deep learning on text processing. Convolutional neural network (CNN), as a popular deep learning model, has shown remarkable success in the task of text classification. In this paper, new baseline models have been studied for text classification using CNN. In these models, documents are fed to the network as a three-dimensional tensor representation to provide sentence-level analysis. Applying such a method enables the models to take advantage of the positional information of the sentences in the text. Besides, analysing adjacent sentences allows extracting additional features. The proposed models have been compared with the state-of-the-art models using several datasets. The results have shown that the proposed models have better performance, particularly in the longer documents.

Text Classification, Deep Learning, Convolutional Neural Network, Natural Language Processing +
Footnote †: _Corresponding author._

E-mail address: jarrahi@znu.ac.ir

## 1 Introduction

In recent years, the production of unstructured texts (documents) has grown exponentially. Unstructured texts can be found everywhere, e.g., emails, social media, chat conversations, comments and websites. Although text data can be a rich source of information, it is hard to extract value from this type of unstructured data.

Text classification is a fundamental task in natural language processing (NLP). The task is the process of assigning a class label from a set of predefined classes to a given text according to its content, and has many applications such as sentiment analysis [15], spam detection [16] and topic categorization [1].

Text classification can be done manually or automatically. Despite the manual method is more accurate, it is very costly and time consuming. Therefore, to provide scalability, several machine learning, NLP and other techniques are used for automatic text classification.

Supervised learning is a machine learning task of learning a function (classifier) using pre-labeled samples as a training dataset [17]. A key step in supervised learning is feature extraction. Traditional machine learning methods represent text with hand-crafted methods, e.g., n-grams [14]. Recently, deep learning methods have been used for automatic feature extraction, including convolutional neural networks (CNNs) [12], recurrent neural networks (RNNs) [15] and particularly long short-term memory (LSTM) [16].

In this paper, we present a new baseline model for text classification using CNN. In this model, documents are fed to the network as a three-dimensional tensor representation to provide sentence-level analysis.

The paper is structured as follows. The previous works have been summarized in the next section. The details of the proposed methods are described in section 3. We have evaluated our approach on several benchmark datasets. The experimental results are presented in section 4. Finally, the paper concludes with future research directions in section 5.
## 2 Related works

Different approaches have been proposed for text classification. Initial approaches were based on the classical machine learning techniques, which followed two stages, i.e., extracting hand-crafted features and classifying the documents. Typical features include bag-of-words (BoW), n-grams, and their TF-IDF1. Alternatively, several recent studies show the success of deep learning on text classification. As the neural networks receive their inputs numerically, word embeddings, e.g., word2vec [16] or GloVe [14], are usually used to represent words as a numerical vectors by capturing the similarities/regularities between words.

There are variety of deep learning models for text classification. Due to the sequential nature of textual data, recurrent neural networks (RNN), including long short-term memory (LSTM) and gated recurrent units (GRU) [15] have been widely used in text processing. For example, in [17] authors examined generative and discriminative LSTM models for text classification. They found that although the generative models perform better than BoW, they have a higher asymptotic error rates than discriminative RNN-based models.

Another popular model is CNN, which originally invented for computer vision [13]. Subsequently, CNN models have been applied in NLP and have achieved excellent results [12]. Many researchers have worked on the effective use of CNNs in text classification since a single layer word-level CNN was successfully used in sentence classification with a pre-trained word embeddings [10]. The proposed method in [11] was the first attempt to perform text classification entirely at the character-level, and reported competitive results. Their models use 70 characters by one-hot encoding, including 26 English letters, 10 digits, 33 other characters and the new line character. [1] adopted very deep convolutional networks, i.e., ResNet [1], to the character-level text classification.

Some researchers tried to improve performance of the models by applying extra mechanisms. Attention is one of the most effective mechanism that selects significant information to achieve superior results [15]. Deep neural networks with attention mechanism can yield better results. Some of the remarkable examples include source-target attention and self-attention [10]. Particularly, two-level attention mechanism, i.e., word attention and sentence attention, was developed on GRU by [18] for document classification. In [18], authors used dense connections with multi-scale feature attention in order to produce variable n-gram features. Since this paper aims to present a new baseline model, employing such mechanisms has been avoided.

## 3 Method

In this section, we describe the architecture of proposed Sentence-Level Convolutional Neural Network (SLCNN) for classifying the documents. The key idea of the model is that using positional information of each sentence in the document may improve the performance of the classifier. Furthermore, analysing adjacent sentences allows extracting some extra features, e.g., writing style features, which can be useful in some applications, such as spam review detection and fake news detection. Hence, we present two baseline models based on the CNN architecture for the text classification task. For this purpose, we introduce a three-dimensional representation of documents to enable sentence-level analysis. The pre-processing phase and the architecture of the SLCNN and its variant SLCNN+V are explained in the following subsections.

### Pre-processing

During the pre-processing phase, the documents are cleaned by removing some unimportant characters, like the html tags and the punctuations. Then all words are normalized by converting to their lowercase forms. After that, as the most important step, each document is transformed into a three-dimensional tensor, illustrated in Figure 1. As shown in the figure, the sentences of the document form the first dimension of the tensor. In the same way, the words of the sentences shape the second dimension, while the third dimension represents the word vectors of the words. The pre-trained word embeddings, e.g., word2vec and GloVe, could be used for representing the word vectors.

Since, the input size of the network must be fixed, and according to different size of both the texts and the sentences, we consider two thresholds, one for the number of sentences in the documents, T\({}_{d}\), and another for the number of
words in the sentences, T\({}_{s}\). The documents and the sentences longer than the thresholds would be cropped and shorter ones would be padded by zeros.

After some statistical analysis on the datasets in our experiments, as well as considering the structure of the SLCNN, we chose T\({}_{s}\)=46. In the same way, the threshold for the number of sentences in the documents is calculated by the following equation:

\[T_{d}=\left[\mu+1.5\;\sigma\right] \tag{1}\]

where \(\mu\) is the average number of sentences in the documents, and \(\sigma\) is the standard deviation. As a result, the outlier sizes are ignored to prevent model from constructing very large and sparse tensors. The relevant statistical data is provided in section 4.

### 3-2. The Architecture

The architecture of the proposed models is illustrated in Figure 2. Overall, in the input layer, the documents are provided in the form of the 3D tensor, introduced in section 3-1. After that, using four horizontal convolutional blocks (HCB), one feature per filter is extracted for each sentence individually. In other words, one feature vector for each sentence is provided just before the fully-connected layers with the size equal to the number of filters. In this way, in addition to the word-level features, the positional information of the sentences is also used in the learning process. Moreover, as mentioned before, analysing of adjacent sentences can extract some useful features. For this purpose, the second model (SLCNN+V) is created by adding a vertical convolutional block (VCB) before fully-connected layers. Finally, there are two fully-connected (dense) layers which end to the output layer.

Figure 1: Shape of the converted documents.

Figure 2: The architecture of the proposed models. The dashed block (VCB) is used only in SLCNN+V.
Looking at the details of the convolutional blocks, as shown in Figure 3, there are two sequential convolution layers, each one followed by a Rectified Linear Unit (ReLU) activation function, _f(x)= max (0, x)_. A convolution operation consists of a filter \(w\in\mathbb{R}^{s\times t\times d}\), which is applied to each possible window of \(s\kappa t\) features from its input _feature map_, \(X\), to produce a new feature map by equation 3:

\[\begin{array}{l}X=\begin{bmatrix}x_{1,1}&x_{1,2}&...&x_{1,n}\\ x_{2,1}&x_{2,2}&...&x_{2,n}\\ \vdots&\vdots&\vdots\\ x_{m,1}&x_{m,2}&...&x_{m,n}\\ \end{bmatrix}\\ \\ \tilde{x}_{i,j}=f(w.\,x_{i,j:i+s-1,j+t-1}+b)\end{array} \tag{2}\]

where \(x_{ij,x}\) is the concatenation of features within the specified interval, \(b\in\vec{H}\) is a bias term and \(f\) is a non-linear function such as the ReLU. For the HCB, we consider \(s\)=1 and \(t\)=2, and for the VCB \(s\)=2 and \(t\)=1. It should be noted that, in the first convolution layer of the first HCB, \(d\) (the third dimension of the filters) is equal to the size of the word vectors, and in other cases \(d\)=1. At the end of the blocks, there is a max-pooling operation, with the pooling size = 2, that is applied over the generated intermediate feature map to select the maximum value from any two adjacent features as a more important feature. The new feature map is calculated by following equations:

\[\begin{array}{l}\tilde{x}_{i,j}=\begin{cases}\max\{\,x_{i,2j-1},x_{i,2j}\}&, for\,the\,HCB\\ \max\{\,x_{2i-1,j},x_{2i,j}\}&,for\,the\,VCB\end{cases}\end{array} \tag{3}\]

The process of extracting one feature from one filter was described. The model uses multiple filters to obtain multiple features. The final extracted features are passed to the fully-connected layers that end to a softmax output layer which is the probability distribution over labels. For regularization, a dropout module (Hinton et al. 2012) is employed after each fully-connected layer.

## 4 Experiments

### Experimental settings

The Natural Language Toolkit (NLTK) was used in order to tokenize words and sentences. In the input layer, as mentioned before, pre-trained word-embeddings are used to convert the words into the corresponding word vectors. We used 100-dimensional GloVe in our experiments. Out-Of-Vocabulary (00V) words were initialized from a uniform distribution with range [-0.01, 0.01]. We set number of filters to 128 for all the convolutional blocks. Also, we considered two different sizes for fully-connected layers, shown in Table 1. Both the dropout rates were set to 0.5. The model's parameters were trained by the Adam Optimizer (Kingma and Ba 2014), with the initial learning rate of 0.001. The model has been implemented using Keras and run for 50 epochs.

Figure 3: The convolutional blocks. k is the number of filters. (a) HCB and (b) VCB.
### Benchmark Datasets

We utilized six datasets covering different classification tasks compiled by [22]. General specifications are presented in Table 2. All data are evenly distributed across class labels. AG and DBPedia are news and ontology classification datasets, respectively. Yelp and Amazon are sentiment classification datasets, where '.P' (Polarity) in the dataset names indicates that the labels are binary while '.F' (Full) means that the labels refer to the number of stars.

Some of the statistical information extracted from the datasets, after the pre-processing step, is summarized in Table 3. As presented in the table, by considering T\({}_{s}\)=46, the proportions of cropped sentences are between 2 and 2.9 percent, that shows the length of sentences in the different datasets are almost similar. By contrast, the number of sentences of the documents in the different datasets are quite different. By utilizing Equation 1, T\({}_{d}\) for AG News, DBPedia, Amazon and Yelp are equal to 4, 6, 10 and 20 respectively. Also, the proportions of cropped documents, using relevant T\({}_{d}\), are 0.4, 3, 3.6 and 6 percent for AG News, Amazon, DBPedia and Yelp respectively, which means that the variance of the number of sentences in the documents of Yelp is greater than others.

### Results

We compared our models with several popular base models, e.g., linear models [22], RNN-based model, i.e., Discriminative-LSTM [30], and CNN-based models including classical word-level CNN [22], character-level CNN [22], very deep CNN [15] and CNN

\begin{table}
\begin{tabular}{l|c} \hline \hline Layers & Small & Large \\ \hline Fully-connected 1 & 512 & 1024 \\ \hline Fully-connected 2 & 512 & 1024 \\ Output & Depends on the problem \\ \hline \hline \end{tabular}
\end{table}
Table 1: Fully-connected layers in our experiments.

\begin{table}
\begin{tabular}{l|c c c c c|c c} \hline \hline Datasets & AG News & DBPedia & Yelp.P & Yelp.F & Amazon.P & Amazon.F \\ \hline \# of training samples & 120k & 560k & 560k & 650k & 3600k & 3000k \\ \hline \# of test samples & 7.6k & 70k & 38k & 50k & 400k & 650k \\ \hline \# of classes & 4 & 14 & 2 & 5 & 2 & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Datasets in our experiments.

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \hline Statistics & AG & DBPedia & Yelp.P & Yelp.F & Amazon.P & Amazon.F \\ \hline \# of sentences & 164k & 1505k & 5082k & 5958k & 18654k & 16986k \\ \hline Cropped sentences (\%) & 2 & 2.9 & 2.6 & 2.6 & 2.4 & 2.5 \\ \hline Cropped documents (\%) & 0.4 & 3.6 & 6 & 6 & 3.1 & 3 \\ \hline Documents that contain cropped sentences (\%) & 2.5 & 6.9 & 16.1 & 16.4 & 10.3 & 10.9 \\ \hline \# of sentences in the longest text & 15 & 25 & 141 & 151 & 85 & 99 \\ \hline \# of words in the longest sentence & 135 & 1302 & 1104 & 1175 & 522 & 520 \\ \hline Vocab size & 62k & 786k & 283k & 311k & 1546k & 1464k \\ \hline T\({}_{d}\) & 4 & 6 & 20 & 20 & 10 & 10 \\ \hline \# of trainable parameters in SLCNN _small_ & 783k & 920k & 1831k & 1832k & 1176k & 1177k \\ \hline \# of trainable parameters in SLCNN _large_ & 1835k & 2107k & 3930k & 3933k & 2619k & 2622k \\ \hline \# of trainable parameters in SLCNN+V _small_ & 653k & 723k & 1176k & 1177k & 848k & 850k \\ \hline \# of trainable parameters in SLCNN+V _large_ & 1508k & 1649k & 2554k & 2557k & 1899k & 1902k \\ \hline Training time for a single epoch (s) & 10 & 51 & 150 & 170 & 510 & 440 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The statistical information of the datasets.
with fastText embedding (Joulin et al. 2017). Since our aim was to provide new baseline models, and using other mechanisms, such as the attention, has been avoided, therefore such models have been excluded from the comparison. The results are listed in Table 4 based on accuracy. Overall, it can be seen that the proposed models have outperformed all the models in half of the datasets, DBPedia, Yelp.P and Yelp.F. Especially, the improvement is significant in Yelp datasets, i.e., around 2 percent in Yelp.P and around 5 percent in Yelp.F compared to character-level and word-level CNNs. In terms of Amazon datasets, the SLCNN+V was ranked third after VDCNN and character-level CNN with around 94 and 58.1 percent in Amazon.P and Amazon.F, respectively.

If we look at AG News, despite competitive results with other CNN models, n-grams and Discriminative-LSTM have achieved better results. One of the main reasons we can mention is the number of sentences in the documents. So that the proposed models perform better in documents with large number of sentences, i.e., Yelp. Another reason that hinders better performance in Amazon datasets is the very high vocabulary size (see Table 3), since we used the word embedding with just over 1M vocabularies in our experiments.

## 5 Conclusion and future works

This paper offers new baseline models for text classification using a sentence-level CNN. The key idea is representing the documents as a 3D tensor to enable the models to sentence-level analysis. The proposed models have been compared with the state-of-the-art models using several datasets. The results have shown that the proposed models have better performance, particularly in the longer documents.

As future works, the attention mechanism will be utilized in the proposed models in order to improve the overall performance. Also, we will work on sentence standardization. We believe that applying a standard form of sentences enables the proposed models to use compositional methods (with different 3D filters), due to the 3D structure of the input tensor.
