# Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes
Multinomial Naive Bayes classification algorithm tends to be a baseline solution for sentiment analysis task. The basic idea of Naïve Bayes technique is to find the probabilities of classes assigned to texts by using the joint probabilities of words and classes. Given the dependent feature vector $(x_1, \dots, x_n)$ and the class $C_k$. Bayes' theorem is stated mathematically as the following relationship:
$$
\begin{align}
P(C_k \mid x_1,\dots,x_n) = \frac{P(C_k)P(x_1,\dots,x_n \mid C_k)}{P(x_1,\dots,x_n)}
\end{align}
$$
According to the "naive" conditional independence assumptions, for the given class $C_k$ each feature $x_{i}$ is conditionally independent of every other feature $x_{j}$ for$j\neq i$.
$$
\begin{align}
P(x_i \mid C_k,x_1,\dots,x_n) = P(x_i \mid C_k)
\end{align}
$$
Thus, the relation can be simplified to
$$
\begin{align}
P(C_k \mid x_1,\dots,x_n) = \frac{P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k)}{P(x_1,\dots,x_n)}
\end{align}
$$
Since $P(x_1,...x_n)$ is constant, if the values of the feature variables are known, the following classification rule can be used:
$$
\begin{align}
P(C_k \mid x_1,\dots,x_n) \propto  P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k) \\ \Downarrow \\ \hat{y} = \underset{k}{\arg\max} P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k)
\end{align}
$$
To avoid underflow, log probabilities can be used.
$$
\begin{align}
\hat{y} = \underset{k}{\arg\max}(\ln{P(C_k)}+\sum_{i=1}^{n}\ln{P(x_i \mid C_k)})
\end{align}
$$
The variaty of naive Bayes classifiers primarly differs between each other by the assumptions they make regarding the distribution of $P(x_i \mid C_k)$, while $P(C_k)$ is usually defined as the relative frequency of class $C_k$ in the training dataset.

The multinomial distribution is parametrized by vectors $\theta_k = (\theta_{k1},\ldots,\theta_{kn})$ for each class $C_k$, where $n$ is the number of features (i.e. the size of the vocabulary) and $\theta_{ki}$ is the probability $P(x_i \mid C_k)$ of feature $i$ appearing in a sample that belongs to the class $C_k$.

The parameters $\theta_y$ is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:
$$
\begin{align}
\hat{\theta}_{ki} = \frac{ N_{ki} + \alpha}{N_k + \alpha n}
\end{align}
$$
where $N_{ki} = \sum_{x \in T} x_i$ is the number of times feature $i$ appears in a sample of class y in the training set $T$, and $N_{y} = \sum_{i=1}^{|T|} N_{ki}$ is the total count of all features for class $C_k$. The smoothing priors $\alpha \ge 0$ accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting $\alpha = 1$ is called Laplace smoothing, while $\alpha < 1$ is called Lidstone smoothing.

Thus, the final decision rule is defined as follows:
$$
\begin{align}
\hat{y} = \underset{k}{\arg\max}(\ln{P(C_k)}+\sum_{i=1}^{n}\ln{\frac{ N_{ki} + \alpha}{N_k + \alpha n}})
\end{align}
$$



## 1. Loading data
Sentiment dataset of Tweets in Russian [1] is avaialble at http://study.mokoron.com/. The files positive.csv and negative.csv contain positively labelled and negatively labelled tweets, respectively.


```python
import pandas as pd
import numpy as np
```


```python
# Read the data from CSV files
n = ['id', 'date','name','text','typr','rep','rtw','faw','stcount','foll','frien','listcount']
data_positive = pd.read_csv('positive.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])
data_negative = pd.read_csv('negative.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])
```


```python
# Create balanced dataset
sample_size = min(data_positive.shape[0], data_negative.shape[0])
raw_data = np.concatenate((data_positive['text'].values[:sample_size], 
                           data_negative['text'].values[:sample_size]), axis=0) 
labels = [1]*sample_size + [0]*sample_size
```

## 2. Preprocessing data
Texts generated by humans in social media sites contain lots of noise that can significantly affect the results of the sentiment classification process. Moreover, depending on the features generation approach, every new term seems to add at least one new dimension to the feature space. That makes the feature space more sparse and high-dimensional. Consequently, the task of the classifier has become more complex.

To prepare messages, such text preprocessing techniques as replacing URLs and usernames with keywords, removing punctuation marks and converting to lowercase were used in this program. 


```python
import re

def preprocess_text(text):
    text = re.sub('((www\.[^\s]+)|(https?://[^\s]+))','URL', text)
    text = re.sub('@[^\s]+','USER', text)
    text = text.lower().replace("ё", "е")
    text = re.sub('[^a-zA-Zа-яА-Я1-9]+', ' ', text)
    text = re.sub(' +',' ', text)
    return text.strip()

data = [preprocess_text(t) for t in raw_data]
```

## 3. Training MultinomialNB
A Pipeline class was used to make the vectorizer => transformer => classifier easier to work with. Such hyper-parameters as n-grams range, IDF usage, TF-IDF normalization type and Naive Bayes alpha were tunned using grid search. The performance of the selected hyper-parameters was measured on a test set that was not used during the model training step.


```python
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split, GridSearchCV

text_clf = Pipeline([('vect', CountVectorizer()),
                     ('tfidf', TfidfTransformer()),
                     ('clf', MultinomialNB())])

tuned_parameters = {
    'vect__ngram_range': [(1, 1), (1, 2), (2, 2)],
    'tfidf__use_idf': (True, False),
    'tfidf__norm': ('l1', 'l2'),
    'clf__alpha': [1, 1e-1, 1e-2]
}
```

The dataset was plited into train and test subsets.


```python
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, random_state=42)
```


```python
from sklearn.metrics import classification_report

score = 'f1_macro'
print("# Tuning hyper-parameters for %s" % score)
print()
np.errstate(divide='ignore')
clf = GridSearchCV(text_clf, tuned_parameters, cv=10, scoring=score)
clf.fit(x_train, y_train)

print("Best parameters set found on development set:")
print()
print(clf.best_params_)
print()
print("Grid scores on development set:")
print()
for mean, std, params in zip(clf.cv_results_['mean_test_score'], 
                             clf.cv_results_['std_test_score'], 
                             clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))
print()

print("Detailed classification report:")
print()
print("The model is trained on the full development set.")
print("The scores are computed on the full evaluation set.")
print()
print(classification_report(y_test, clf.predict(x_test), digits=4))
print()
```

    # Tuning hyper-parameters for f1_macro
    
    Best parameters set found on development set:
    
    {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    
    Grid scores on development set:
    
    0.732 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.741 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.705 (+/-0.008) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.728 (+/-0.004) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.730 (+/-0.005) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.702 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.734 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.753 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.711 (+/-0.007) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.738 (+/-0.004) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.750 (+/-0.005) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.712 (+/-0.008) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.726 (+/-0.005) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.748 (+/-0.008) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.707 (+/-0.009) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.738 (+/-0.005) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.752 (+/-0.006) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.711 (+/-0.008) for {'clf__alpha': 0.1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.717 (+/-0.006) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.737 (+/-0.007) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.699 (+/-0.008) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.734 (+/-0.005) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.750 (+/-0.007) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.707 (+/-0.007) for {'clf__alpha': 0.1, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.710 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.729 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.692 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.727 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.744 (+/-0.007) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.700 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    0.706 (+/-0.005) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
    0.715 (+/-0.007) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
    0.686 (+/-0.009) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
    0.721 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
    0.731 (+/-0.006) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
    0.694 (+/-0.008) for {'clf__alpha': 0.01, 'tfidf__norm': 'l2', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
    
    Detailed classification report:
    
    The model is trained on the full development set.
    The scores are computed on the full evaluation set.
    
                 precision    recall  f1-score   support
    
              0     0.7397    0.7941    0.7659     37078
              1     0.7759    0.7183    0.7460     36792
    
    avg / total     0.7577    0.7564    0.7560     73870
    
    


## 4. Conslusion
The model, which was trained on the development set, demonstrated $F_1=0.765$ on the evaluation set.

## References
1. Y. Rubtsova, "Constructing a Corpus for Sentiment Classification Training", Software & Systems, vol. 109, no. 1, pp. 72-78, 2015. 
2. "Naive Bayes", scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/modules/naive_bayes.html. [Accessed: 26- Aug- 2018]. 
3. "Working With Text Data", scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html. [Accessed: 26- Aug- 2018].


```python

```

5cf6268e393c7733065560aed231ac71d4488108
ipynb
Jupyter Notebook
Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes.ipynb
sismetanin/sentiment-analysis-of-tweets-in-russian
23a849502341c9bfedf28a65ba2df9818a06184e
MIT
2018-09-07T10:00:08.000Z
2021-11-18T15:35:55.000Z
Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes.ipynb
sismetanin/sentiment-analysis-of-tweets-in-russian
23a849502341c9bfedf28a65ba2df9818a06184e
MIT
2019-09-15T18:45:36.000Z
2021-02-27T12:08:05.000Z
Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes.ipynb
sismetanin/sentiment-analysis-of-tweets-in-russian
23a849502341c9bfedf28a65ba2df9818a06184e
MIT
2018-09-02T07:13:22.000Z
2021-06-04T05:31:34.000Z
Qwen/Qwen-72B
1. YES
2. YES