i.e., it is very easy to include new (off-the-shelf) methods. Also, no training data is necessary for this method, which fits well with our purpose of an unsupervised solution. On the other hand, majority voting is not as flexible as 10SENT in coping with all the diversity of the methods.This is due to the training phase of 10SENT that allows it to capture some idiosyncracies of each one of them.

0

More specifically, this combination works as follows: given an unlabeled instance $x$, the labels candidates set $L = \{ w_1 , w_2 , w_3 \}$ and a set of voting methods $M = \{ m_1 , m_2 , m_3 , …, m_n \}$ , we define set $V_{M\times L}$ of votes $v_{ij}$ of a sentence $x$ as classes $w_j$ given by the method $m_i$ as follows:

$v_{ij}$ =

1,& ;\
0,& .

The final result $R$ is given by the class $j$ with maximum amount of votes: $$R = argmax\bigg( \sum_{i=1}^M V_{ij} \bigg)$$

As **evaluation metric** the use the popular Micro and Macro-F1 scores. Micro-F1 captures the overall accuracy across all classes. Macro-F1 calculates the F1 score for each class separately report the average of these scores for all 