# A simple language-agnostic yet very strong baseline system for hate speech
and offensive content identification

Yves Bestgen
Laboratoire d’analyse statistique des textes - Statistical Analysis of Text
Laboratory (LAST - SATLab)
Institut de recherche en sciences psychologiques
Université catholique de Louvain
Place Cardinal Mercier, 10 1348 Louvain-la-Neuve, Belgium
<EMAIL_ADDRESS>

###### Abstract

For automatically identifying hate speech and offensive content in tweets, a
system based on a classical supervised algorithm only fed with character
n-grams, and thus completely language-agnostic, is proposed by the SATLab
team. After its optimization in terms of the feature weighting and the
classifier parameters, it reached, in the multilingual HASOC 2021 challenge, a
medium performance level in English, the language for which it is easy to
develop deep learning approaches relying on many external linguistic
resources, but a far better level for the two less resourced language, Hindi
and Marathi. It ends even first when performances are averaged over the three
tasks in these languages, outperforming many deep learning approaches. These
performances suggest that it is an interesting reference level to evaluate the
benefits of using more complex approaches such as deep learning or taking into
account complementary resources.

The diffusion of hate speech and offensive content in social networks has
become a crucial problem. The tremendous number of posts broadcasted at any
given time prevents their identification by human evaluation. This task is
made even more complex by the large number of languages in which these
offensive contents are spread. Not surprisingly, a lot of research is being
done to develop automatic detection systems. As in many NLP domains, deep
learning approaches and the use of pre-computed embeddings have proven to be
the most efficient, even in languages with few resources [Mandl et al., 2019,
Mandl et al., 2020]. However, traditional machine learning systems have
sometimes proven to be very competitive [Mujadia et al., 2019, Saroj et al.,
2019]. One may thus wonder what level of performance can be achieved by a much
simpler yet heavily optimized classical supervised approach, completely
language-agnostic, based only on a few thousand examples to feed the
supervised learner but without any additional resources. If this system is
(relatively) successful, it would give a computationally easy baseline that
could help evaluating the benefits of additional knowledge, complex
architectures, deep learning or language expertise.

The HASOC 2021 shared task ”Hate Speech and Offensive Content Identification
in English and Indo-Aryan Languages” [Modha et al., 2021] is particularly
relevant for developing such a system because it proposes three languages.
Among them, one, English, is obviously the most studied language in automatic
language processing and the one in which the largest number of resources is
available. Hindi and, even more so, Marathi have been much less studied and
are still classified as low-resource languages [Haffari et al., 2018, Ortega
et al., 2021, Gaikwad et al., 2021]. One can think a priori that the approach
proposed here will be much more competitive in these two languages.

The remainder of this paper presents the datasets made available for this
shared task and the challenge rules, the system developed, and the results
obtained which confirms that the proposed approach is a strong language-
agnostic baseline for hate speech and offensive content identification.

##  1 Materials and Task

The SATLab participated in subtask 1 of the HASOC 2021 shared task ”Hate
Speech and Offensive Content Identification in English and Indo-Aryan
Languages” which proposes two problems to be solved in three languages [Modha
et al., 2021]. The first problem requires to categorize tweets into two
categories: Hate and Offensive (HOF) or not (NOT). It is proposed for English,
Hindi and Marathi. The second problem requires categorizing the same tweets
into four categories, by dividing the Hate and Offensive category into three
subcategories: Hate speech (HATE), Offensive (OFFN) and Profane (PRFN). It is
offered for English and Hindi.

For each language, learning and test materials have been provided by the task
organizers [Mandl et al., 2021, Gaikwad et al., 2021]. The frequencies (#) and
percentages (%) in each category of each problem for each language are given
in Table 1.

Table 1: Dataset statistics of subtask 1 |  | Learning Phase | Learning | Test
---|---|---|---|---
|  | Problem 1 |  | Problem 2 |  | Phase | Phase
|  | NOT | HOF |  | NONE | HATE | OFFN | PRFN |  | Total | Total
English | # | 1342 | 2501 |  | 1342 | 683 | 622 | 1196 |  | 3843 | 1281
| % | 34.9 | 65.1 |  | 34.9 | 17.8 | 16.2 | 31.1 |  | 75.0 | 25.0
Hindi | # | 3161 | 1433 |  | 3161 | 566 | 654 | 213 |  | 4594 | 1532
| % | 68.8 | 31.2 |  | 68.8 | 12.3 | 14.2 | 4.6 |  | 75.0 | 25.0
Marathi | # | 1205 | 669 |  |  |  |  |  |  | 1874 | 525
| % | 64.3 | 35.7 |  |  |  |  |  |  | 78.1 | 21.9

This table deserves several comments. First of all, the learning set is much
smaller in Marathi (18% of the total) than in the other two languages, the
difference between the two latter being much smaller (37% of the total in
English and 45% of the total in Hindi). The proportion of tweets in the HOF
category is much larger in English than in the other two languages. The
difference clearly comes from the PRFN category which is much more frequent in
English than in Hindi where it represents only a very small percentage.

###  1.1 Challenge rules

The rules of the challenge allowed teams to use any additional resources
including materials from previous HASOC tasks, lexical norms such as emotional
word lists, precomputed embeddings, the use of syntactic parsers or even
machine translation systems to analyze other languages in English. The system
proposed by the SATLab does not include any of these additional resources.

The official measure chosen by the organizers to rank the teams in the
challenge is the Macro-F1 which has the advantage of giving the same weight to
all categories, however rare they may be (e.g., less than 5% of PRFN in
Hindi).

Each team was allowed to submit five runs for each subtask between August 20
and 30, 2021, and the team’s best performance was displayed in the
Leaderboard. Compared to the ten or so other shared tasks I participated in,
it is important to underline that the submission system proposed by the
challenge organizers (https://hasocfire.github.io/submission/login.html) was
particularly ergonomic. Moreover, the fact that the teams could not hide their
best score, as it is often the case in other systems, made, in my opinion, the
competition more fair.

##  2 Proposed System

In order to meet the requirements presented in the introduction, the proposed
system is only based on character n-grams [Bestgen, 2017], an approach
frequently used in automatic language processing when the developed system has
to support several languages. These n-grams were extracted from the lowercased
tweets with the only specificity that those starting or ending the tweet were
distinguished from the others by the presence of a specific character. All
character n-grams observed at least twice in the material were retained.

During the n-gram extraction, three parameters had to be set:

  * •

The length of the n-grams in number of characters. The minimum length was
systematically set to 1 while the maximum lengths evaluated varied between
four and eight characters.

  * •

The weighting applied to the frequency of each feature in each instance. Two
well-established weighting schema were evaluated:

    * –

Sublinear TF-IDF:

$\text{(sl)TF-IDF}=(1+\log(tf))\times\log\frac{N}{df}$ (1)

where $tf$ refers to the frequency of the term in the document, $N$ is the
number of documents in the set and $df$ the number of documents that include
the term.

    * –

BM25 ([Robertson and Zaragoza, 2009, Bestgen, 2021b]), which is considered as
one of the most efficient weighting schema [Manning et al., 2008]. It is a
kind of TF-IDF that takes into account the length of the document. The
following formula was used:

$\text{BM25}=\frac{tf}{tf+k_{1}*(1-b+b*\frac{dl}{dl-
avg_{dl}})}\times\log\frac{N-df+0.5}{df+0.5}$ (2)

in which

      * *

$\frac{tf}{tf+k_{1}}$ is the TF component which, contrarily to the usual TF-
IDF, has an asymptotic maximum tuned by the $k_{1}$ parameter.

      * *

$(1-b+b*\frac{dl}{dl-avg_{dl}})$, where $dl$ is the length of the document and
$avg_{dl}$, the average length of the documents in the set, is the document
length normalization factor whose impact is tuned by parameter $b$ (and by
$k_{1}$).

      * *

The second part of the formula is a variant of the usual IDF, proposed by
Robertson and Spärck Jones [Robertson and Zaragoza, 2009].

In our analyses, $k_{1}$ was set to 2 and $b$ to 0.75.

  * •

Normalization of the feature scores for each instance:

    * –

The classical L2 regularization.

    * –

A MinMax transformation:

$\text{MinMax}=\frac{Feature_{i}\\_score-min}{max-min}+0.01$ (3)

It is important to note that this transformation is applied independently to
each instance and not, as is often the case, to each feature. The value of
0.01 is added to distinguish the lowest scoring feature of an instance with
the value of 0, which codes the absence of a feature.

These character n-grams were the only features provided to the supervised
learning procedure. Two well-established procedures were evaluated:

  * •

The (dual) L2-regularized logistic regression as implemented in the LIBLinear
package [Fan et al., 2008], an extremely fast approach and very simple to use
because it only requires the optimization of two parameters. The two
parameters to optimize are the regularization parameter C and -wi which allows
to adjust this parameter C for the different categories. This approach was
used for the initial submission to each of the five problems.

  * •

A much slower and more complex approach to optimize because it requires the
optimization of many parameters, but that has recently outperformed all deep-
learning based systems participating in the CMCL 2021 shared task on
predicting gaze data during reading [Bestgen, 2021a]: a gradient boosting
decision tree approach as implemented in the LightGBM free software [Ke et
al., 2017]. This approach has been used only in a second time.

The system was independently optimized for each language during the learning
phase using a 3-fold cross-validation procedure, whose folds were stratified
according to the four categories of problem 2 for English and Hindi and the
two categories of problem 1 for Marathi. This cross-validation step led to
setting the parameters described above as shown in Table 2 for the initial
SATLab submissions.

Table 2: Parameters for the initial submissions Language | English | Hindi | Marathi
---|---|---|---
Problem | 1 | 2 | 1 | 2 | 1
N-gram length | 5 | 5 | 5 | 5 | 5
Weighting | TF-IDF | TF-IDF | BM25 | TF-IDF | BM25
Normalization | MinMax | L2 | L2 | MinMax | L2
C | 1.1 | 2.5 | 3.7 | 0.083 | 6
w_HOF | 0.5 |  | 2.2 |  | 2
w_HATE |  | 2.0 |  | 1.87 |
w_OFFN |  | 3.0 |  | 0.93 |
w_PRFN |  | 0.8 |  | 5.60 |

##  3 Results

In this section, the performance of the initial system proposed by the SATLab
and the various optimization attempts that have been made are first presented.
Secondly, these performances are compared to those of other teams in order to
determine whether the proposed approach is competitive enough to serve as a
baseline for evaluating the benefits of using deep learning approaches and
resources supplementary to those provided in the task itself.

Table 3: Macro-F1 during cross-validation and on the test set Language | English | Hindi | Marathi
---|---|---|---
Problem | 1 | 2 | 1 | 2 | 1
CV | 0.7483 | 0.5876 | 0.7551 | 0.5133 | 0.8565
Initial | 0.7635 | 0.6114 | 0.7718 | 0.5563 | 0.8547
Best LR |  |  |  | 0.5586 | 0.8595
Best LGBM | 0.7823 |  |  |  | 0.8749

###  3.1 SATLab submissions

Table 3 presents the performance of the main versions of the SATLab system
submitted for the five problems and thus the benefits brought by the
optimization attempts on the test set. The first row reports the performance
of the original system for each problem during the cross-validation step.
Logically, the performances are less good for problems requiring the
identification of more than two categories as well as when a category is
particularly rare (Hindi-2). We also observe strong differences between the
three languages. Since only one split into three folds was used, one can
assume that these scores are, at least slightly, overestimated.

The second row shows the performance of the same versions on the test set and
thus the initial submissions to the challenge. All scores are higher on the
test set than during the cross-validation step.

As it was allowed to submit five runs for each problem, I first tried to
optimize the classifier based on logistic regression by modifying very
slightly the two LIBLinear parameters (i.e., C and -wi). These attempts
brought a (very) slight benefit for two of the five problems as shown in the
third row of Table 3.

In a second step, an LightGBM classifier was trained using a random grid
search procedure for each of the five problems to try optimizing the
parameters. As shown in the fourth row of Table 3, this step resulted in a
stronger performance improvement in two problems: English-1 and Marathi. For
the other three problems, LightGBM did not improve the performance of
LIBLinear. The selected parameters for the two successful problems are given
in Appendix 1. The number of boosting iterations was determined during cross-
validation by using the LightGBM early stopping procedure which stops training
when the performance on the validation fold doesn’t improve in the last 200
rounds. The final system values on the test set for the five problems are
bolded in the table. The run names of these solutions in the official
leaderboard are respectively: English 1b, English 2, Hindi 1, Hindi B S4 and
Marathi 3.

Table 4: Macro-F1 on the test set for the five problems English-1 N=56 | English-2 N=37
---|---
Rank | Team | Macro-F1 | Rank | Team | Macro-F1
1 | NLP-CIC | 0.8305 | 1 | NLP-CIC | 0.6657
2 | HUNLP | 0.8215 | 2 | neuro-utmn-thales | 0.6577
3 | neuro-utmn-thales | 0.8199 | 3 | HASOC21rub | 0.6482
… |  |  | … |  |
22 | hate-busters | 0.7894 | 15 | KuiYongyi | 0.6116
23 | SATLab | 0.7823 | 16 | SATLab | 0.6114
24 | TAD | 0.7776 | 17 | hate-busters | 0.6096
… |  |  | … |  |
Hindi-1 N=34 | Marathi N=25
Rank | Team | Macro-F1 | Rank | Team | Macro-F1
1 | t1 | 0.7825 | 1 | WLV-RIT | 0.9144
2 | Super Mario | 0.7797 | 2 | neuro-utmn-thales | 0.8808
3 | Hasnuhana | 0.7797 | 3 | Hasnuhana | 0.8756
… |  |  | 4 | SATLab | 0.8749
6 | KuiYongyi | 0.7725 | 5 | PreCog IIIT | 0.8734
7 | SATLab | 0.7718 | … |  |
8 | neuro-utmn-thales | 0.7682 |  |  |
… |  |  |  |  |
Hindi-2 N=24 |  |  |
Rank | Team | Macro-F1 |  |  |
1 | NeuralSpace | 0.5603 |  |  |
2 | SATLab | 0.5586 |  |  |
3 | hate-busters | 0.5582 |  |  |
… |  |  |  |  |
Table 5: Transformed Macro-F1 for the five problems |  | Nbr. of problems the team participated in
---|---|---
Rank | Team | 5 | 4 | 3 | 2 | 1
1 | WLV-RIT |  |  |  |  | 1.0000
2 | NLP-CIC | 0.9814 |  |  |  |
3 | neuro-utmn-thales |  | 0.9800 |  |  |
4 | HASOC21rub |  |  |  | 0.9693 |
5 | NeuralSpace | 0.9666 |  |  |  |
6 | SATLab | 0.9601 |  |  |  |
7 | KuiYongyi | 0.9596 |  |  |  |
8 | CAROLL Passau |  |  |  |  | 0.9590
9 | IMS-SINAI |  |  |  |  | 0.9569
10 | hate-busters | 0.9517 |  |  |  |
… |  |  |  |  |  |
Number of Teams | 16 | 8 | 6 | 18 | 15
Table 6: Transformed Macro-F1 for the two English problems and for the Hindi and Marathi problems English 1 & 2 | Hindi 1 & 2 and Marathi
---|---
|  | #problems |  |  | #problems
Rk | Team | 2 | 1 | Rk | Team | 3 | 2 | 1
1 | NLP-CIC | 1.0000 |  | 1 | WLV-RIT |  |  | 1.0000
2 | neuro-utmn-thales | 0.9876 |  | 2 | SATLab | 0.9800 |  |
3 | HASOC21rub | 0.9693 |  | 3 | NeuralSpace | 0.9787 |  |
4 | HUNLP | 0.9675 |  | 4 | neuro-utmn-thales |  | 0.9725 |
5 | HNLP | 0.9674 |  | 5 | KuiYongyi | 0.9707 |  |
… |  |  |  | 6 | NLP-CIC | 0.9690 |  |
19 | hate-busters | 0.9331 |  | 7 | hate-busters | 0.9640 |  |
20 | SATLab | 0.9302 |  | 8 | CAROLL Passau |  |  | 0.9590
21 | TeamOulu |  | 0.9272 | 9 | BIU |  | 0.9484 |
… |  |  |  | … |  |  |  |
Number of Teams | 38 | 25 | Number of Teams | 17 | 13 | 8

###  3.2 Benchmarking the approach

The main objective of SATLab’s participation in HASOC 2021 was to propose a
competitive system relying only on the training data and employing only
classical supervised learning procedures. To determine whether this goal was
achieved, Tables 4-6 compare the performance of the approach to that of the
other participating systems.

Table 4 shows for each of the five problems the number of teams that
participated, the scores of the top three teams, the scores of the best SATLab
version, and the scores of the two contiguous teams. As it can be seen, it is
clearly in the two less resourced languages, Hindi and Marathi, that the
performance of the approach is among the best since it is even second, very
close to the first (and the third), in the Hindi-2 problem. In English on the
other hand, the system is ranked in the middle of the pack of average scores
at 0.048 and 0.054 of the best team.

The difference in performance between English and the other two languages is
particularly evident in Tables 5 and 6, which present the average scores of
the teams for the five problems (Table 5), the two problems in English and the
three problems in less endowed languages (Table 6). Before calculating these
averages, the scores for each problem were divided by the maximum score
obtained for the problem in question. This transformation111This
transformation of the scores considers that the minimum score in each task is
the same, 0, and that therefore no correction should be made at this level.
This seems to me justified by the fact that, even if it is unlikely, a system
can be wrong on all instances, but also and especially because it is the
deviation from the maximum score that is important. allows to give an
equivalent weight to the scores of all problems. It is then possible to
present in the same table, without distorting the results, all the teams,
whatever the number of problems they have participated in. Without this
transformation, the teams that participated in the most difficult tasks are
penalized compared to those that did not. In these tables, the number of
problems each team participated in is given by the column in which the score
is found and the total number of teams that participated in a given number of
problems is presented in the last row.

In terms of the overall average (Table 5), SATLab ranks sixth overall and
third among the 16 teams that participated in the five tasks. In English
(Table 6), on the other hand, it ranks only 20th. In the two less endowed
languages (Table 6), it is second, exceeded only by a team that participated
in only one of the five tasks.

##  4 Conclusion

A system, based exclusively on the character n-grams present in the posts to
be categorized, employing no additional linguistic resources and thus
completely language-agnostic, is proposed to automatically identify hate
speech and offensive content in social network posts. It relies on traditional
machine learning procedures such as logistic regression. Used in the HASOC
2021 challenge [Modha et al., 2021], it reached a medium performance level in
English, the language for which it was easy to develop deep learning
approaches relying on many external linguistic resources. Its performance,
averaged on the two Hindi problems and the Marathi problem, ranks it in first
place among the teams that proposed systems for at least two of these
problems. These performances suggest that it is an interesting reference level
to evaluate the benefits of using more complex approaches that are frequently
used to address this type of task such as deep learning or taking into account
complementary resources [Mandl et al., 2019, Mandl et al., 2020, Mandl et al.,
2021]. However, it is essential to note that the proposed system never ranked
first in any specific task. It is therefore clearly not the best performing
system for any of the five tasks.

## Acknowledgments

The author wishes to thank the organizers of this shared task for putting
together this valuable event and the reviewers for their very constructive
comments. He is a Research Associate of the Fonds de la Recherche Scientifique
- FNRS (Fédération Wallonie Bruxelles de Belgique). Computational resources
have been provided by the supercomputing facilities of the Université
Catholique de Louvain (CISM/UCL) and the Consortium des Equipements de Calcul
Intensif en Fédération Wallonie Bruxelles (CECI).

## References

  * [Bestgen, 2017]  Yves Bestgen.  2017\.  Improving the character ngram model for the DSL task with BM25 weighting and less frequently used feature sets.  In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 115–123, Valencia, Spain, April.
  * [Bestgen, 2021a]  Yves Bestgen.  2021a.  LAST at CMCL 2021 shared task: Predicting gaze data during reading with a gradient boosting decision tree approach.  In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 90–96, Online, June. Association for Computational Linguistics.
  * [Bestgen, 2021b]  Yves Bestgen.  2021b.  Optimizing a supervised classifier for a difficult language identification problem.  In Proceedings of the Eigth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 96–101, April.
  * [Fan et al., 2008]  Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin.  2008\.  LIBLINEAR: A library for large linear classification.  Journal of Machine Learning Research, 9:1871–1874.
  * [Gaikwad et al., 2021]  Saurabh Gaikwad, Tharindu Ranasinghe, Marcos Zampieri, and Christopher M Homan.  2021\.  Cross-lingual offensive language identification for low resource languages: The case of Marathi.  In Proceedings of RANLP.
  * [Haffari et al., 2018]  Reza Haffari, Colin Cherry, George Foster, Shahram Khadivi, and Bahar Salehi, editors.  2018\.  Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, Melbourne, July. Association for Computational Linguistics.
  * [Ke et al., 2017]  Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  2017\.  LightGBM: A highly efficient gradient boosting decision tree.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3146–3154. Curran Associates, Inc.
  * [Mandl et al., 2019]  Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandalia, and Aditya Patel.  2019\.  Overview of the HASOC track at FIRE 2019: Hate speech and offensive content identification in indo-european languages.  In Prasenjit Majumder, Mandar Mitra, Surupendu Gangopadhyay, and Parth Mehta, editors, FIRE ’19: Forum for Information Retrieval Evaluation, Kolkata, India, December, 2019, pages 14–17. ACM.
  * [Mandl et al., 2020]  Thomas Mandl, Sandip Modha, Anand Kumar M, and Bharathi Raja Chakravarthi.  2020\.  Overview of the HASOC track at FIRE 2020: Hate speech and offensive language identification in tamil, malayalam, hindi, english and german.  In Prasenjit Majumder, Mandar Mitra, Surupendu Gangopadhyay, and Parth Mehta, editors, FIRE 2020: Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, pages 29–32. ACM.
  * [Mandl et al., 2021]  Thomas Mandl, Sandip Modha, Gautam Kishore Shahi, Hiren Madhu, Shrey Satapara, Prasenjit Majumder, Johannes Schäfer, Tharindu Ranasinghe, Marcos Zampieri, Durgesh Nandini, and Amit Kumar Jaiswal.  2021\.  Overview of the HASOC subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages.  In Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation. CEUR, December.
  * [Manning et al., 2008]  Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze.  2008\.  An Introduction to Information Retrieval.  Cambridge University Press.
  * [Modha et al., 2021]  Sandip Modha, Thomas Mandl, Gautam Kishore Shahi, Hiren Madhu, Shrey Satapara, Tharindu Ranasinghe, and Marcos Zampieri.  2021\.  Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages and Conversational Hate Speech.  In FIRE 2021: Forum for Information Retrieval Evaluation, Virtual Event, 13th-17th December 2021. ACM, December.
  * [Mujadia et al., 2019]  Vandan Mujadia, Pruthwik Mishra, and Dipti Misra Sharma.  2019\.  Iiit-hyderabad at HASOC 2019: Hate speech detection.  In Parth Mehta, Paolo Rosso, Prasenjit Majumder, and Mandar Mitra, editors, Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings, pages 271–278. CEUR-WS.org.
  * [Ortega et al., 2021]  John Ortega, Atul Kr. Ojha, Katharina Kann, and Chao-Hong Liu.  2021\.  Proceedings of the 4th workshop on technologies for machine translation of low-resource languages: Introduction.  In Proceedings of Machine Translation Summit XVIII, pages I–VI.
  * [Robertson and Zaragoza, 2009]  Stephen Robertson and Hugo Zaragoza.  2009\.  The probabilistic relevance framework: BM25 and beyond.  Foundations and Trends in Information Retrieval, 3(4):333–389, April.
  * [Saroj et al., 2019]  Anita Saroj, Rajesh Kumar Mundotiya, and Sukomal Pal.  2019\.  Irlab@iitbhu at HASOC 2019: Traditional machine learning for hate speech and offensive content identification.  In Parth Mehta, Paolo Rosso, Prasenjit Majumder, and Mandar Mitra, editors, Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings, pages 308–314. CEUR-WS.org.


(a) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(b) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

Figure 20: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 15-layer
Sequential CNN trained on MNIST using independent Optimized Search. Acceptable
loss $\epsilon_{l,p}=0.3\%$.

(c) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(d) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

Figure 21: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 15-layer
Sequential CNN trained on SVHN using independent Optimized Search. Acceptable
loss $\epsilon_{l,p}=0.3\%$.

(a) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(b) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

Figure 22: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 15-layer
Sequential CNN trained on CIFAR10 using independent Optimized Search.
Acceptable loss $\epsilon_{l,p}=0.3\%$.

(c) 15-layer sequential - Fashion-MNIST

(d) 15-layer sequential - SVHN

(e) 15-layer sequential - CIFAR10

Figure 23: Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer of the
pre-trained 15-layer Sequential CNN trained on CIFAR10 found using independent
Optimized Search. Acceptable loss $\epsilon_{l,p}=0.3\%$.

(a) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(b) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

(c) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer

Figure 24: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 23-layer
Branched CNN trained on MNIST using independent Optimized Search. Acceptable
loss $\epsilon_{l,p}=0.3\%$.

(d) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(e) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

(f) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer

Figure 25: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 23-layer
Branched CNN trained on Fashion-MNIST using independent Optimized Search.
Acceptable loss $\epsilon_{l,p}=0.3\%$.

(a) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(b) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

(c) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer

Figure 26: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 23-layer
Branched CNN trained on SVHN using independent Optimized Search. Acceptable
loss $\epsilon_{l,p}=0.3\%$.

(d) Inference accuracy loss $\Delta a^{I}_{l,p}$ for parameters of each layer
after finding its optimal $(BW,F)_{l,p}$ using Independent Optimized Search
while keeping other parameters at full precision.

(e) Inference accuracy loss of the CNN $\Delta a$ every time a parameter of a
layer is quantized to the respective $(BW,F)^{*}_{l,p}$ found using
Independent Optimized Search. Parameters of the network are quantized
sequentially in the order of weights, biases followed by activations from
layers 1 to L.

(f) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer

Figure 27: Inference accuracy loss measured using two ways after quantizing
parameters to their fixed-point representations of the pre-trained 23-layer
Branched CNN trained on CIFAR10 using independent Optimized Search. Acceptable
loss $\epsilon_{l,p}=0.3\%$.

###  VII-C Additional results for dependent quantization

Fig. 29 \- Fig. 32 presents additional results for dependent quantization.
Work of Section VI-C.

(a) 15 layer Sequential Fashion-MNIST Weights

(b) 15 layer Sequential CIFAR10 Weights

(c) 15 layer Sequential SVHN Weights

Figure 28: Inference accuracy loss $\Delta a^{D}_{l,W}$ after quantizing
Weights of all layers to their optimal $(BW,F)^{*}_{l,W}$ found using
Dependent Optimized Search for different acceptable loss allocation schemes.
Weights of the preceding layer are also quantized to $(BW,F)^{*}_{k,W}$ for
$k<l$.

(d) 15 layer Sequential Fashion-MNIST Weights

(e) 15 layer Sequential CIFAR10 Weights

(f) 15 layer Sequential SVHN Weights

Figure 29: Optimal bitwidths $BW^{*}_{l,W}$ of Weights of each layer found
using Dependent Optimized Search for different acceptable loss allocation
schemes on a 15 layer Sequential CNN for the other three datasets. Weights of
the preceding layer are also quantized to $(BW,F)^{*}_{k,W}$ for $k<l$.

(a) 15 layer Sequential Fashion-MNIST Weights

(b) 15 layer Sequential CIFAR10 Weights

(c) 15 layer Sequential SVHN Weights

Figure 30: Total memory consumption of the quantized weights with optimal
bitwidths $BW^{*}_{l,W}$ in Fig. 29.

(a) 15 layer Sequential MNIST Activations

(b) 15 layer Sequential Fashion-MNIST Activations

(c) 15 layer Sequential CIFAR10 Activations

(d) 15 layer Sequential SVHN Activations

Figure 31: Inference accuracy loss $\Delta a^{D}_{l,A}$ after quantizing
activations of all layers to their optimal $(BW,F)^{*}_{l,A}$ found using
Dependent Optimized Search for different acceptable loss allocation schemes.
Activations of the preceding layer are also quantized to $(BW,F)^{*}_{k,A}$
for $k<l$.

(a) 15 layer Sequential MNIST Activations

(b) 15 layer Sequential Fashion-MNIST Activations

(c) 15 layer Sequential CIFAR10 Activations

(d) 15 layer Sequential SVHN Activations

Figure 32: Optimal bitwidths $BW^{*}_{l,A}$ of Activations of each layer found
using Dependent Optimized Search for different acceptable loss allocation
schemes on a 15 layer Sequential CNN for the other three datasets. Activations
of the preceding layer are also quantized to $(BW,F)^{*}_{k,A}$ for $k<l$.

###  VII-D Additional results for the final method

Fig. 34 \- Fig. 39 present additional results for our final method.

(a) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(b) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 33: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 15-layer
sequential CNN trained on Fashion-MNIST.

(c) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(d) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 34: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 15-layer
sequential CNN trained on SVHN.

(a) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(b) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 35: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 15-layer
sequential CNN trained on CIFAR10.

(c) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(d) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 36: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 23-layer branched
CNN trained on MNIST.

(a) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(b) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 37: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 23-layer branched
CNN trained on Fashion-MNIST.

(c) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(d) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 38: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 23-layer branched
CNN trained on SVHN.

(a) Acceptable and measured inference accuracy loss $\Delta a^{D}_{l,p}$ after
sequentially quantizing the parameters $\mathbf{p}_{l}$ in the order
($\mathbf{W}\rightarrow\mathbf{B}\rightarrow\mathbf{A}$) from layers 1 to L.

(b) Optimal bitwidths $BW^{*}_{l,p}$ of parameters of each layer.

Figure 39: Results for acceptable and measured inference accuracy loss and the
optimal bitwidths of the quantized CNN resulting from dependent optimized
search using a linear acceptable loss allocation scheme on a 23-layer branched
CNN trained on CIFAR10.

###  VII-E Architectures of Branched and Sequential Model

Visual illustrations of the architectures used for our experiments are shown
in Fig. 40

Figure 40: Architectures of the Branched model (left) and Sequential model
(right) tested on the four datasets


11institutetext: Lunit, Seoul 06241, Republic of Korea 11email:
kcholmin@gmail.com,cglee<EMAIL_ADDRESS>
akalsdnr3<EMAIL_ADDRESS>
http://lunit.io

# Variability Matters : Evaluating inter-rater variability in histopathology
for robust cell detection

Cholmin Kang Chunggi Lee 11 0000-0001-6321-0003 11 0000-0002-6164-2563 Heon
Song 11 0000-0003-3435-0271 Minuk Ma 11 0000-0003-0416-8479 Sérgio Pereira 11
0000-0002-4298-0903

###### Abstract

Large annotated datasets have been a key component in the success of deep
learning. However, annotating medical images is challenging as it requires
expertise and a large budget. In particular, annotating different types of
cells in histopathology suffer from high inter- and intra-rater variability
due to the ambiguity of the task. Under this setting, the relation between
annotators’ variability and model performance has received little attention.
We present a large-scale study on the variability of cell annotations among
120 board-certified pathologists and how it affects the performance of a deep
learning model. We propose a method to measure such variability, and by
excluding those annotators with low variability, we verify the trade-off
between the amount of data and its quality. We found that naively increasing
the data size at the expense of inter-rater variability does not necessarily
lead to better-performing models in cell detection. Instead, decreasing the
inter-rater variability with the expense of decreasing dataset size increased
the model performance. Furthermore, models trained from data annotated with
lower inter-labeler variability outperform those from higher inter-labeler
variability. These findings suggest that the evaluation of the annotators may
help tackle the fundamental budget issues in the histopathology domain

##  1 Introduction

Histopathology plays an important role in the diagnosis of cancer and its
treatment planning. The process, typically, requires pathologists to localize
cells and segment tissues in whole slide images (WSIs) [1]. However, with the
advent of digital pathology, these tedious and error-prone tasks can be done
by Deep Learning models. Still, these models are known to require the
collection of large annotated datasets. Particularly, cell detection needs
annotations from very large numbers of individually annotated cells.

The identification of cells in WSIs is challenging due to their diversity,
subtle morphological differences, and the astounding amount of cells that
exist in a single WSI. Therefore, when manually performed, this task is time-
consuming, and suffers from high inter- and intra-rater variability among
annotators [2, 3]. For example, Figure 1 shows the large discrepancy in cell
annotations among different annotators.

The inconsistencies among the annotators negatively affect the performance of
the model [4]. For example, when using such inconsistent data, the model may
overfit to the noise and generalize poorly [5]. Therefore, some degree of
consistency in the annotation must be guaranteed.

Figure 1: Example of inter-rater variability in the annotation of cells. The
colored circles represent: red \- tumor cell, blue \- lymphocytes, and green
\- other cells.

To improve the consistency among annotators and reduce the variability due to
annotations in the dataset, we propose to exclude annotators with large
variability from other annotators. This method is based on the existence of an
anchor annotator who works as a reference point to which each annotator are
compared. The concept of anchor has been recently proposed [6, 7], where a
cluster center is fixed for more efficient and stable training. We propose to
have an anchor annotator that annotated a control set. In this way, we can
measure the conformity of the other annotators in relation to the anchor one.
During annotation, a given expert can receive a new image to annotate, or,
instead, with some probability a control image. The set of annotated control
images of an annotator is used to measure its conformity to the anchor
annotator through a modified F1-score (mF1). In this way, annotators with high
conformity would achieve lower variability, thus, more reliable annotations in
relation to the anchor.

We collected and annotated a total of 29,387 patches from 12,326 WSIs. A total
of 120 board-certified pathologists participated in the annotation process,
which, to the best of our knowledge, is the largest-scale pathology annotation
analysis reported in the literature. Based on our conformity, we divide the
annotated data and perform an analysis on the discrepancy among annotators.
Our results show the conformity of 0.70, with a standard deviation of 0.08.
Furthermore, the kappa score showed a mean agreement of $\kappa$=0.43 with a
minimum value between $\kappa$=-0.01 and $\kappa$=0.87. These results show
relatively low conformity among most of the annotators. Furthermore, we also
verify that the data with lower variability leads to better-performing models.
Indeed, the model trained with a smaller dataset with lower variability showed
better performance when compared to the model trained with the full dataset.
This supports the hypothesis that a smaller higher quality dataset can lead to
a well-performing model, at a reduced cost, when compared with a large dataset
with high inter-rater variability.

The contributions of this paper are five-fold. 1) We propose a method for
calculating the conformity between annotators and an anchor annotator. In this
way, we can measure the variability among annotators in cell detection tasks
in histopathology. 2) We observe that the annotations created by pathologists
have high inter-rater variability based on our conformity method. 3) By
employing the before-mentioned conformity method, we propose a process to
reduce annotation variability within a dataset by excluding annotators with
low conformity in relation to an anchor. This leads to more efficient usage of
the annotation budget, without sacrificing performance. 4) We show that a
dataset with low inter-rater variability leads to better model performance. 5)
To the best of our knowledge, this is the first work that tackles the problem
of measuring variability among annotators in a large-scale dataset for a cell
detection task, with the participation of a large crowd of board-certified
pathologists.

##  2 Related work

Deep Learning techniques, particularly Convolutional Neural Networks (CNN)
revolutionized the field of computer vision and image recognition [8, 9] by
achieving unprecedented performances. As such, it has also been extensively
explored in the domain of digital pathology [10]. Cell detection tasks have
received attention due to its laborious and error-prone nature [11, 12, 13,
14]. However, there has been little attention to the effect of inter-rater
variability in cell detection tasks, and how it affects the performance of the
models.

In comparison to natural images, the annotation of medical images is a more
challenging task. First, domain experts are needed, e.g., medical doctors.
Second, due to the difficulty and ambiguity of the tasks, these tasks are more
prone to subjectivity which leads to the disagreement between experts [15,
16]. The inter-rater variability and reproducibility of annotations have been
studied in some medical-related tasks, such as sleep pattern segmentation in
EEG data [17], or segmentation in Computed Tomography images [18]. In the case
of histopathology, some studies demonstrate that computer-assisted reading of
images decreases the variability among observers [15, 16].

To overcome the disagreements, some works attempt to utilize the annotations
of multiple raters by learning several models from single annotations and from
the agreement of multiple annotators [19]. In this way, the effect of
disagreement can be mitigated, but it assumes multiple annotations for the
same images. Other approaches accept that inter-rater variability will exist,
and explore crowds of non-experts to collect large amounts of data by crowd-
sourcing, either with or without the assistance of experts. Nonetheless, the
scopes were limited to tasks that do not require a high degree of expertise or
the annotation of many instances [20, 21]. Such assumptions do not hold in
cell annotation in histopathology. Crowd-sourcing cell annotations were
previously performed [22], but, the evaluation of the conformity of the
annotators in this setting, where there is high inter-rater variability,
remains unexplored. Inconsistencies among the annotators introduce noise to
the annotations [4]. In turn, it can lead to models that overfit to the noise
and generalize poorly [5]. Therefore, some degree of label variability must be
guaranteed.

Other related prior work tackles the problem of lack of conformity between
annotators from the perspective of a noisy label [23, 24], or, tried to
classify the reliability of annotators using the probabilistic model[25].
Other works tried to resolve large variability of annotator performance using
the EM algorithm[26] or Gaussian process classifiers [27]. However, these
approaches rely on the training dynamics, so, it is not possible to
distinguish if a given sample is noisy or just difficult. Moreover, the noisy
data is discarded, raising two concerns: 1) resources are used to gather
unused data, and 2) the dataset needs to be sufficiently large to allow
discarding data without impacting the performance of the model. Instead, we
hypothesize that noise can be introduced by annotators with high variability
to other annotators, and preventive selection of conforming annotators leads
to higher quality annotations. This becomes even more necessary in large-scale
settings, where a large pool of annotators needs to be sourced externally.

##  3 Evaluating annotators and variability

During the annotation process, the annotators receive unlabeled patches to
annotate from the training set $\mathbf{T}$. Control set $\mathbf{C}$ refers
to the unlabeled dataset that will later be used to measure the variability
among annotators. Images from the control set $\mathbf{C}$ are randomly, and
blindly, assigned to the annotators. Since the annotators do not know when a
control image is provided, this ensures that the annotation skill showed in
$\mathbf{C}$ is similar to that of $\mathbf{T}$. Similar methods are used in
practice when evaluating crowd-sourced workers [28]. An anchor annotator is
chosen as the reference annotator to perform analysis on discrepancy (i.e.,
variability) among annotators. The annotations of each annotator are compared
with the annotations done by the anchor annotator. Conformity is measured
between each annotated cell of an annotator and the annotated cells of an
anchor annotator. The conformity is used to show the variability of annotators
and annotators with large conformity will have small variability among them.

Finally, annotators are divided into groups according to their conformity
score. The model is, then, trained on the data of each group and aggregation
of groups. The annotation, conformity measurement, and model development
process are depicted in Figure 2(a).

Figure 2: Evaluation of inter-rater variability. (a) The control set is
created to blindly evaluate the annotator conformity during the annotation
process. A score is computed per annotator, which is used to evaluate the
impact of their annotations in the cell detection task. (b) Annotator
selection method. Conformity is measured between each annotation done by an
annotator and anchor annotator. In the annotator selection phase, annotations
are divided into subgroups (such as $g_{1}$, $g_{2}$, or $g_{3}$ in the
figure) according to annotator conformity percentiles.

###  3.1 Anchor annotator

Figure 3:  A conceptual overview of anchor annotator selection. Anchor
annotator is chosen among annotators with the most amount of annotations done
in control set $C$. As shown in the figure, the patches annotated by A4 is the
superset of patches annotated by A1, A2, A3, and A5. Thus, A4 is chosen as an
anchor annotator.

In order to perform an analysis of the variability among annotators, reference
data should be set to measure how much each annotator is far from the
reference point. In this work, one annotator is defined as the anchor
annotator and is chosen as a criterion to evaluate variability among
annotators. The rationale for setting an anchor annotator and evaluating
variance is motivated by the concept of an anchor cluster center [6, 7], where
a cluster center is fixed for more efficient and stable training. The
conceptual overview of anchor annotator selection is shown in Figure 3. As
shown in the Figure, an anchor annotator is the one with most number of images
annotated so that the annotated images are the superset of the patches
annotated by other annotators.

###  3.2 Annotator variability and conformity calculation

Input: $\mathbf{A}$ and $\mathbf{C}$: sets of annotations
$\mathbf{A}_{n}^{k}=\left\\{\mathbf{a}_{n,1}^{k},\ldots\right\\}$ and
$\mathbf{C}_{n}^{k}=\left\\{\mathbf{c}_{n,1}^{k},\ldots\right\\}$ where
$\mathbf{a}_{n,p}^{k}$ and $\mathbf{c}_{n,q}^{k}$ indicates the $p$-th and
$q$-th annotations of the $n$-th control patch and class $k$

$\theta$: distance threshold

Output: $conf$: conformity

$\mathbf{s}\leftarrow\mathbf{0}$ ;

$\triangleright$ Initialize a vector indicating F1-score for each class

foreach  _class $k$_ do

$TP\leftarrow 0$, $FP\leftarrow 0$, and $FN\leftarrow 0$ ;

$\triangleright$ Initialize TP, FP, and FN

foreach  _$n$ -th control data patch_ do

$G_{A}\leftarrow\emptyset$ and $G_{M}\leftarrow\emptyset$ ;

$\triangleright$ Initialize assigned index sets

$D\in\mathbb{R}^{|\mathbf{A}_{n}^{k}|\times|\mathbf{C}_{n}^{k}|}\leftarrow\mathbf{0}$
;

$\triangleright$ Initialize the pairwise distance matrix

foreach  _index $p$ and $q$_ do

$D_{p,q}\leftarrow\|\mathbf{a}_{n,p}^{k}-\mathbf{c}_{n,q}^{k}\|_{2}$ ;

$\triangleright$ Compute the euclidean distance

end foreach

foreach  _$i$ -th index pair $(p^{*}_{i},q^{*}_{i})$ such that $\forall
j>i,D_{p^{*}_{i},q^{*}_{i}}\leq D_{p^{*}_{j},q^{*}_{j}}$_ do

if  _$D_{p^{*}_{i},q^{*}_{i}}\leq\theta$ , $p^{*}_{i}\notin G_{A}$, and
$q^{*}_{i}\notin G_{M}$_ then

$G_{A}\leftarrow G_{A}\cup\\{p^{*}_{i}\\}$ and $G_{M}\leftarrow
G_{M}\cup\\{q^{*}_{i}\\}$ ;

$\triangleright$ Assign indices

end if

end foreach

$TP\leftarrow TP+|G_{A}|$ ;

$\triangleright$ Update TP

$FP\leftarrow FP+|\mathbf{A}_{n}^{k}|-|G_{A}|$ ;

$\triangleright$ Update FP

$FN\leftarrow FN+|\mathbf{C}_{n}^{k}|-|G_{M}|$ ;

$\triangleright$ Update FN

end foreach

$s_{k}\leftarrow\frac{2TP}{2TP+FP+FN}$ ;

$\triangleright$ Compute the F1-score for class $k$

end foreach

$conf\leftarrow\frac{1}{|\mathbf{s}|}\sum_{k}s_{k}$ ;

$\triangleright$ Compute the conformity as mean F1-score

Algorithm 1 Conformity measurement algorithm for an individual annotator

Annotator conformity is calculated using algorithm 1. To compute the
annotators’ conformity, a hit criterion needs to be defined. We consider a
circular region of radius $\theta=10$ micrometers centered in each cell of
$\mathbf{C}$ in a given patch. Annotations from $\mathbf{C}$ that are located
inside a given circle are True Positive (TP) candidates. Among these
candidates, the closest point that matches the same class as the annotation
from $\mathbf{C}$ is considered a TP. Given this criterion, we also compute
False Positive (FP) and False Negative (FN) quantities. Based on the TP, FP,
and FN, F1-score per class is computed and the conformity of the annotator is
defined as mean F1-score over classes.

###  3.3 Annotator grouping based on the conformity

Finally, annotators are divided into subgroups according to their conformity
to the anchor annotator. Figure 2(b) shows the annotators’ annotation
distribution. First, an anchor annotator is selected. Then, conformity between
the anchor annotator and other annotators is calculated. Finally, each
annotator is evaluated by computing the conformity, as described in algorithm
1. In this work, we consider this conformity as a variability. If all
annotations done by all annotators are equal to those done by the anchor
annotator, the conformity will be 1. However, if the annotations are different
from the anchor annotator, the annotator conformity calculated by algorithm 1
will decrease, leading to increased variability and decreasing conformity.

##  4 Experimental Results

###  4.1 Dataset

Table 1: Description of the Control, Training, and Validation sets in terms of the number patches, annotated cells, and WSIs. Dataset | Patches | Annotated cells | Avg annotated cells per patch | Number of WSI
---|---|---|---|---
Control set | 150 | 32,746 | 218.31 | 141
Training set | 21,795 | 3,727,343 | 171.02 | 8,875
Validation set | 7,442 | 1,390,551 | 186.85 | 3,310
Figure 4:  Example of the reference images provided to annotators alongside
the annotation guidelines.

We collected a large-scale dataset for the task of tumor and lymphocyte cells
detection in H&E-stained WSIs, from 16 primary origins cancers (e.g. lung,
breast, kidney, etc). As WSIs are typically large for exhaustive annotation,
occupying several giga-pixels squared in area, it is a common practice to
divide them into smaller patches [29]. We sampled patches of size 1024
$\times$ 1024 pixels from a total of 12,326 WSIs. The annotations consist of
point annotations that locate the nuclei of cells, as well as their classes.
We collect 3 sets at the WSI level: control set ($\mathbf{C}$), Training set
($\mathbf{T}$) and Validation set ($\mathbf{V}$). The validation set is
annotated by three annotators, and has no overlap with $\mathbf{C}$. The
number of images and cell annotations in each set is presented in Table 1. A
total of 29,387 unique patches are annotated. As can be seen in the Table, the
average number of annotated cells in a single image is similar among $C$, $T$,
and $V$. A large crowd of 120 board-certified pathologists participated in the
annotation process. During the annotation process, annotators are provided
with guidelines on how to annotate each cell. Annotators are guided to
annotate each cell to one of six classes: Tumor Cell, Lymphoplasma cells,
Macrophage, Fibroblast, Endothelial cell, and others. The class ”other”
include nucleated cells with vague morphology that are not included in the
specific cell types. Each cell is explained with its definition and its
clinical characteristics. Furthermore, visual examples were provided to
exemplify the guides, such as in Figure 4.

###  4.2 Cell detection method and experimental details

Having the conformity scores of the annotators, we evaluate its efficacy in
filtering out annotations from annotators with higher variability by training
a model in the downstream task of cell detection. Similar to [30], we pose the
problem as a dense pixel classification task, but we use a DeepLabv3+ [31]
architecture. The point annotation of each cell is converted to a circle of
radius of 5 pixels, hence, resulting in a map similar to semantic segmentation
tasks. The output is a dense prediction likelihood omap that, similar to [30],
requires a post-processing stage to retrieve the unique locations of the
cells111Post-processing consists of Gaussian filtering ($\sigma=3$) the
likelihood maps, followed by local maxima detection within a radius of 3
pixels.. In this experiment, we target Tumor-Infiltrating Lymphocyte (TIL)
task which requires the number of tumor cells and lymphocytes in given WSI.
Therefore, among the annotations, we regard macrophage, fibroblast, and
endothelial cell as ”others” class. All models are trained using the Adam [32]
optimizer (learning rate of 1e-4) and the soft dice loss function [33].
Experiments are performed on 4 NVIDIA V100 GPUs, and implemented using PyTorch
(version 1.7.1) [34].

###  4.3 Conformity of the annotators

Figure 5: Distributions of the variability of the external annotators as
mF1-score (left), and the agreement of the annotations by the external
annotators in relation to the set $\mathbf{C}$, measured in terms of Fleiss’
kappa (right).

Following the proposed evaluation procedure of the annotators, we computed the
individual annotator conformity. In this work, an annotator with the most
number of annotations done from set $C$ is chosen as an anchor annotator so
that most annotators can have conformity calculated. The closer a score is to
1, the better the annotator conforms with an anchor annotator. Figure 5, left,
shows the distribution of the conformity score. The median conformity is 0.70
(minimum=0.38, and maximum=0.83). We further conduct a Shapiro-Wilk Normality
test [35] and verify that it does not follow a Normal distribution at a
significance level $\alpha=0.05$, with a p-value$=9.427e-8$. Moreover, we
observe a negative skewness of $-1.3158$, which confirms the observation of
the longer left tail.

Varying expertise is a known problem when annotating cells in histopathology.
Nonetheless, one would expect the conformity of the experts to reference
annotation to be normally distributed. Instead, we observe that while the
median annotator conformity is relatively high, there are some annotators on
the left tail that perform significantly worse than the majority. This
supports the observation of high inter-rater variability.

To further analyze the inter-rater variance, we measured the agreement among
all annotators on each Control patch using Fleiss’ kappa [36], which is a
statistical measure of the reliability of agreement for categorical ratings
with more than two raters. This test was run to show inter-rater variability
rather than to show absolute quality of annotators.

In this analysis, the inter-rater agreement is measured in relation to three
categories: tumor cells, lymphocytes, and unmatched annotations. As previously
described, the annotated cells from each pathologist have been matched to the
nearest cell of the same class in $\mathbf{C}$, following the hit criterion.
On the other hand, if the hit criterion is not satisfied, we treat it as an
unmatched annotation. We compute the Fleiss’ kappa for all patches in
$\mathbf{C}$. The mean agreement among the annotations of the pathologists was
$\kappa$=0.43 with the minimum and maximum of $\kappa$=-0.01 and
$\kappa$=0.87, respectively. This range (-0.01 to 0.87) of kappa values
suggest poor to an excellent agreement, but the mean is located in a fair to
moderate agreement range. This further reinforces the observation that there
exists high inter-rater variability. We observe that the classification of the
cells is sensitive in specific patches and pathologists, as shown in Figure 1.

###  4.4 Impact of the conformity of the annotators in cell detection

Table 2: Model performance by each cancer primary origin and annotation conformity percentile range. For example, the column “$p_{0-25}$” denotes the training set with patches annotated by annotators with conformity in percentiles 0 to 25, i.e., the 25% annotators with the lowest conformity. At the bottom, it is presented the total annotation time in hours. | Percentile ranges
---|---
Primary Origin |  $p_{0-25}$ |  $p_{75-100}$ |  $p_{0-50}$ |  $p_{50-100}$ |  $p_{25-100}$ |  $p_{0-100}$
Biliary Tract |  62.09 | 62.9 |  63.71 |  64.06 |  65.1 |  64.89
Breast |  62.32 | 64.74 |  64.53 |  65.53 |  64.83 |  65.3
Colorectum |  61.87 |  63.09 |  63.83 |  63.8 |  64.02 |  63.84
Esophagus |  64.11 |  61.42 |  65.56 |  66.24 |  66.33 |  66.19
Head and Neck |  59.07 |  61.68 |  61.76 |  62.54 |  62.63 |  62.61
Kidney |  49.68 |  53.94 |  52.10 |  56.07 |  57.34 |  56.49
Liver |  62.50 |  65.16 |  64.06 |  66.09 |  65.29 |  65.16
Lung |  58.18 |  60.24 |  60.27 |  61.24 |  60.75 |  60.72
Melanoma |  59.08 |  60.03 |  60.18 |  61.64 |  61.73 |  61.27
Ovary |  53.92 |  55.64 |  56.57 |  57.56 |  57.79 |  57.32
Pan Urinary |  62.71 |  64.82 |  65.33 |  65.45 |  65.02 |  64.8
Pancreas |  56.68 |  58.49 |  58.62 |  59.93 |  61.16 |  60.06
Prostate |  55.70 |  56.41 |  56.12 |  58.13 |  58.39 |  58.64
Stomach |  59.74 |  61.07 |  62.13 |  62.79 |  63.54 |  63.10
Uterine Cervix |  61.86 |  64.33 |  64.37 |  65.23 |  65.69 |  65.64
Uterine Endometrium |  53.18 |  54.70 |  54.13 |  56.07 |  55.53 |  55.51
Average mF1 |  58.92 |  60.77 |  60.87 |  62.03 |  62.18 |  61.76
Annotation time (hours) |  598 |  575 |  1,122 |  1,176 |  1,703 |  2,298

In this experiment, we divide the annotators by their conformity, from lower
to higher. Following this conformity, we define 4 subsets of the data based on
the percentile of the conformity of the annotators. Therefore, the interval
$p_{0-25}$ contains 25% of the patches in the dataset that were annotated by
annotators with high variability. The intervals $p_{25-50}$, $p_{50-75}$, and
$p_{75-100}$ also contain 25% of the patches each, with decreasing annotator
variability as compared to the anchor annotator. Therefore, following our
hypothesis, the interval $p_{75-100}$ must contain the annotations with the
lowest variability. We hypothesize that data annotated by annotators with
lower variability should lead to better-performing cell detection models.
Table 2 shows the performance of the cell detection model in terms of
conformity by the before-mentioned intervals of data across the 16 cancer
primary origin organs. The results are the average of 3 models trained with
different random seeds. We use the paired Wilcoxon Signed Rank test [37] at a
significance level $\alpha=0.05$ to compare results and test our hypothesis.

We compare the performance of models trained with similar amounts of data, but
with different conformity in relation to the annotation. First, we contrast
interval $p_{0-25}$ and $p_{75-100}$, i.e., the data annotated by the highest
variance and lowest variance annotators, respectively. Note that both sets
contain 25% of patches of the full training dataset. We observe that the data
from the best performing annotators (interval $p_{75-100}$) leads to better
performance compared to interval $p_{0-25}$. This is true for all of cancer
primary origins and is corroborated by an existing statistical difference
between the results (p-value=0.00614). Similarly, when we use 50% of the data
to train the cell detection model (intervals $p_{0-50}$ and $p_{50-100}$), a
similar trend is observed. The model trained from the data from annotators
with lower variability outperforms the model trained with data from interval
$p_{0-50}$; again, a statistical difference was found (p-value=0.00054). This
result shows that given the same amount of data, the model trained with high
conformity data outperforms that trained with low conformity data.

Additionally, we evaluate the performance of models trained from smaller
datasets labeled by annotators with lower variability compared to the use of
the full training set. The intervals $p_{25-100}$ and $p_{50-100}$ contain an
upper 75% and 50% of the training dataset, respectively. Nonetheless, in both
cases, they outperform the model trained with the full training set, with
statistical significance in the case of the interval $p_{25-100}$
(p-value=0.01778). These results suggest that smaller but low inter-rater
variability datasets can outperform models trained from the larger datasets,
but with higher variability annotations.

The data from the interval $p_{50-100}$ represent the upper 50% of the
available training data, but the performance is on par with the variability
setting of the interval $p_{25-100}$. Therefore, in retrospect, we could have
collected 50% fewer data, which would have reduced the total amount of
annotation time from 2298 hours to 1176 hours. This is especially significant
as the annotation process requires expert pathologists. Thus, we believe that
the proposed annotation scoring procedure can be helpful in using the
available budget more effectively, by requesting annotations from the experts
with low variability to anchor annotator and obtain either a smaller dataset
with good model performance or a large-scale dataset with lower inter-rater
variability annotations in the future data collection process.

##  5 Conclusions

In this paper, we proposed a simple, yet effective method to assess the
variability of a crowd of 120 experts in the time-consuming and difficult task
of cell annotation in histopathology images stained with H&E. Our findings
support the previous observations that this task suffers from high inter-rater
variability. Furthermore, we conducted a proof-of-concept experiment to show
the effect of inter-rater variability on a machine learning model for the task
of cell detection. From the experiment, we showed that with the same size of
data, the data labeled by annotators with lower inter-rater variability led to
better model performance than the data labeled by annotators with higher
inter-rater variability. Furthermore, excluding the data with the most inter-
rater variability from the full dataset was beneficial, despite resulting in a
smaller dataset. Hence, contrary to common findings in deep learning research,
the fewer amount of data with lower inter-rater variability resulted in
better-performing machine learning model. These findings are useful in the
medical domain where annotations are costly and laborious. With this finding,
we can argue that collecting fewer data from low inter-rater variability is
more beneficial than collecting data without considering the inter-rater
variability. Despite the findings that increasing conformity leads to
increasing model performance, we recognize that there are some limitations in
our work. First, our work is based on the premise that the anchor annotator is
well-performing. The work is based on the assumption that all annotators are
sufficiently expert and have done their best to annotate. Further
investigation is needed with iterated experiments on setting annotators as
anchor annotators. Second, the generalizability of our work on stained WSI
other than H&E stain is not guaranteed. Further investigation on other
staining methods with more diverse skill backgrounds of digital pathologist
annotators is needed. We hope the idea of scoring the annotators helps improve
the annotation budget management in the histopathology domain.

## References

  * [1] Diao, J.A., Wang, J.K., Chui, W.F., Mountain, V., Gullapally, S.C., Srinivasan, R., Mitchell, R.N., Glass, B., Hoffman, S., Rao, S.K., et al.:  Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes.  Nature communications 12(1) (2021) 1–15
  * [2] Lutnick, B., Ginley, B., Govind, D., McGarry, S.D., LaViolette, P.S., Yacoub, R., Jain, S., Tomaszewski, J.E., Jen, K.Y., Sarder, P.:  An integrated iterative annotation technique for easing neural network training in medical image analysis.  Nature machine intelligence 1(2) (2019) 112–119
  * [3] Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A.W.M., de Bruijne, M.:  Early experiences with crowdsourcing airway annotations in chest CT.  In: MICCAI workshop. (2016)
  * [4] Guo, B., Chen, H., Yu, Z., Xie, X., Huangfu, S., Zhang, D.:  Fliermeet: a mobile crowdsensing system for cross-space public information reposting, tagging, and sharing.  Transactions on Mobile Computing 14(10) (2014) 2020–2033
  * [5] Li, J., Wong, Y., Zhao, Q., Kankanhalli, M.S.:  Learning to learn from noisy labeled data.  In: CVPR. (2019)
  * [6] Wang, Y., Wang, D., Pang, W., Miao, C., Tan, A.H., Zhou, Y.:  A systematic density-based clustering method using anchor points.  Neurocomputing 400 (2020) 352–370
  * [7] Miller, D., Sunderhauf, N., Milford, M., Dayoub, F.:  Class anchor clustering: A loss for distance-based open set recognition.  In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. (2021) 3570–3578
  * [8] Krizhevsky, A., Sutskever, I., Hinton, G.E.:  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25 (2012)
  * [9] Girshick, R., Donahue, J., Darrell, T., Malik, J.:  Rich feature hierarchies for accurate object detection and semantic segmentation.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2014) 580–587
  * [10] Meijering, E.:  Cell segmentation: 50 years down the road [life sciences].  IEEE Signal Processing Magazine 29(5) (2012) 140–145
  * [11] Chen, T., Chefd’Hotel, C.:  Deep learning based automatic immune cell detection for immunohistochemistry images.  In: International workshop on machine learning in medical imaging, Springer (2014) 17–24
  * [12] Gao, Z., Wang, L., Zhou, L., Zhang, J.:  Hep-2 cell image classification with deep convolutional neural networks.  IEEE journal of biomedical and health informatics 21(2) (2016) 416–428
  * [13] Cireşan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.:  Mitosis detection in breast cancer histology images with deep neural networks.  In: International conference on medical image computing and computer-assisted intervention, Springer (2013) 411–418
  * [14] Xue, Y., Ray, N.:  Cell detection in microscopy images with deep convolutional neural network and compressed sensing.  arXiv preprint arXiv:1708.03307 (2017)
  * [15] Hekler, A., Utikal, J.S., Enk, A.H., Solass, W., Schmitt, M., Klode, J., Schadendorf, D., Sondermann, W., Franklin, C., Bestvater, F., et al.:  Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images.  European Journal of Cancer 118 (2019) 91–96
  * [16] Bertram, C.A., Aubreville, M., Donovan, T.A., Bartel, A., Wilm, F., Marzahl, C., Assenmacher, C.A., Becker, K., Bennett, M., Corner, S., et al.:  Computer-assisted mitotic count using a deep learning–based algorithm improves interobserver reproducibility and accuracy.  Veterinary pathology 59(2) (2022) 211–226
  * [17] Schaekermann, M., Beaton, G., Habib, M., Lim, A., Larson, K., Law, E.:  Understanding expert disagreement in medical data analysis through structured adjudication.  Proceedings of the ACM on Human-Computer Interaction 3(CSCW) (2019) 1–23
  * [18] Haarburger, C., Müller-Franzes, G., Weninger, L., Kuhl, C., Truhn, D., Merhof, D.:  Radiomics feature reproducibility under inter-rater variability in segmentations of ct images.  Scientific reports 10(1) (2020) 1–10
  * [19] Sudre, C.H., Anson, B.G., Ingala, S., Lane, C.D., Jimenez, D., Haider, L., Varsavsky, T., Tanno, R., Smith, L., Ourselin, S., et al.:  Let’s agree to disagree: Learning highly debatable multirater labelling.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2019) 665–673
  * [20] Maier-Hein, L., Mersmann, S., Kondermann, D., Bodenstedt, S., Sanchez, A., Stock, C., Kenngott, H.G., Eisenmann, M., Speidel, S.:  Can masses of non-experts train highly accurate image classifiers? - A crowdsourcing approach to instrument segmentation in laparoscopic images.  In: MICCAI. (2014)
  * [21] Kwitt, R., Hegenbart, S., Rasiwasia, N., Vécsei, A., Uhl, A.:  Do we need annotation experts? A case study in celiac disease classification.  In: MICCAI. (2014)
  * [22] Amgad, M., Atteya, L.A., Hussein, H., Mohammed, K.H., Hafiz, E., Elsebaie, M.A., Alhusseiny, A.M., AlMoslemany, M.A., Elmatboly, A.M., Pappalardo, P.A., et al.:  Nucls: A scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentation.  arXiv:2102.09099 (2021)
  * [23] Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., Belongie, S.:  Learning from noisy large-scale datasets with minimal supervision.  In: CVPR. (2017)
  * [24] Jiang, L., Zhou, Z., Leung, T., Li, L.J., Fei-Fei, L.:  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In: ICML. (2018)
  * [25] Yan, Y., Rosales, R., Fung, G., Subramanian, R., Dy, J.:  Learning from multiple annotators with varying expertise.  Machine learning 95(3) (2014) 291–327
  * [26] Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.:  Learning from crowds.  Journal of machine learning research 11(4) (2010)
  * [27] Rodrigues, F., Pereira, F., Ribeiro, B.:  Gaussian process classification and active learning with multiple annotators.  In: International conference on machine learning, PMLR (2014) 433–441
  * [28] Bartolo, M., Thrush, T., Riedel, S., Stenetorp, P., Jia, R., Kiela, D.:  Models in the loop: Aiding crowdworkers with generative annotation assistants.  arXiv:2112.09062 (2021)
  * [29] Khened, M., Kori, A., Rajkumar, H., Krishnamurthi, G., Srinivasan, B.:  A generalized deep learning framework for whole-slide image segmentation and analysis.  Scientific reports 11(1) (2021) 1–14
  * [30] Swiderska-Chadaj, Z., Pinckaers, H., van Rijthoven, M., Balkenhol, M., Melnikova, M., Geessink, O., Manson, Q., Sherman, M., Polonia, A., Parry, J., et al.:  Learning to detect lymphocytes in immunohistochemistry with deep learning.  MIA 58 (2019) 101547
  * [31] Chen, L., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.:  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In: ECCV. (2018)
  * [32] Kingma, D.P., Ba, J.:  Adam: A method for stochastic optimization.  In Bengio, Y., LeCun, Y., eds.: ICLR. (2015)
  * [33] Milletari, F., Navab, N., Ahmadi, S.A.:  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In: 3DV. (2016)
  * [34] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.:  Automatic differentiation in pytorch.  (2017)
  * [35] SHAPIRO, S.S., WILK, M.B.:  An analysis of variance test for normality (complete samples)†.  Biometrika 52(3-4) (12 1965) 591–611
  * [36] Falotico, R., Quatto, P.:  Fleiss’ kappa statistic without paradoxes.  Quality & Quantity 49(2) (2015) 463–470
  * [37] Woolson, R.F.:  Wilcoxon signed-rank test.  Wiley encyclopedia of clinical trials (2007) 1–3