# Optimizing small BERTs trained for German NER

Jochen Zöllner  University of Rostock
Institute of Mathematics
<EMAIL_ADDRESS>Konrad Sperfeld  University of Rostock
Institute of Mathematics
<EMAIL_ADDRESS>Christoph Wick  Planet AI GmbH Rostock
<EMAIL_ADDRESS>Roger Labahn  University of Rostock
Institute of Mathematics
<EMAIL_ADDRESS>
###### Abstract

Currently, the most widespread neural network architecture for training
language models is the so called BERT which led to improvements in various
Natural Language Processing (NLP) tasks. In general, the larger the number of
parameters in a BERT model, the better the results obtained in these NLP
tasks. Unfortunately, the memory consumption and the training duration
drastically increases with the size of these models, though. In this article,
we investigate various training techniques of smaller BERT models and evaluate
them on five public German Named Entity Recognition (NER) tasks of which two
are introduced by this article. We combine different methods from other BERT
variants like ALBERT, RoBERTa, and relative positional encoding. In addition,
we propose two new fine-tuning techniques leading to better performance:
Class-Start-End-tagging and a modified form of Linear Chain s. Furthermore, we
introduce a new technique called Whole-Word Attention which reduces BERTs
memory usage and leads to a small increase in performance.

##  1 Introduction

NER is a well-known task in the field of NLP. The
NEISS111https://www.neiss.uni-rostock.de project in which we work in close
cooperation with Germanists is devoted to the automation of diverse processes
during the creation of digital editions. One key task in this area is the
automatic detection of entities in text corpora which corresponds to a common
NER task. Based on this work we prepared two new German NER datasets that were
created from existing digital editions whose entities were previously tagged
manually.

In recent years, the best results for NER tasks have been achieved with
Transformer-based Vaswani et al. (2017) language models such as Bidirectional
Encoder Representations from Transformers (BERT) Devlin et al. (2019).
Classically, a BERT is first pre-trained with large amounts of unlabeled text
to obtain a robust language model and then fine-tuned to a downstream task.
Especially for the pre-training step, many variants of BERT like ALBERT Lan et
al. (2020), RoBERTa Liu et al. (2019), or XLNet Yang et al. (2019) were
already investigated. Pre-training is resource-intensive and takes much time
(several weeks) for training. For that reason online-platforms such as Hugging
Face222https://huggingface.co/ offer a zoo of already pre-trained networks
that can directly be used to train a downstream task. However, the available
models are not always suitable for a certain task such as NER in German,
because they can be pre-trained on a different domain (e.g. language, time
epoch, or text style). Therefore, in this article, we examine which techniques
are optimal for pre-training and fine-tuning of a BERT to solve NER tasks in
German. We investigate this on smaller BERT models with six layers that can be
pre-trained on a single GPU (RTX 2080 Ti) within two weeks while fine-tuning
can be performed on a notebook CPU in a few hours.

When philologists create new digital editions, different research priorities
can be set, so that a different associated NER task is created each time. That
is why philologists must also be able to train individual NER tasks themselves
who commonly only have access to limited compute resources. For this reason,
the aim is to train NER tasks on smaller BERT models as best as possible.

In our experiments, we evaluated the BERT models on five different German NER
tasks. Two of these tasks rely on new NER datasets which we generated from
digital text editions. We first compared different well-established pre-
training techniques such as Mask Language Model (MLM), Sentence Order
Prediction (SOP), and Next Sentence Prediction (NSP) on the final result of
the downstream NER task. Furthermore, we investigated the influence of
absolute and relative positional encoding, as well as Whole-Word Masking
(WWM).

As a second step, we compared various approaches for carrying out fine-tuning,
since the tagging rules cannot be learned consistently by classical fine-
tuning approaches. In addition to existing approaches such as the use of
Linear Chain s (LCRFs), we propose the so-called Class-Start-End (CSE)-tagging
and an especially modified form of Linear Chain s for NER which led to an
increased performance. Furthermore, for decoding, we introduced a simple rule-
based approach, which we call Entity-Fix rule, to further improve the results.

As already mentioned, the training of a BERT requires much resources. One of
the reasons is that the memory amount of BERT depends quadratically on the
sequence length when calculating the energy values or the attention scores in
its attention layers which leads to memory problems for long sequences. In
this article, we propose Whole-Word Attention (WWA), a new modification of the
Transformer architecture that not only reduces the number of energy values to
be calculated by about factor 2, but also results in slightly improved
results.

In summary, the main goal of this article is to enable the training of
efficient BERT models for German NER on limited resources. For this, the
article provides different methodology and claims the following contributions:

  * •

We introduce and share two new datasets for German NER formed from existing
digital editions.

  * •

We investigate the influence of different BERT pre-training methods, such as
pre-training tasks, varying positional encoding, and adding WWM on a total of
five different NER datasets.

  * •

On the same NER tasks, we investigate different approaches to perform fine-
tuning. Hereby, we propose two new methods which led to performance
improvements: CSE-tagging and a modified form of LCRFs.

  * •

We introduce a novel rule-based decoding strategy achieving further
improvements.

  * •

We propose WWA, a modification of the BERT architecture that reduces the
memory requirements of the BERT models, especially for processing long
sequences, and also leads to further performance improvements.

  * •

We share the datasets (see Section 2) and our source
code333https://github.com/NEISSproject/tf2_neiss_nlp/tree/berNer21 which is
based on Wick et al. (2021) with the community

The remainder of this article is structured as follows: In Section 2 we
present our datasets including the two new German NER datasets. In Section 3
we introduce the different pre-training techniques, while Section 4 describes
fine-tuning. In both sections we provide an overview of the existing
techniques which we adopted and also introduce our novel methods.
Subsequently, in Section 5, we introduce WWA. After that, Section 6 shows the
conducted experiments and their results. We conclude this article by
discussing our results and giving an outlook on future work.

##  2 Datasets

In this section, we list the different datasets. First, we describe the
dataset used for pre-training throughout our experiments. Then, we introduce
five NER datasets for the downstream tasks.

###  2.1 Pre-training Data

Large amounts of unlabeled text is necessary as input data for pre-training a
BERT. We collected the German Wikipedia and a web crawl of various German
newspaper portals to pre-train our BERT. The dump of the German Wikipedia was
preprocessed by the Wiki-Extractor of Attardi (2015) resulting in about 6 GB
of text data. In addition, we took another 2 GB of German text data from
different newspaper portals444We used various German newspaper portals like
https://www.faz.net/aktuell/ or https://www.berliner-zeitung.de/, on August
2020 crawled with the news-please framework of Hamborg et al. (2017) .

###  2.2 NER Datasets

We evaluated our methods on five different NER tasks. In addition to three
already existing German NER datasets, the frequently used GermEval 2014
dataset and two NER datasets on German legal texts, we introduce two new NER
tasks of two existing digital Editions. In the following, we describe each of
the five tasks.

#### GermEval 2014

One of the most widespread German NER datasets is GermEval 2014 Benikova et
al. (2014) which comprises several News Corpora and Wikipedia. In total, it
contains about 590,000 tokens which are tagged into four main entity classes:
“person”, “organisation”, “location”, and “other”. Each main class can appear
in a default, a partial, or a derived variant, resulting in 12 overall
classes. In the GermEval task, entities can be tagged in the two levels outer
and inner (nested entities). Since there are few inner annotations in the
dataset, we restrict ourselves to evaluating the outer entities in our
experiments as it is often the approach in other papers. This is called the
outer chunk evaluation scheme which is described in more detail by Riedl and
Padó (2018).

#### Legal Entity Recognition

The Legal Entity Recognition (LER) dataset Leitner, Rehm, and Moreno-Schneider
(2020) contains 2.15 million tokens with 54,000 manually annotated entities
from German court decision documents of 2017 and 2018. The entities are
divided into seven main classes and 19 subclasses which we label by Coarse-
Grained (CG) and Fine-Grained (FG), respectively. The FG task (LER FG) is more
difficult than the CG task (LER CG) due to its larger number of possible
classes.

#### Digital Edition: Essays from H. Arendt

We created a novel NER dataset from the digital edition "Sechs Essays" by H.
Arendt. It consists of 23 documents from the period 1932-1976 which are
published online by Hahn, Breysach, and Pischel (2020) as TEI files (see TEI-
Consortium, 2017). In these documents, certain entities were manually tagged.
Since some of the original NER tags comprised too few examples and some
ambiguities (e.g. place and country), we joined several tags as shown in Table
1.

Table 1: Distribution of NER entities in H. Arendt Edition. Column “Original attributes” lists which attributes from the original TEI files were combined into one “Entity” for the NER dataset. On average, an entity consists of 1.36 words. Entity | # All | # Train | # Test | # Devel |  Original attributes
---|---|---|---|---|---
person | 1,702 | 1,303 | 182 | 217 |  person, biblicalFigure, ficticiousPerson, deity, mythologicalFigure
place | 1,087 | 891 | 111 | 85 |  place, country
ethnicity | 1,093 | 867 | 115 | 111 |  ethnicity
organisation | 455 | 377 | 39 | 39 |  organisation
event | 57 | 49 | 6 | 2 |  event
language | 20 | 14 | 4 | 2 |  language
unlabeled words | 153,223 | 121,154 | 16,101 | 15,968 |

Note, that we removed any annotation of the class “ship” since only four
instances were available in the dataset and no other similar class is
available. We provide the resulting dataset
online555https://github.com/NEISSproject/NERDatasets/tree/main/Arendt in a
format similar to the CONLL-X format (see Buchholz and Marsi, 2006) and in a
simple JSON format under a CC BY-NC-SA 3.0 DE license together with the
training, development, and test partition. Since not all entities are equally
distributed over the 23 documents, the sentences of all documents are shuffled
before splitting them into partitions.

#### Digital Edition: Sturm Edition

The second new NER dataset consists of 174 letters of the years 1914-1922 from
the Sturm Edition T. Schrade (2018) available online in TEI format. It is much
simpler than the dataset from the H. Arendt edition and contains only persons,
places, and dates as tagged entities. From the original TEI files, we built an
NER dataset with tags distributed as shown in Table 2. Similarly to the H.
Arendt dataset, the resulting dataset is available
online666https://github.com/NEISSproject/NERDatasets/tree/main/Sturm in a
format similar to the CONLL-X format and in a simple JSON format under a CC-BY
4.0 license together with the training, development, and test partition. In
contrast to the H. Arendt dataset we split the 174 letters without shuffling
the sentences across all documents.

Table 2: Distribution of NER entities in the Sturm Edition. On average, an entity consists of 1.12 words. Entity | # All | # Train | # Test | # Devel
---|---|---|---|---
pers | 930 | 763 | 83 | 84
date | 722 | 612 | 59 | 51
place | 492 | 374 | 59 | 59
unlabeled words | 33,809 | 27,047 | 3,306 | 3,456

##  3 Pre-training Techniques

In this section, we provide an overview of several common pre-training
techniques for a BERT which we examined in our experiments.

###  3.1 Pre-training Tasks

In the original BERT Devlin et al. (2019), pre-training is performed by
simultaneously minimising the loss of the so called Mask Language Model (MLM)
and the Next Sentence Prediction (NSP) task. The MLM task first tokenizes the
text input with a subword tokenizer, then 15% of the tokens are masked
randomly. Hereby, 80% of these masked tokens are replaced by a special mask
token, 10% are replaced by a randomly chosen other token, and the remaining
10% keep the original correct token. Therefore, the goal of the MLM task is to
find the original token for the 15% randomly masked tokens which is only
possible by understanding the language and thus learning a robust language
model.

Since BERT should also be able to learn the semantics of different sentences
within a text, Devlin et al. (2019) also included NSP. When combining NSP with
MLM, the input for pre-training are two masked sentences which are
concatenated, separated by a special separator token. In 50% of the cases, two
consecutive sentences from the same text document are used whereas in the
other 50% two random sentences from different documents are selected. The goal
of the NSP task is to identify which of the two variants it is.

In the follow-up papers RoBERTa Liu et al. (2019) and XLNet Yang et al.
(2019), experiments showed that the NSP task often had no positive effect on
the performance of the downstream tasks. Therefore, both papers recommended
that the pre-training should solely be performed by the MLM task. In the
ALBERT paper, Lan et al. (2020) investigated this in more detail. They assumed
that the ineffectiveness of the NSP task was only due to its simplicity which
is why they introduced Sentence Order Prediction (SOP) as a more challenging
task that aims to learn relationships between sentences similar to the NSP
task: BERT always receives two consecutive sentences, but in 50% of the cases
the order is wrong by flipping them. The SOP task is to learn the correct
order of the sentences.

In this article, we examine the influences of the different pre-training tasks
(MLM, NSP, SOP) with the focus on improving the training of BERT for the
German language.

###  3.2 Absolute and Relative Positional Encoding

The original Transformer architecture of Vaswani et al. (2017) was based
exclusively on attention mechanisms to process input sequences. Attention
mechanisms allow every sequence element to learn relations to all other
elements. By default, Attention does not take into account information about
the order of the elements in the sequence. But since information about the
order of the input sequence elements is mandatory in almost every NLP tasks,
the original Transformer architecture introduced the so-called absolute
positional encoding: a fixed position vector $p_{j}\in\mathbb{R}^{d_{model}}$
was added to each embedded input sequence element $x_{j}$ at position
$j\in\left\\{1,\ldots,n\right\\}$ for an input sequence of length $n$, thus

$x_{j}^{\prime}=x_{j}+p_{j}.$

In the original approach the position vector $p_{j}$ is built by computing
sinusoids of different wavelength in the following way:

$\displaystyle p_{j,2k}$ $\displaystyle:=$
$\displaystyle\sin\left(j/10000^{2k/d_{model}}\right),$ $\displaystyle
p_{j,2k+1}$ $\displaystyle:=$
$\displaystyle\cos\left(j/10000^{2k/d_{model}}\right)$

where $k\in\left\\{1,\ldots,\lfloor\frac{d_{model}}{2}\rfloor\right\\}$. While
the experiments of Vaswani et al. (2017) showed great results, the
disadvantage of absolute positional encoding is that the performance is
significantly reduced in cases where the models are applied on sequences
longer than those on which they were trained because the respective position
vectors were not yet seen during training. Therefore, Rosendahl et al. (2019)
investigated and compared other variants for positional encoding on
translation tasks. The most promising approach was relative positional
encoding Shaw, Uszkoreit, and Vaswani (2019): a trainable distance information
$d_{j-i}^{K}$ is added in the attention layer when computing the energy
$e_{i,j}$ of the $i$th sequence element to the $j$th one. Thus, if $x_{i}$ and
$x_{j}$ are the $i$th and $j$th input elements of a sequence in an attention
layer, instead of multiplying just the query vector $W^{Q}x_{i}$ with the key
vector $W^{K}x_{j}$, one adds the trainable distance information $d_{j-i}^{K}$
to the key vector resulting in

$e_{i,j}:=\frac{\left(W^{Q}x_{i}\right)^{T}\left(W^{K}x_{j}+d_{j-i}^{K}\right)}{\sqrt{d_{k}}}$
(1)

where $W_{n}^{Q},W_{n}^{K}\in\mathbb{R}^{d_{model}\times d_{k}}$. In addition,
when multiplying the energy (after applying softmax) with the values, another
trainable distance information $d_{j-i}^{V}$ is added. Finally, the output
$y_{i}$ for the $i$th sequence element of a sequence of length $n$ with
relative positional encoding is computed by

$y_{i}=\sum\limits_{j=1}^{n}\alpha_{i,j}\left(W^{V}x_{j}+d_{j-i}^{V}\right)$
(2)

where
$\alpha_{i,j}=\frac{\exp\left(e_{i,j}\right)}{\sum\limits_{k=1}^{n}\exp\left(e_{i,k}\right)}$
and $W^{V}\in\mathbb{R}^{d_{model}\times d_{v}}$. To train $d_{j-i}^{K}$ and
$d_{j-i}^{V}$, a hyperparameter $\tau$ (called the clipping distance), the
trainable embeddings $r_{-\tau}^{K},\ldots,r_{\tau}^{K}\in\mathbb{R}^{d_{k}}$,
and $r_{-\tau}^{V},\ldots,r_{\tau}^{V}\in\mathbb{R}^{d_{v}}$ are introduced.
These embeddings are used to define the distance terms $d_{j-i}^{K}$ and
$d_{j-i}^{V}$, where distances longer than the clipping distance $\tau$ are
represented by $r_{\tau}$ or $r_{-\tau}$, thus:

$\displaystyle d_{j^{\prime}-j}^{K}$ $\displaystyle=$ $\displaystyle
r_{\textrm{clip}_{\tau}(j^{\prime}-j)}^{K}$ (3) $\displaystyle
d_{j^{\prime}-j}^{V}$ $\displaystyle=$ $\displaystyle
r_{\textrm{clip}_{\tau}(j^{\prime}-j)}^{V}$ (4)
$\displaystyle\textrm{clip}_{\tau}(x)$ $\displaystyle=$
$\displaystyle\max\left(-\tau,\min\left(\tau,x\right)\right)$

Rosendahl et al. (2019) already showed that relative positional encoding
suffers less from the disadvantages of absolute position encoding of unseen
sequence lengths. In this article, we examine the influence of these two
variants of positional encoding during the training of German BERT models.

###  3.3  Whole-Word Masking (WWM)

Whole-Word Masking (WWM) is a small modification of the MLM task described in
section 3.1. In contrast to the classic MLM task, WWM does not mask token-
wisely but instead word-wisely. This means that only either all tokens
belonging to a word are masked or none of the tokens. Recent work of Chan,
Schweter, and Möller (2020) and Cui et al. (2019) already showed the positive
effect of WWM in pre-training on the performance of the downstream task. In
this article, we also examine the differences between the original MLM task
and the MLM task with WWM.

##  4 Fine-tuning Techniques for NER

The task of NER is to detect entities, such as persons or places, which
possibly consist of several words within a text. As proposed by Devlin et al.
(2019), the traditional approach for fine-tuning a BERT to a classification
task like NER is to attach an additional feed-forward layer to a pre-trained
BERT which predicts token-wise labels. In order to preserve and obtain
information about the grouping of tokens into entities, Inside-Outside-
Beginning (IOB) tagging Ramshaw and Marcus (1999) is usually applied. IOB
tagging introduces two versions of each entity class $e$, one marking the
beginning of the entity and one representing the interior of an entity, and
one “other” class, which all together results in a total of $\gamma=2e+1$
classes. Table 3 shows an example in which the beginning token of an entity is
prefixed with a “B-” and all other tokens with an “I-”.

Table 3:  IOB tagging example with unlabeled words (O) and the two entities: “location” (Loc) and “person” (Per). The first tag of each entity is prefixed with “B-”, while all following tokens of that entity are marked with an “I-”. The first row are the words of the sentence which are split into one or more tokens (second row). The third row shows the tagged tokens based on the given entities (last row). The example sentence can be translated as “Peter lives in Frankfurt am Main”.  Words | Peter | lebt | in | Frankfurt | am | Main
---|---|---|---|---|---|---
Tokens | Peter | lebt | in | Frank | _furt | am | Main
Tagged Tokens | B-Per | O | O | B-Loc | I-Loc | I-Loc | I-Loc
Entities | Person |  |  | Location

In compliance with the standard evaluation scheme of NER tasks by Sang and De
Meulder (2003), we compute an entity-wise $F_{1}$ score denoted by E-$F_{1}$.
Instead of computing a token- or word-wise $F_{1}$ score, E-$F_{1}$ evaluates
a complete entity as true positive only if all tokens belonging to the entity
are correct. Our implementation of E-$F_{1}$ relies on the widely used Python
library _seqeval_ Nakayama (2018).

Usually, IOB tagging is trained by a token-wise softmax cross-entropy loss.
However, this setup of one feed-forward layer and a cross-entropy loss does
not take into account the context of the tokens forming an entity. This can
lead to inconsistent tagging, for example, an inner tag may only be preceded
by an inner or beginning tag of the same entity, and thus results in a
devastating impact on the E-$F_{1}$-score. Therefore, we propose and compare
three modified strategies that include context to prevent inconsistent NER
tagging during training or decoding. The first approach uses LCRFs, the second
proposal is a modification of the IOB tagging, the last attempt applies rules
to fix a predicted tagging. In the following, the default approach of fine-
tuning the BERT is called Simple Fine-tuning (SFT).

Most papers on BERT models dealing with German NER, for example Chan,
Schweter, and Möller (2020) or Labusch, Neudecker, and Zellhöfer (2019), do
not focus on an investigation of different variants for fine-tuning. However,
there are already studies for NER tasks in other languages (e.g. Luoma and
Pyysalo, 2020; Souza, Nogueira, and Lotufo, 2020) which show that the
application of LCRFs can be beneficial for fine-tuning. Souza, Nogueira, and
Lotufo (2020) also investigated whether it is advantageous for the fine-tuning
of BERT models on NER tasks to link the pre-trained BERT models with LSTM
layers. However, these experiments did not prove to be successful.

###  4.1 Fine-tuning with CSE tagging

In this section, we propose an alternative to the IOB tagging which we call
Class-Start-End (CSE) tagging. The main idea is to split the task into three
objectives as shown in Table 4: finding start and end tokens, and learning the
correct class.

Table 4:  CSE tagging example. Rows refer to the tokens and its respective target for Start, End, and Class.  Tokens | Peter | lebt | in | Frank | _furt | am | Main
---|---|---|---|---|---|---|---
Start | 1 | 0 | 0 | 1 | 0 | 0 | 0
End | 1 | 0 | 0 | 0 | 0 | 0 | 1
Class | Per | O | O | Loc | Loc | Loc | Loc

CSE appends two additional dense layers with logistic-sigmoid activation to
the last BERT layer with scalar outputs, one for the start $p^{\text{start}}$,
and one for the end $p^{\text{end}}$ token. In summary, the complete output
for an input sample consisting of $n$ tokens is
$\left(\left(p^{\text{start}}_{1},p^{\text{end}}_{1},y_{1}\right),\left(p^{\text{start}}_{2},p^{\text{end}}_{2},y_{2}\right),\dots,\left(p^{\text{start}}_{n},p^{\text{end}}_{n},y_{n}\right)\right)\in\mathbb{R}^{n\times(2+e)}$
where $e$ is the number of possible entities.

The objective for $y_{i}$ is trained with softmax cross entropy as before but
without the distinction between B- and I-, while the start and end vectors
contribute extra losses $J^{\text{start}}$ and $J^{\text{end}}$:

$J^{\text{start}}=-\sum_{j=1}^{n}\bigl{[}t^{\text{start}}_{j}\cdot\log{(p^{\text{start}}_{j})}+(1-t^{\text{start}}_{j})\cdot\log{(1-p^{\text{start}}_{j})}\bigr{]}\;,$
(5)

where $t^{\text{start}}$ and $p^{\text{start}}$ are the target and prediction
vectors for start as shown in Table 4. $J^{\text{end}}$ is defined analog.

Converting the CSE into IOB tagging is realized by accepting tokens which
exceeds the threshold of 0.5 as start or end markers. If an end marker is
missing between two start markers, the position of the highest end probability
between the two locations is used as an additional end marker. This approach
is applied analogue in reverse for missing start markers. Finally, all class
probabilities between each start and end marker pairs (including start and
end) is averaged to obtain the entity class. In conclusion, an inconsistent
tagging is impossible.

###  4.2 Fine-tuning with Linear Chain  with NER-Rule
($\mathrm{LCRF}_{\mathrm{NER}}$)

Another approach to tackle inconsistent IOB tagging during fine-tuning of a
BERT is based on Linear Chain s (LCRFs) which are a modification of
Conditional Random Fields (CRFs), both proposed by Lafferty, McCallum, and
Pereira (2001). LCRFs are a common approach to train neural networks that
model a sequential task and are therefore well suited for fine-tuning NER. The
basic idea is to take into account the classification of the neighboring
sequence members when classifying an element of a sequence.

The output $Y=(y_{1},y_{2},\ldots,y_{n})\in\mathbb{R}^{n\times\gamma}$ of our
neural network for the NER task consists of a sequence of $n$ vectors whose
dimension corresponds to the number of classes $\gamma\in\mathbb{N}$. LCRF
introduce so-called transition values $\mathfrak{T}$ which are a matrix
$W^{\mathfrak{T}}$ of trainable weights, in the basic approach:
$\mathfrak{T}:=W^{\mathfrak{T}}\in\mathbb{R}^{\gamma\times\gamma}$. An entry
$\mathfrak{T}_{i,j}$ of this matrix $\mathfrak{T}$ can be seen as the
potential that a tag of class $i$ is followed by a tag of class $j$. In one of
the easiest forms of LCRFs which we choose, decoding aims to find the sequence
$C^{p}:=\left\\{c^{p}_{1},c^{p}_{2},\ldots,c^{p}_{n}\right\\}\in\left\\{1,2,\ldots,\gamma\right\\}^{n}$
with the highest sum of corresponding transition values and elements of the
corresponding output vectors as shown in eq. (6).

$C^{p}:=\operatorname*{arg\,max}_{C\in\\{1,\ldots,\gamma\\}^{n}}\left(\sum\limits_{j=1}^{n}y_{j,c_{j}}+\sum\limits_{j=1}^{n-1}\mathfrak{T}_{c_{j},c_{j+1}}\right)$
(6)

Eq. (6) is efficiently solved by the Viterbi-Algorithm (see, e.g., Sutton and
McCallum, 2010). During training, a log-likelihood loss is calculated that
takes into account the transition values $\mathfrak{T}$ and the network output
$Y$. Sutton and McCallum (2010) provide a detailed description for its
implementation.

Since the IOB tagging does not allow all possible transitions, at first, we
tried to simply ban these forbidden transitions completely by assigning fixed
non-trainable high negative values to the associated entries in
$\mathfrak{T}$. However, this did not lead to any improvement in performance.
For this reason we extend the upper default LCRF approach by explicitly
modeling these constraints by adding additional trainable weights to the model
when computing the transition values $\mathfrak{T}$. In the following, we call
our adapted algorithm $\mathrm{LCRF}_{\mathrm{NER}}$.

Assume an NER task comprises the set of entities
$\text{X}_{1},\text{X}_{2},\ldots,\text{X}_{e}$ which results in $\gamma=2e+1$
classes following the IOB tagging scheme. Thus, beside a label $O$ for
unlabeled elements, for each entity $\text{X}_{i}$ there is a begin label
$\text{B-X}_{i}$ and an inner label $\text{I-}X_{i}$. For simplicity, we order
these classes by
$\text{B-X}_{1}$,$\ldots$,$\text{B-X}_{e}$,$\text{I-X}_{1}$,$\ldots$,$\text{I-X}_{e}$,$O$,
that is:

$\text{Class $i$ belongs to label}\begin{cases}\text{B-X}_{i}&\text{if $i\leq
e$}\\\ \text{I-X}_{i-e}&\text{if $e<i\leq 2e$}\\\
O&\text{otherwise.}\end{cases}$

With respect to this ordering, we introduce the matrix
$\mathfrak{F}\in\left\\{0,1\right\\}^{\gamma\times\gamma}$ as

$\mathfrak{F}_{i,j}=\begin{cases}1&\text{if $e<j\leq 2e$ and $i\neq j$ and
$i\neq j-e$}\\\ 0&\text{otherwise.}\end{cases}$

Thus, an element $\mathfrak{F}_{i,j}$ is $1$, if and only if a tag of class
$j$ can not follow on a tag of class $i$ in the given NER task. This maps the
constraint that the label of the predecessor of an interior tag of label I-X
can only be the same interior label I-X or the corresponding begin label B-X.
Likewise, we define the matrix
$\mathfrak{A}\in\left\\{0,1\right\\}^{\gamma\times\gamma}$ by
$\mathfrak{A}_{i,j}=1-\mathfrak{F}_{i,j}$ as the matrix of all allowed tag
transitions. $\mathrm{LCRF}_{\mathrm{NER}}$ introduces two additional
trainable weights
$\omega^{\mathfrak{F}}_{\mathrm{factor}},\omega^{\mathfrak{F}}_{\mathrm{absolute}}\in\mathbb{R}$
besides the weights $W^{\mathfrak{T}}$ and constructs $\mathfrak{T}$ by

$\mathfrak{T}:=(\mathfrak{A}+\omega^{\mathfrak{F}}_{\mathrm{factor}}\mathfrak{F})\odot
W^{\mathfrak{T}}-\omega^{\mathfrak{F}}_{\mathrm{absolute}}\mathfrak{F},$ (7)

where $\odot$ is the point-wise product. If setting
$\omega^{\mathfrak{F}}_{\mathrm{factor}}=1$ and
$\omega^{\mathfrak{F}}_{\mathrm{absolute}}=0$ this defaults to the original
LCRF approach. In this way, the model can learn an absolute penalty by
$\omega^{\mathfrak{F}}_{\mathrm{absolute}}$ and a relative penalty by
$\omega^{\mathfrak{F}}_{\mathrm{factor}}$ for forbidden transitions. Note,
that $\mathrm{LCRF}_{\mathrm{NER}}$ is mathematically equivalent to LCRF, the
only purpose is to simplify and to stabilize the training.

###  4.3 Decoding with Entity-Fix Rule

Finally, we propose a rule-based approach to resolve inconsistent IOB tagging
which can for example occur if an I-X tag is subsequent to a token that is not
I-X or B-X (for any possible entity class X). Our so-called Entity-Fix rule
replaces forbidden I-X tags with the tag of the previous token. If the
previous token has a B-X tag, the inserted token is converted to the
corresponding I-X tag. In the special case where an I-X tag is predicted at
the start of the sequence, it is converted to B-X of the same class. See Table
5 for an example. The advantage of this approach is that it can be applied as
a post-processing step independent of training. Furthermore, since only tokens
which already form an incorrect entity are affected by this rule, the
E-$F_{1}$ score can never decrease by applying it. Note that this does not
necessarily hold for the token-wise $F_{1}$ score, though.

Table 5: Example for Entity-Fix rule. Rows refer to the tokens, its respective target, prediction, and the prediction resulting from decoding with Entity-Fix rule. Changes are emphasized in bold. Tokens | Peter | lebt | in | Frank | _furt | am | Main
---|---|---|---|---|---|---|---
Target | B-Per | O | O | B-Loc | I-Loc | I-Loc | I-Loc
Prediction | I-Per | O | O | B-Loc | I-Org | I-Org | I-Loc
Prediction with Fix-Rule | B-Per | O | O | B-Loc | I-Loc | I-Loc | I-Loc

##  5 BERT Architecture with Whole-Word Attention (WWA)

In this section, we describe our proposed word-wise attention layers used by
some of our BERT models during pre-training and fine-tuning. This Whole-Word
Attention (WWA) was inspired by the benefits of the WWM. It comprises two
components: the first one called $\mathrm{mha}_{\mathrm{wwa}}$ applies
traditional multi-head attention on words instead of tokens, while the second
component is a windowed attention module called
$\mathrm{mha}_{\mathrm{wind}}$.

#### Traditional Approach

In opposite to current NLP network architectures, previous approaches for
tokenizing text (e.g. Mikolov et al., 2013) did not apply a tokenizer to break
down each word of a sentence into possibly more than one token. Instead, they
trained representations for a fixed vocabulary of words. The major drawback
was that this required a large vocabulary and out-of-vocabulary words could
not be represented. Modern approaches like Transformers tokenize words by a
vocabulary of subwords which allows to compose unknown words by known tokens.
However, in Transformer models, attention is computed between pairs of tokens.
As a consequence the number of energy values (see eq. (1)) to be calculated
increases quadratically with sequence length resulting in a large increase of
memory and computation time for long sequences.

There exist different approaches to tackle this problem. The most prominent
ones are BigBird of Zaheer et al. (2020) and Longformer of Beltagy, Peters,
and Cohan (2020). In their work, the focus is on pure sparse attention
strategies: Instead of a full attention, they try to omit as many calculations
of energy values as possible, so that as little performance as possible is
lost. Instead, we propose to rejoin tokens into word-based tokens which also
has a quadratic dependence on the sequence length but by a lower slope.

#### Our Methodology

The purpose of the first module, $\mathrm{mha}_{\mathrm{wwa}}$, is to map
tokens back to words and then to compute a word-wise attention. However, since
$\mathrm{mha}_{\mathrm{wwa}}$ loses information about the order of tokens, we
introduce $\mathrm{mha}_{\mathrm{wind}}$ as additional component which acts on
the original tokens. $\mathrm{mha}_{\mathrm{wind}}$ scales linearly with the
sequence length since only a window of tokens is taken into account when
computing the energy vectors. In summary, $\mathrm{mha}_{\mathrm{wwa}}$ learns
the global coarser dependence of words whereas $\mathrm{mha}_{\mathrm{wind}}$
allows to resolve and learn relations of tokens but only in a limited range.
In the following, we first describe $\mathrm{mha}_{\mathrm{wwa}}$ and then
$\mathrm{mha}_{\mathrm{wind}}$.

Let $T$ denote the input of our BERT model which is a part of text and can
thus be seen as a sequence of words $T=(w_{1},w_{2},\ldots,w_{m})$ with
$m\in\mathbb{N}$. Similar to a classical BERT, a tokenizer $\mathcal{T}$
Transforms $T$ into a sequence of tokens
$\mathcal{T}(T)=:\theta=(\theta_{1},\theta_{2},\ldots,\theta_{n})\in\mathbb{N}^{n}$
with $m\leq n$ because we only consider traditional tokenizers that encode the
text word-wisely by decomposing a word into one or more tokens. Such a
tokenizer provides a mapping function
$F_{T,\mathcal{T}}:\left\\{1,2,\ldots,n\right\\}\rightarrow\left\\{1,2,\ldots,m\right\\}$
which uniquely maps an index $i$ of the token sequence $\theta$ to the index
$j$ of its respective word $w_{j}$.

Each encoder layer $\ell$ of the classical BERT Transformer architecture
contains a multi-head attention layer $\mathrm{mha}^{\ell}$ which maps its
input sequence
$X_{T,\mathcal{T}}^{\ell}=(x_{1}^{\ell},x_{2}^{\ell},\ldots,x_{n}^{\ell})\in\mathbb{R}^{n\times
d}$ to an output $Y_{T,\mathcal{T}}^{\ell}$ of equal length $n$ and dimension
$d$:

$\mathrm{mha}^{\ell}(X_{T,\mathcal{T}}^{\ell})=Y_{T,\mathcal{T}}^{\ell}=(y_{1}^{\ell},y_{2}^{\ell},\ldots,y_{n}^{\ell})\in\mathbb{R}^{n\times
d},$

where the $i$th output vector $y^{\ell}_{i}$ is defined as the concatenation
of the resulting vectors for every attention head computed by equation (2).
Our $\mathrm{mha}_{\mathrm{wwa}}$ layer modifies this by applying attention
only on the sequence
$\hat{X}_{T,\mathcal{T}}^{\ell}=(\hat{x}^{\ell}_{1},\hat{x}^{\ell}_{2},\ldots,\hat{x}^{\ell}_{m})\in\mathbb{R}^{m\times
d}$, where

$\hat{x}_{j}:=\frac{1}{\left|\left\\{i:F_{T,\mathcal{T}}(i)=j\right\\}\right|}\sum\limits_{i:F_{T,\mathcal{T}}(i)=j}x_{i}$
(8)

and $\left\\{i:F_{T,\mathcal{T}}(i)=j\right\\}$ is the set of all tokens $i$
belonging to the word $j$. In other words, we average the corresponding token
input vectors for each word. Next, we apply $\mathrm{mha}$ on
$\hat{X}_{T,\mathcal{T}}^{\ell}$ yielding the output

$\mathrm{mha}^{\ell}(\hat{X}_{T,\mathcal{T}}^{\ell})=:\hat{Y}_{T,\mathcal{T}}^{\ell}=(\hat{y}_{1}^{\ell},\hat{y}_{2}^{\ell},\ldots,\hat{y}_{m}^{\ell})\in\mathbb{R}^{m\times
d}$

which is a sequence of length $m$ only. Finally, to again obtain a sequence of
length $n$, we transform the output sequence back to the length $n$ by
repeating the output vector for each word according to the number of
associated tokens. Thus, the final output of a layer
$\mathrm{mha}_{\mathrm{wwa}}^{\ell}$ is defined as

$\mathrm{mha}_{\mathrm{wwa}}^{\ell}(X_{T,\mathcal{T}}^{\ell}):=Z_{T,\mathcal{T}}^{\ell}=(z_{1}^{\ell},z_{2}^{\ell},\ldots,z_{n}^{\ell})\in\mathbb{R}^{n\times
d}$

where $z_{i}^{\ell}:=\hat{y}_{F_{T,\mathcal{T}}(i)}^{\ell}$.

We perform positional encoding by utilizing relative positional encoding
because absolute positional encoding adds the positional vectors to the token
sequence directly after the embedding. This is not compatible with WWA of eq.
(8). Instead, relative positional encoding adds a word-wise relative
positional encoding to the vectors of the word-wise sequence within
$\mathrm{mha}^{\ell}(\hat{X}_{T,\mathcal{T}}^{\ell})$.

As an experiment, we also pre-trained a BERT which solely uses
$\mathrm{mha}_{\mathrm{wwa}}$ layers for attention. However, it was already
apparent in the pre-training that it could only achieve a very low MLM
accuracy. The main reason for this is that $Z_{T,\mathcal{T}}^{\ell}$ does not
take into account any information about the position of the tokens within a
word since the output elements of them are equal which is why these tokens are
no longer related to each other via attention. To tackle this problem, for
each $\ell$, we introduce a second multi-head attention layer
$\mathrm{mha}_{\mathrm{wind}}^{\ell}$ based on windowed attention as used by
Beltagy, Peters, and Cohan (2020) or Zaheer et al. (2020). Because the sole
purpose of $\mathrm{mha}_{\mathrm{wind}}^{\ell}$ is to map the relationships
and positions of the tokens within a word in the model, we use a very small
sliding window size of $\omega=5$ tokens in each direction. Hence, in contrast
to Beltagy, Peters, and Cohan (2020) or Zaheer et al. (2020), we do also not
arrange our input sequence into blocks or chunks.

Formally, we define $\mathrm{mha}_{\mathrm{wind}}$ as

$\mathrm{mha}_{\mathrm{wind}}^{\ell}(X_{T,\mathcal{T}}^{\ell}):=Y_{T,\mathcal{T}}^{\ell}=(y_{1}^{\ell},y_{2}^{\ell},\ldots,y_{n}^{\ell})\in\mathbb{R}^{n\times
d}$

where the $i$th output vector $y^{\ell}_{i}$ is the concatenation of the
resulting vectors $y^{\ell,h}_{i}$ for every attention head $h$. Each
$y^{\ell,h}_{i}$ is given by

$y^{\ell,h}_{i}=\sum\limits_{j=i-\omega}^{i+\omega}\alpha_{i,j}^{h}\left(W^{V,h}x_{j}+d_{j-i}^{V,h}\right)$

where
$\alpha_{i,j}^{h}=\frac{exp\left(e_{i,j}^{h}\right)}{\sum\limits_{k=1}^{n}exp\left(e_{i,k}^{h}\right)}$,$W^{V,h}\in\mathbb{R}^{d_{model}\times
d_{v}}$. The energy values $e_{i,j}^{h}$ for each attention head $h$ are
defined as in equation (1) and the distance vectors
$d_{j-i}^{V,h},d_{j-i}^{K,h}$ are given by the equations (3) and (4).

In summary, in our BERT model using WWA, the total output of the $\ell$-th
attention layer of the $\ell$-th encoder layer is, after adding the input
sequence $X_{T,\mathcal{T}}^{\ell}$ as residual,

$X_{T,\mathcal{T}}^{\ell}+\mathrm{mha}_{\mathrm{wwa}}^{\ell}(X_{T,\mathcal{T}}^{\ell})+\mathrm{mha}_{\mathrm{wind}}^{\ell}(X_{T,\mathcal{T}}^{\ell})$

compared to the original approach

$X_{T,\mathcal{T}}^{\ell}+\mathrm{mha}^{\ell}(X_{T,\mathcal{T}}^{\ell}).$

$\mathrm{mha}_{\mathrm{wind}}$ introduces additional trainable variables
compared to the traditional Transformer architecture. However, the number of
energy values increases only linearly compared to the sequence length with
respect to the window size $\omega$ by a factor $2\omega+1$. Hence, the impact
on the memory can be neglected for long sequences and small $\omega$. In order
to quantify the overall reduction of memory consumption using WWA, we provide
an example using our tokenizer777We use a tokenizer with a vocabulary of about
30,000 as common in most BERT models. built on the German Wikipedia which
transforms on average a sequence of $m$ words in $n\approx
1.5m\approx\frac{3}{2}m$ tokens. Therefore, while the traditional multi-head
attention layer calculates $n^{2}$ energy values per each head, our WWA
approach only requires

$\overbrace{m^{2}}^{\mathrm{mha}_{\mathrm{wwa}}}+\overbrace{(2\omega+1)n}^{\mathrm{mha}_{\mathrm{wind}}}\approx\frac{4}{9}n^{2}+(2\omega+1)n$
(9)

energy values. Therefore, for large $n$ (and small $\omega$), the number of
energy values to be calculated is more than halved.

##  6 Experiments

To evaluate our proposed methods for German NER tasks, we conducted several
experiments. First, we compared the pre-training variants presented in Section
3 and then fine-tuning techniques presented in Section 4. Afterwards, we
applied WWA on the overall training of the NER task. Finally, we discuss our
results in relation to the state-of-the-art models of the LER and GermEval
tasks.

###  6.1 Comparing Pre-training Techniques

In our first experiments, we investigated which of the known pre-training
techniques (see Section 3) yields the best models for a subsequent fine-tuning
on German NER tasks. For this purpose, we combined the three presented pre-
training tasks (MLM, MLM with SOP, and MLM with NSP) with relative and
absolute positional encoding and optionally enabled WWM. For each resulting
combination, we pre-trained a (small) BERT with a hidden size of 512 with 8
attention heads on 6 layers for 500 epochs and 100,000 samples per epoch. Pre-
training was performed with a batch size of 48 and a maximal sequence length
of 320 tokens which is limited by the 11 GB memory of one GPU (RTX 2080 Ti).
Each of the 12 resulting BERTs was then fine-tuned using the SFT on the five
German NER tasks described in Section 2.2. For fine-tuning, we chose a batch
size of 16 and trained by default for 30 epochs and 5,000 samples per epoch.
The number of epochs was increased to 50 for the larger LER tasks.

Table 6:  Average E-$F_{1}$ score (cf. Sang and De Meulder, 2003) of three
fine-tuning runs with SFT for five datasets. Rows refer to the respective pre-
training task, absolute or relative positional encoding (PE), and use of WWM;
best results are emphasized.

Pre-training task |  PE | WWM | GermEval | H. Arendt | Sturm | LER CG | LER FG | Average
---|---|---|---|---|---|---|---|---
MLM |  abs. | - | 0.7785 | 0.7600 | 0.8236 | 0.9061 | 0.8969 | 0.8330
MLM |  abs. | ✓ | 0.7901 | 0.7681 | 0.8102 | 0.9192 | 0.9028 | 0.8381
MLM |  rel. | - | 0.7852 | 0.7674 | 0.8011 | 0.9198 | 0.9144 | 0.8376
MLM |  rel. | ✓ | 0.8086 | 0.7741 | 0.8555 | 0.9347 | 0.9206 | 0.8587
MLM, NSP |  abs. | - | 0.7609 | 0.7688 | 0.8085 | 0.9067 | 0.8928 | 0.8275
MLM, NSP |  abs. | ✓ | 0.7484 | 0.7621 | 0.8110 | 0.9047 | 0.8930 | 0.8238
MLM, NSP |  rel. | - | 0.7802 | 0.7471 | 0.8172 | 0.9193 | 0.9105 | 0.8348
MLM, NSP |  rel. | ✓ | 0.7750 | 0.7564 | 0.8191 | 0.9168 | 0.9044 | 0.8343
MLM, SOP |  abs. | - | 0.7530 | 0.7669 | 0.7942 | 0.8979 | 0.8817 | 0.8188
MLM, SOP |  abs. | ✓ | 0.7355 | 0.7590 | 0.7995 | 0.9047 | 0.8950 | 0.8187
MLM, SOP |  rel. | - | 0.7745 | 0.7548 | 0.8052 | 0.9176 | 0.9040 | 0.8312
MLM, SOP |  rel. | ✓ | 0.7863 | 0.7807 | 0.8550 | 0.9246 | 0.9122 | 0.8518

The results in Table 6 reveal that the best performing BERTs for NER tasks
used relative positional encoding, WWM, and were solely pre-trained with MLM.
This is plausible since the samples, the academical NER datasets consists of,
are only single sentences and hence using additional sentence-spanning losses
during pre-training (SOP or NSP) is even harmful for these downstream tasks.

Furthermore, our experiments show that relative positional encoding yields
significantly better results than absolute positional encoding. This is an
interesting observation since the experiments from Rosendahl et al. (2019)
stated that relative positional encoding only performs better when applied to
sequences with length longer than those on which the network was previously
trained. To investigate this in detail, we performed an analysis of the
E-$F_{1}$ score in dependence of the sequence length. We sorted the samples in
the test list for each dataset by token length in increasing order and then
split them into seven parts of equal number of samples. The left chart of
Figure 1 shows the averaged E-$F_{1}$ score of the five datasets for each
part. We observe that relative encoding (rel) is outperforming absolute
encoding (abs) for almost any sequence length. For long sequences (last part),
the gap between the two approaches increases which is expected. The right
chart of Figure 1 shows the E-$F_{1}$ score for each part of only the H.
Arendt dataset. For this dataset, the discrepancy between relative and
positional encoding is very small which shows that the benefit of relative
positional encoding highly depends on the dataset. Nevertheless, on average,
our experiments suggest to use relative positional as the mean of choice for
German NER tasks.

Figure 1: E-$F_{1}$ score for a BERT trained on MLM with WWM for relative and
absolute positional encoding fine-tuned with SFT tested on seven parts of the
test list, which are sorted by token length, starting with short sequences.
Left shows the mean over all datasets, Right shows the results on one dataset
(H. Arendt).

Furthermore, we observe that WWM together with relative positional encoding
led to significant improvements when combined with MLM with or without SOP as
pre-training task. In summary, our best setup combined solely MLM, relative
positional encoding, and WWM.

###  6.2 Comparing Fine-tuning Techniques

In this section, we examine our four different variants of fine-tuning: SFT,
CSE (see Section 4.1), and LCRF or $\text{\acs{LCRF}}_{\text{\acs{NER}}}$ (see
Section 4.2). The new fine-tuning techniques CSE and
$\text{\acs{LCRF}}_{\text{\acs{NER}}}$ were specifically designed to help the
network learn the structure of IOB tagging.

First, we evaluated the impact of the fine-tuning method in dependence of the
pre-training techniques examined in Section 6.1. For this purpose, we fine-
tuned all BERT models in analogy to Section 6.1 on all five NER tasks using
the four fine-tuning methods mentioned. The individual results shown in Table
7 are averaged across the five tasks whereby the outcome of each task is the
mean of three runs.

Table 7: Average E-$F_{1}$ score of all five NER tasks and three fine-tuning runs. Rows refer to the respective pre-training task, absolute or relative positional encoding (PE), and use of WWM; Columns refer to the fine-tuning method; best results per column are emphasized. Pre-training task |  PE | WWM | Avg. SFT | Avg. CSE | Avg. LCRF | Avg. $\text{LCRF}_{\text{NER}}$
---|---|---|---|---|---|---
MLM |  abs. | - | 0.8330 | 0.8608 | 0.8455 | 0.8469
MLM |  abs. | ✓ | 0.8381 | 0.8635 | 0.8443 | 0.8516
MLM |  rel. | - | 0.8376 | 0.8682 | 0.8477 | 0.8539
MLM |  rel. | ✓ | 0.8587 | 0.8785 | 0.8623 | 0.8651
MLM, NSP |  abs. | - | 0.8275 | 0.8536 | 0.8346 | 0.8418
MLM, NSP |  abs. | ✓ | 0.8238 | 0.8543 | 0.8358 | 0.8416
MLM, NSP |  rel. | - | 0.8348 | 0.8639 | 0.8457 | 0.8471
MLM, NSP |  rel. | ✓ | 0.8343 | 0.8595 | 0.8417 | 0.8459
MLM, SOP |  abs. | - | 0.8188 | 0.8515 | 0.8292 | 0.8325
MLM, SOP |  abs. | ✓ | 0.8187 | 0.8597 | 0.8312 | 0.8369
MLM, SOP |  rel. | - | 0.8312 | 0.8598 | 0.8450 | 0.8473
MLM, SOP |  rel. | ✓ | 0.8518 | 0.8714 | 0.8557 | 0.8605
Overall Average | 0.8340 | 0.8608 | 0.8432 | 0.8476

Regardless of the choice of the fine-tuning method, the results confirm that
pre-trained BERTs that were only pre-trained with MLM, use relative Positional
Encoding, and use WWM are the most suitable for German NER tasks. Therefore,
we will only consider this fine-tuning setup in the following.

Furthermore, the results show that $\text{LCRF}_{\text{NER}}$ consistently
outperforms LCRF. We think that the introduction of the new weights
$\omega^{\mathfrak{F}}_{\mathrm{factor}},\omega^{\mathfrak{F}}_{\mathrm{absolute}}$
(see eq. 7) enables the fine-tuning with $\text{LCRF}_{\text{NER}}$ to
outperform LCRF because the network can learn more easily to avoid
inconsistent tagging. However, CSE performs best on average. We suspect that
this is due to the fact that the way of decoding in CSE completely prevents
inconsistent tagging.

Table 8 provides more details for the best setup by listing separate results
per NER task. Furthermore, we examined the influence of applying the rule
based Entity-Fix (see Section 4.3) as a post-processing step.

Table 8:  Average E-$F_{1}$ score of three fine-tuning runs on BERTs pre-
trained with MLM, used relative positional encoding, and WWM. Rows refer to
the fine-tuning method; Columns refer to the NER task; best results per column
are emphasized.

Fine-tuning task |  Entity-Fix rule | GermEval | H. Arendt | Sturm |  LER CG |  LER FG | Average
---|---|---|---|---|---|---|---
SFT | - | 0.8086 | 0.7741 | 0.8555 | 0.9347 | 0.9206 | 0.8587
SFT | ✓ | 0.8408 | 0.7903 | 0.8706 | 0.9474 | 0.9427 | 0.8783
CSE | - | 0.8397 | 0.8048 | 0.8647 | 0.9429 | 0.9401 | 0.8785
CSE | ✓ | 0.8397 | 0.8048 | 0.8647 | 0.9429 | 0.9401 | 0.8785
LCRF | - | 0.8216 | 0.7822 | 0.8453 | 0.9365 | 0.9261 | 0.8623
LCRF | ✓ | 0.8422 | 0.7941 | 0.8629 | 0.9477 | 0.9410 | 0.8776
$\text{LCRF}_{\text{NER}}$ | - | 0.8220 | 0.7857 | 0.8508 | 0.9394 | 0.9278 | 0.8651
$\text{LCRF}_{\text{NER}}$ | ✓ | 0.8448 | 0.7999 | 0.8783 | 0.9488 | 0.9455 | 0.8823

The Entity-Fix rule was specifically designed to fix inconsistencies that
otherwise would lead to an error on the metric. Since the CSE decoding already
includes the rules of the metric, no changes are present if applying the
Entity-Fix rule. This post-processing step led to clear improvements on all
other fine-tuning methods. Surprisingly, although LCRF and
$\text{LCRF}_{\text{NER}}$ were specially designed to learn which consecutive
tags were not allowed in the sequence, they still show significant
improvements with the Entity-Fix rule. Thus, they were not able to learn the
structure of the IOB tagging scheme sufficiently.

The general result is that $\text{LCRF}_{\text{NER}}$ represents the best
fine-tuning method if combined with the Entity-Fix rule.

Additionally, we examined why, in contrast to all other tasks, the H. Arendt
task with CSE yielded slightly better results than with
$\text{LCRF}_{\text{NER}}$ and Entity-Fix rule by comparing the errors made on
the test set. Unfortunately, no explanation could be found in the data.
Nevertheless, $\text{LCRF}_{\text{NER}}$ emerges as the best fine-tuning
method from our experiments in general.

###  6.3 Results of Whole-Word Attention (WWA)

Next, we investigated the influence of replacing the original multi-head
attention layers with our proposed WWA approach. For this, we pre-trained
BERTs with only MLM, relative positional encoding, and WWM with the same
hyper-parameter setup as in Section 6.1. Since, as shown in Eq. (9), WWA
allows to increase the maximal token sequence length, we first pre-trained a
BERT with a token sequence length of 320 (similar to the previous experiments)
but also one with a maximum sequence length of 300 words, that is roughly
about 450 tokens on average (see (9)). This value is chosen so that
approximately the same number of energy values were calculated in this BERT
model as in the comparable model without WWA.

After pre-training of the two BERTs was finished, we fine-tuned all NER tasks
with our best fine-tuning method $\text{LCRF}_{\text{NER}}$. Table 9 compares
the results of using WWA to those without (see Table 8).

Table 9:  Average E-$F_{1}$ score of three fine-tuning runs with $\text{LCRF}_{\text{NER}}$ on BERTs pre-trained with MLM, used relative positional encoding, and WWM. Rows refer to the use of WWA and the maximal sequence length in pre-training; Columns refer to the use of the Entity-Fix rule. WWA |  Max seq. length pre-train | Avg. E-$\bm{F_{1}}$ |  Avg. E-$\bm{F_{1}}$ with Entity-Fix rule
---|---|---|---
- |  320 token | 0.8651 | 0.8823
✓ |  320 token | 0.8676 | 0.8832
✓ |  300 words | 0.8674 | 0.8832

The results show that, on average, WWA slightly improved the original
approach, even though training was done with reduced memory consumption due to
the smaller number of energy values to be calculated and stored. The drawback
is that the pre-training of a BERT with WWA takes about 1.5 times longer than
pre-training without WWA due to the additional window attention layer and the
transformation of the sequence from token to word and vice versa. The runtime
could probably be accelerated by improving the implementation for the
transformation from token sequence to word sequence without looping over the
samples of the batch.

Increasing the maximum sequence length to 300 words did not result in an
additional advantage of the performance. We suspect that the reason is that
the maximum sequence length for NER tasks is not a primary concern because the
samples in all five NER datasets tested almost never exhausted the maximum
sequence length used in pre-training. However, we expect a benefit for other
downstream tasks such as document classification or question answering where
longer sequences and long-range relations are more important.

###  6.4 Comparing Results with the State of the Art

In this section, we compare our results with the current state of the art.
This is only possible for the two LER tasks and the GermEval task since the
other two NER datasets were newly introduced by this paper.

To the best of our knowledge, the current state of the art in both LER tasks
was achieved by Leitner, Rehm, and Moreno-Schneider (2019). They applied a
non-Transformer model on the basis of bidirectional LSTM layers and a LCRF
with classical pre-trained word embeddings. For evaluation, in contrast to our
used E-$F_{1}$ score, they took a token-wise $F_{1}$ score, thus, calculating
precision and recall per token.

Table 10 shows the comparison of the token-wise $F_{1}$ score to our best
results.

Table 10: Comparison of our best results on the two LER-tasks with the previous state of the art models of Leitner, Rehm, and Moreno-Schneider (2019), where T-$F_{1}$ and E-$F_{1}$ are the token-, and entity-wise $F_{1}$ score, respectively. Task |  Model | T-$\bm{F_{1}}$ | E-$\bm{F_{1}}$
---|---|---|---
LER CG |  Previous SoTA Leitner, Rehm, and Moreno-Schneider (2019) | 0.9595 | -
LER CG |  our best | 0.9842 | 0.9488
LER FG |  Previous SoTA Leitner, Rehm, and Moreno-Schneider (2019) | 0.9546 | -
LER FG |  our best | 0.9811 | 0.9455

Note however, that the comparison of the token-wise $F_{1}$ score is not
entirely correct because in our models, words can sometimes break down into
several tokens whereas in Leitner, Rehm, and Moreno-Schneider (2019),
presumably always exactly one token is used per word. Furthermore, since not
published, we were not able to use the identical splits of train, validation,
and test data.

Next, in Table 11, we compare our results for the GermEval task with the
current state of the art which, to the best of our knowledge, was achieved by
Chan, Schweter, and Möller (2020). In addition, we list the best results that
were achieved without a Transformer architecture Riedl and Padó (2018).

Table 11: Comparison of our best results on the GermEval-task with other state of the art models, where E-$F_{1}$ is the entity-wise $F_{1}$ score. Our shown result is again the average of 3 fine-tuning runs. Model | Params | E-$\bm{F_{1}}$
---|---|---
BiLSTM-WikiEmb Riedl and Padó (2018) | unknown | 0.8293
$\text{GBERT}_{\text{Base}}$ Chan, Schweter, and Möller (2020) | 110 mio | 0.8798
$\text{GELECTRA}_{\text{LARGE}}$ Chan, Schweter, and Möller (2020) | 335 mio | 0.8895
our best on our smaller BERTs | 34 mio | 0.8448

While our approach outperforms the BiLSTM results of Riedl and Padó (2018), we
were not able to reach the results of Chan, Schweter, and Möller (2020). One
of the reasons is that our BERTs are smaller and also pre-trained on a much
smaller text corpus.

##  7 Conclusion and Future Work

In this article, we investigated which pre-training method is best to solve a
German NER task on three standard and two newly introduced (H. Arendt and
Sturm) NER datasets. We examined different pre-training tasks, absolute and
relative positional encoding, and masking. We observed that a BERT pre-trained
only on the MLM task combined with relative positional encoding and WWM
yielded the overall best results on these down-stream tasks.

We also introduced two new fine-tuning variants, CSE and
$\text{LCRF}_{\text{NER}}$, designed for NER tasks. Their investigation in
combination with SFT and common LCRF showed that the best pre-training
technique of the BERT remains the best, regardless of the chosen fine-tuning
variant. Furthermore, we introduced the Entity-Fix rule for decoding. Our
results showed that for most German NER tasks, $\text{LCRF}_{\text{NER}}$
together with the Entity-Fix rule delivers the best results, although there
are also tasks for which the best performance can be achieved with CSE
tagging.

In addition, our novel WWA which modifies the Transformer architecture
resulted in small improvements by simultaneously halving the number of energy
values to be calculated. For future work, it would be particularly interesting
to investigate WWA in connection with other downstream tasks such as document
classification or question answering, where the processing of longer sequences
is more important than in NER tasks. Another approach would be to combine WWA
with a sparse-attention mechanism like BigBird Zaheer et al. (2020).

In general, we conducted our research on comparatively small BERT models which
addresses real-world applications with limited hardware resources, leading to
a broader scope of applications and making it accessible to a wider audience.
We plan to investigate whether the results obtained in this article by our
novel techniques can be transferred analogously to the training of larger BERT
models.

###### Acknowledgements.

This work was funded by the European Social Fund (ESF) and the Ministry of
Education, Science, and Culture of Mecklenburg-Western Pomerania (Germany)
within the project Neural Extraction of Information, Structure, and Symmetry
in Images (NEISS) under grant no ESF/14-BM-A55-0006/19.

## List of Abbreviations

BERT

    Bidirectional Encoder Representations from Transformers
CSE

    Class-Start-End
CG

    Coarse-Grained
CRF

    Conditional Random Field
ESF

    European Social Fund
FG

    Fine-Grained
IOB

    Inside-Outside-Beginning
LCRF

    Linear Chain Conditional Random Field
LER

    Legal Entity Recognition
MLM

    Mask Language Model
NLP

    Natural Language Processing
NER

    Named Entity Recognition
NEISS

    Neural Extraction of Information, Structure, and Symmetry in Images
NLP

    Natural Language Processing
NSP

    Next Sentence Prediction
SFT

    Simple Fine-tuning
SOP

    Sentence Order Prediction
WWA

    Whole-Word Attention
WWM

    Whole-Word Masking

## References

  * Attardi (2015) Attardi, Giusepppe. 2015.  Wikiextractor.  https://github.com/attardi/wikiextractor.
  * Beltagy, Peters, and Cohan (2020) Beltagy, Iz, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Benikova et al. (2014) Benikova, Darina, Chris Biemann, Max Kisselew, and Sebastian Pado. 2014.  Germeval 2014 named entity recognition shared task: Companion paper.  In _http://nbn-resolving.de/urn:nbn:de:gbv:hil2-opus-3006_.
  * Buchholz and Marsi (2006) Buchholz, Sabine and Erwin Marsi. 2006.  CoNLL-x shared task on multilingual dependency parsing.  In _Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)_ , pages 149–164, Association for Computational Linguistics.
  * Chan, Schweter, and Möller (2020) Chan, Branden, Stefan Schweter, and Timo Möller. 2020.  German’s next language model.  _arXiv preprint arXiv:2010.10906_.
  * Cui et al. (2019) Cui, Yiming, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019.  Pre-training with whole word masking for chinese bert.  _arXiv preprint arXiv:1906.08101_.
  * Devlin et al. (2019) Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Technologies_ , 1:4171–4186.
  * Hahn, Breysach, and Pischel (2020) Hahn, Barbara, Barbara Breysach, and Christian Pischel. 2020.  Hannah Arendt Digital. Kritische Gesamtausgabe. Sechs Essays.  https://hannah-arendt-edition.net/3p.html.
  * Hamborg et al. (2017) Hamborg, Felix, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017.  news-please: A generic news crawler and extractor.  In _Proceedings of the 15th International Symposium of Information Science_ , pages 218–223.
  * Labusch, Neudecker, and Zellhöfer (2019) Labusch, Kai, Clemens Neudecker, and David Zellhöfer. 2019.  BERT for named entity recognition in contemporary and historic german.  In _Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers_ , pages 1–9, German Society for Computational Linguistics & Language Technology.
  * Lafferty, McCallum, and Pereira (2001) Lafferty, John, Andrew McCallum, and Fernando Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  _International Conference on Machine Learning (ICML)_.
  * Lan et al. (2020) Lan, Zhenzhong, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  _International Conference on Learning Representations_.
  * Leitner, Rehm, and Moreno-Schneider (2019) Leitner, Elena, Georg Rehm, and Julian Moreno-Schneider. 2019.  Fine-grained named entity recognition in legal documents.  In _Semantic Systems. The Power of AI and Knowledge Graphs_ , Lecture Notes in Computer Science, pages 272–287, Springer International Publishing.
  * Leitner, Rehm, and Moreno-Schneider (2020) Leitner, Elena, Georg Rehm, and Julián Moreno-Schneider. 2020.  A Dataset of German Legal Documents for Named Entity Recognition.  _arXiv:2003.13016 [cs]_.  ArXiv: 2003.13016.
  * Liu et al. (2019) Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luoma and Pyysalo (2020) Luoma, Jouni and Sampo Pyysalo. 2020.  Exploring cross-sentence contexts for named entity recognition with BERT.  _http://arxiv.org/abs/2006.01563_.
  * Mikolov et al. (2013) Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26:3111–3119.
  * Nakayama (2018) Nakayama, Hiroki. 2018.  seqeval: A python framework for sequence labeling evaluation.  Software available from https://github.com/chakki-works/seqeval.
  * Ramshaw and Marcus (1999) Ramshaw, L. A. and M. P. Marcus. 1999.  Text chunking using transformation-based learning.  In Susan Armstrong, Kenneth Church, Pierre Isabelle, Sandra Manzi, Evelyne Tzoukermann, and David Yarowsky, editors, _Natural Language Processing Using Very Large Corpora_ , Text, Speech and Language Technology. Springer Netherlands, pages 157–176.
  * Riedl and Padó (2018) Riedl, Martin and Sebastian Padó. 2018.  A named entity recognition shootout for german.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 120–125, Association for Computational Linguistics.
  * Rosendahl et al. (2019) Rosendahl, Jan, Viet Anh Khoa Tran, Weiyue Wang, and Hermann Ney. 2019.  Analysis of positional encodings for neural machine translation.  _IWSLT, Hong Kong, China_.
  * Sang and De Meulder (2003) Sang, Erik F and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  _arXiv preprint cs/0306050_.
  * Shaw, Uszkoreit, and Vaswani (2019) Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. 2019.  Self-attention with relative position representations.  _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2:464–468.
  * Souza, Nogueira, and Lotufo (2020) Souza, Fábio, Rodrigo Nogueira, and Roberto Lotufo. 2020.  Portuguese named entity recognition using BERT-CRF.  _http://arxiv.org/abs/1909.10649_.
  * Sutton and McCallum (2010) Sutton, Charles and Andrew McCallum. 2010.  An introduction to conditional random fields.  _https://homepages.inf.ed.ac.uk/csutton/publications/crftutv2.pdf_.
  * T. Schrade (2018) T. Schrade, M. Trautmann. 2018.  DER STURM. Digitale Quellenedition zur Geschichte der internationalen Avantgarde.  _https://sturm-edition.de/id/S.0000001_.
  * TEI-Consortium (2017) TEI-Consortium. 2017.  Guidelines for electronic text encoding and interchange.  https://tei-c.org/.
  * Vaswani et al. (2017) Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wick et al. (2021) Wick, Christoph, Benjamin Kühn, Gundram Leifert, Konrad Sperfeld, Tobias Strauß, Jochen Zöllner, and Tobias Grüning. 2021.  tfaip - a generic and powerful research framework for deep learning based on tensorflow.  https://github.com/Planet-AI-GmbH/tfaip/blob/master/joss/paper.pdf.  Submitted to JOSS.
  * Yang et al. (2019) Yang, Zhilin, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5753–5763.
  * Zaheer et al. (2020) Zaheer, Manzil, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020\.  Big bird: Transformers for longer sequences.  _arXiv preprint arXiv:2007.14062_.

  *[NER]: Named Entity Recognition
  *[BERT]: Bidirectional Encoder Representations from Transformers
  *[NLP]: Natural Language Processing
  *[BERTs]: Bidirectional Encoder Representations from Transformers
  *[MLM]: Mask Language Model
  *[SOP]: Sentence Order Prediction
  *[NSP]: Next Sentence Prediction
  *[WWM]: Whole-Word Masking
  *[LCRFs]: Linear Chain
  *[CSE]: Class-Start-End
  *[WWA]: Whole-Word Attention
  *[LER]: Legal Entity Recognition
  *[CG]: Coarse-Grained
  *[FG]: Fine-Grained
  *[IOB]: Inside-Outside-Beginning
  *[SFT]: Simple Fine-tuning
  *[CRFs]: Conditional Random Field
  *[LCRF]: Linear Chain
  *[ESF]: European Social Fund
  *[NEISS]: Neural Extraction of Information, Structure, and Symmetry in Images


256.0748pt+\eta_{\nu_{1}}{}^{\mu_{2}}\left(-\xi_{2}+\frac{\alpha_{S_{1}^{\prime}}(\xi_{20}-\xi_{10})}{t_{1}+t_{2}}\right)_{\nu_{2}}\bigg{]}$
$\displaystyle+(D-1)\,\frac{1}{t_{1}+t_{2}}\,\left(\frac{\alpha_{S_{1}^{\prime}}}{\alpha_{S_{2}^{\prime}}}+\frac{\alpha_{S_{2}^{\prime}}}{\alpha_{S_{1}^{\prime}}}-1\right)\eta^{\mu_{1}\mu_{2}}$
$\displaystyle-\left(\frac{\alpha_{S_{2}^{\prime}}(\xi_{10}-\xi_{20})}{t_{1}+t_{2}}\right)^{\mu_{1}}\left(\frac{\alpha_{S_{1}^{\prime}}(\xi_{20}-\xi_{10})}{t_{1}+t_{2}}\right)^{\mu_{2}}-\left(\frac{\alpha_{S_{1}^{\prime}}(\xi_{20}-\xi_{10})}{t_{1}+t_{2}}\right)^{\mu_{1}}\left(\frac{\alpha_{S_{2}^{\prime}}(\xi_{10}-\xi_{20})}{t_{1}+t_{2}}\right)^{\mu_{2}}+\frac{1}{t_{1}+t_{2}}\,\eta^{\mu_{1}\mu_{2}}$
(8.90)

We can impose momentum conservation after the action of Corolla $\hat{C}$. As
advertised earlier in (8.80), all the loop momenta $\xi_{i0}$ appear in a
particular combination such that the dependence on $\ell$ vanishes. Imposing
momentum conservation, we have $\xi_{10}-\xi_{20}=\xi_{1}$, and
$\xi_{2}=-\xi_{1}$. Since we are restricted in one cone, let us substitute
$\alpha_{S_{i}^{\prime}}$ in terms of $t_{i}$:
$\alpha_{S_{i}^{\prime}}=-t_{i+1}$. After some simplifications, we obtain:

$\displaystyle\hat{C}_{\leavevmode\hbox to8.94pt{\vbox
to3.25pt{\pgfpicture\makeatletter\hbox{\hskip 4.46817pt\lower-1.62271pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}} {}{{}}{}{{{}} {}{}{}{}{}{}{}{}
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{1.42271pt}{0.0pt}\pgfsys@curveto{1.42271pt}{0.78575pt}{0.78575pt}{1.42271pt}{0.0pt}{1.42271pt}\pgfsys@curveto{-0.78575pt}{1.42271pt}{-1.42271pt}{0.78575pt}{-1.42271pt}{0.0pt}\pgfsys@curveto{-1.42271pt}{-0.78575pt}{-0.78575pt}{-1.42271pt}{0.0pt}{-1.42271pt}\pgfsys@curveto{0.78575pt}{-1.42271pt}{1.42271pt}{-0.78575pt}{1.42271pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {}{{}}{}
{}{}{}\pgfsys@moveto{1.42271pt}{0.0pt}\pgfsys@lineto{4.26817pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {}{{}}{}
{}{}{}\pgfsys@moveto{-1.42271pt}{0.0pt}\pgfsys@lineto{-4.26817pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\,e^{\ldots}$
$\displaystyle=\frac{1}{(t_{1}+t_{2})^{2}}\left[-\xi_{1}^{\mu_{1}}\xi_{1}^{\mu_{2}}\big{(}2t_{1}^{2}+12t_{1}t_{2}+2t_{2}^{2}\big{)}+\xi_{1}^{2}\,\eta^{\mu_{1}\mu_{2}}\big{(}5t_{1}^{2}+8t_{1}t_{2}+5t_{2}^{2}\big{)}\right]$
$\displaystyle\hskip
28.45274pt+\frac{\eta^{\mu_{1}\mu_{2}}}{t_{1}+t_{2}}\left[3\frac{t_{1}}{t_{2}}+3\frac{t_{2}}{t_{1}}-2\right]e^{-\left(\alpha_{S_{1}^{\prime}}\xi_{20}+\alpha_{S_{2}^{\prime}}\xi_{10}\right)^{2}/{(t_{1}+t_{2})}-\alpha_{S_{1}^{\prime}}\xi_{20}^{2}-\alpha_{S_{2}^{\prime}}\xi_{10}^{2}-s_{1}\xi_{1}^{2}-s_{2}\xi_{2}^{2}}$
(8.91)

(Reg-Res) picks up the particular poles in Schwinger parameters and integrates
the rest. For this case, in this particular cone, we have

$\displaystyle\big{(}\text{Reg-Res}\big{)}$
$\displaystyle=\int\differential^{2}s\left[\left(\int\differential
t_{1}\,\differential t_{2}\
\underset{t_{1},t_{2}}{\text{Reg}}\right)+\left(\int\differential t_{1}\
\underset{t_{1}}{\text{Reg}}\ \underset{t_{2}\to
0}{\text{Res}}\right)+\left(\int\differential t_{2}\
\underset{t_{2}}{\text{Reg}}\ \underset{t_{1}\to
0}{\text{Res}}\right)+\underset{t_{1},t_{2}\to 0}{\text{Res}}\right]$ (8.92)

Thus, we finally get,

$\displaystyle\mathcal{A}_{2}[1,2]^{\text{YM}}$
$\displaystyle=\xi_{1}^{2}\xi_{2}^{2}\,\big{(}\text{Reg-
Res}\big{)}\hat{C}_{\leavevmode\hbox to8.94pt{\vbox
to3.25pt{\pgfpicture\makeatletter\hbox{\hskip 4.46817pt\lower-1.62271pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}} {}{{}}{}{{{}} {}{}{}{}{}{}{}{}
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{1.42271pt}{0.0pt}\pgfsys@curveto{1.42271pt}{0.78575pt}{0.78575pt}{1.42271pt}{0.0pt}{1.42271pt}\pgfsys@curveto{-0.78575pt}{1.42271pt}{-1.42271pt}{0.78575pt}{-1.42271pt}{0.0pt}\pgfsys@curveto{-1.42271pt}{-0.78575pt}{-0.78575pt}{-1.42271pt}{0.0pt}{-1.42271pt}\pgfsys@curveto{0.78575pt}{-1.42271pt}{1.42271pt}{-0.78575pt}{1.42271pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {}{{}}{}
{}{}{}\pgfsys@moveto{1.42271pt}{0.0pt}\pgfsys@lineto{4.26817pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {}{{}}{}
{}{}{}\pgfsys@moveto{-1.42271pt}{0.0pt}\pgfsys@lineto{-4.26817pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\,e^{\ldots}$
$\displaystyle=\int_{-\infty}^{0}\differential t_{1}\,\differential
t_{2}\bigg{[}\frac{1}{(t_{1}+t_{2})^{2}}\bigg{[}-\xi_{1}^{\mu_{1}}\xi_{1}^{\mu_{2}}\big{(}2t_{1}^{2}+12t_{1}t_{2}+2t_{2}^{2}\big{)}$
$\displaystyle\hskip
56.9055pt+\xi_{1}^{2}\,\eta^{\mu_{1}\mu_{2}}\big{(}5t_{1}^{2}+8t_{1}t_{2}+5t_{2}^{2}\big{)}\bigg{]}-2\frac{\eta^{\mu_{1}\mu_{2}}}{t_{1}+t_{2}}\bigg{]}+2\int_{-\infty}^{0}\differential
t\,\eta^{\mu_{1}\mu_{2}}\,e^{0}$ (8.93)

It can be checked that the above result equals the parametric representation
of two point gluon correlator as derived in [38].

##  9 Conclusion and Outlook

The essence of the positive geometry program is to analyse the combinatorial
and geometric structures that _naturally_ arise in the bare kinematic space of
scattering data and study if these structures can give birth to S-matrix of
local unitary quantum field theories. The naturality referred to above arises
from the fact that these structures do not emerge within the world of quantum
field theory computations but appear at first in an apparently disconnected
domain of quiver representation theory and so called gentle algebras. The
simplest example of this “S-matrix without quantum field theory" is the bi-
adjoint scalar amplitudes. Namely, inside the Kinematic space of scattering
data, a rich class of combinatorial geometries live whose embedding inside the
kinematic space makes no reference to Feynman diagrams or physical principles
such as unitarity or locality, [1, 15, 16].

Once we locate these positive geometries inside the kinematic space of scalar
particles, a large class of tree-level scattering amplitudes arise directly as
the canonical forms which are fixed uniquely by the geometries. Several
intriguing results have started coming out of this program in last few years.
The top-form defined by the ABHY associahedron not only generates amplitude of
bi-adjoint scalar theory, but simple linear diffeomorphisms in kinematic space
changes the shape of the associahedron in such a way that the resulting
geometry (and the corresponding form) generate amplitudes of a large class of
scalar theories at tree-level, [20], including the amplitude of ordinary
$\phi^{3}$ theory without color, [44].

ABHY associahedron is hence a “universal polytope" that underlies tree-level S
matrix of an infinite family of scalar theories which appear to be completely
unrelated to each other at the level of Lagrangian or Feynman diagrams. In
fact, more is true. In [19], it was shown that “shadows" of the associahedron
in certain regions of kinematic space are lower dimensional accordiohedra
which are the geometries underlying scalar theories with $\phi^{p},p>4$
interactions. As a result, the effect of integrating out massive modes (at
least at tree-level) was same as mapping of positive geometry to a _different_
class of lower dimensional positive geometry. Both of these results point us
towards the universality of the ABHY associahedron as a geometry that
underlies a large class of tree-level amplitudes. Perhaps, the most striking
evidence of this universality is the new understanding of the NLSM amplitude
as a specific deformation of $\textrm{Tr}(\phi^{3})$ amplitude, [21]. In fact,
this relationship between NLSM and positive geometries underlying $\phi^{3}$
S-matrix extends to all orders in perturbation theory, a result completely
unforseen in traditional perturbative analysis.

S matrix of non-supersymmetric gauge theories have so far proved resistant
against their “integration" into the positive geometry program. There is a
certain irony to it, as the connection between polytopes and amplitudes was
first discovered in the seminal work of Hodges[45] which showed that tree-
level NMHV amplitudes were simply volume of certain polytopes in the kinematic
space of 4 dimensional Yang-Mills theory. This was followed by the striking
discovery of the amplituhedron and the significant insights it continues to
bring into the study of SYM amplitudes, [5, 46, 47]. However in the non-
supersymmetric world, it is the scalar theories which have found their most
natural home in the “amplituhedron program". A breakthrough in this direction
has come in the recent paper by Arkani-Hamed, Cao, Dong, He, Figueiredo, He,
where gluon amplitude is realised as a “deformed" amplitude of so called
scaffolding scalars. This is a remarkable result with far reaching
implications in which the relationship $\textrm{Tr}(\phi^{3})$ amplitude
(which arise simply from the combinatorial structures such as headlight
functions) and the YM S-matrix “factors through" stringy
$\textrm{Tr}(\phi^{3})$ amplitudes.

In this paper we have shown that the canonical form defined by positive
geometries such as associahedron can be used to generate another class of
scalars by contraction of the form with certain multi-vector field. This MVF
is a sum over all the vertices of the associahedron and is defined via the
mutation rules which also determine the canonical form. At each vertex of the
associahedron, the MVF is fixed by a Corolla polynomial which in turn depends
on the triangulation as well as the finer structure that triangulation carries
with it via it’s representation as bracketed words. With such a structure at
hand, ABHY had already shown that color-dressed $\phi^{3}$ amplitude at tree-
level is a contraction of this MVF with the d-log form. As we have shown in
this paper, even the YM ampitudes emerge through such an algebraic operation.
We then showed how these ideas carry over to the planar one loop integrand of
Yang-Mills theory which can also be realised as a contraction of the d log
form associated to $\hat{D}_{n}$ polytope with the MVF which is generated by
the Corolla polynomial associated to pseudo-triangulations. In fact the
seminal results of KSVS along with the curve integral formula for
$\textrm{Tr}(\phi^{3})$ amplitudes can be combined rather immediately to
define _a_ curve integral formula for gluons. Although we explicitly derived
such a formula for tree-level and one loop amplitudes, as emphasized in the
introduction, the extension to higher loop non-planar amplitudes should be
possible.

As the primary motivation of our work is to obtain an insight into how
S-matrix of non super-symmetric gauge theories fit into the positive
geometries program, it is important to mention the limitation of our results
from that perspective. Is there a “momentum amplituhedron" for perturbative
gluon amplitudes? Our analysis does not answer this question. In fact, our
results are premised on the observation that the amplitude of YM theory
without maximal supersymmetry are not d-log forms. Of course, this does not
rule out the possibility that the meromorphic forms corresponding to e.g.
tree-level YM amplitudes [48] may be defined by a new class of geometries,
(See e.g. [34] where the precise relationship via push-forward map of the
d-log form defined by the ABHY associahedron with the d-log form defined by
the momentum amplituhedron was established). However, the use of Corolla
polynomial led us to a different geometric interpretation of these amplitudes,
as scalar functions on the kinematic space. Thus we are far from “integrating
gauge theory S-matrix into the positive geometries program". Nevertheless, our
hope is that the combinatorial structure that glues together all the marked
triangulations may eventually lead us to geometries which would help us
discover a “momentum amplituhedron" for perturbative non-supersymmetric gluon
amplitudes.

We now summarise several avenues for further investigation.

### A homology for each vertex of $A_{n-3}$.

As we saw in section 5, the action of the Corolla polynomial maps a
triangulation to a set of marked triangulations. As each triangulation is dual
to a graph, it is natural to quantify the set of marked triangulations with a
(direct) sum over graphs is the generator of graph homology $[\gamma]$ (see
section 5.2)

$\displaystyle[\gamma]\,:=\,e^{\chi}\,\gamma$ (9.1)

Hence Corolla maps the set of vertices of $A_{n-3}$ to the corresponding
_unique_ homology class $[\gamma_{n}]\,|\,\forall\,\gamma_{n}\,\textrm{with
fixed ordering of external vertices.}$. It would be interesting to investigate
this structure further in light of recent advances in studies in graph
homology, see for example [49, 50].

### Relationship with worldsheet formula and the curve integral formula for
Stringy YM amplitude.

The CHY formula for tree-level amplitude of bi-adjoint scalar theory is simply
a pull back of the d-log form from the kinematic space to the so-called
worldsheet associahedron, [1]. Worldsheet associahedron is the real section of
moduli space of $n$ punctured Riemann sphere compactified using Plucker co-
ordinates. The pull back of the d-log form is through scattering equations
which are simply diffeomorphism between world-sheet and ABHY associahedra. It
is then natural to investigate if there exists a Corolla differential on the
worldsheet and if it can be used to “spin up" the Parke-Taylor form to obtain
the CHY integrand for tree-level YM amplitude.

As discussed previously, the most clear relationship between
$\textrm{Tr}(\phi^{3})$ and YM amplitude to all orders in the loop expansion
has been discovered in [29]. E.g. the tree-level stringy
$\textrm{Tr}(\phi^{3})$ amplitude of $2n$ scalars, under specific deformation
of the planar kinematic variables generate stringy YM amplitude once pair of
scalars $(2i-1,2i)$ are “combined" to form a spin-1 state.252525This operation
has been called scaffolding in [29]. It will be extremely interesting to see
if there is a representation of the Corolla polynomial which maps the stringy
$\textrm{Tr}(\phi^{3})$ amplitude to stringy gluon amplitude at least at tree-
level. This may lead to a direct connection between the ideas advocated in
this paper and in [29].

### Corolla for Gravitational amplitude

An $n$ point tree-level gravitational amplitude requires higher order contact
terms with degree $4\leq\ d\leq\ n$. Clearly the underlying structure must far
more intricate than the Corolla polynomial whose differential representation
generates graph homology in which quartic vertices and marked edges are
identified. An intriguing (but highly speculative) possibility is to “fuse"
two Corolla differentials using the KLT kernel. Needless to say, this is
perhaps one of the major open questions that face the idea of using Graph
homology to map $\phi^{3}$ amplitude to a gravitational S-matrix.

## Acknowledgement

We would like to thank Pinaki Banerjee, Frédéric Chapoton, Chandramouli
Chowdhury, Nima-Arkani Hamed, Subramanya Hegde, Mrunmay Jagadale, Vincent
Pilaud, Prashanth Raman, Ashoke Sen, Aninda Sinha and Adarsh Sudhakar for
insightful discussions over last few years on many aspects of the positive
geometry program. We would especially like to thank Pinaki Banerjee and
Chandramouli Chowdhury for crucial discussions during the initial phase of
this project. AL thanks Center for High Energy Physics (CHEP) at IISc,
International Center for Theoretical Sciences (ICTS), and IIT Kanpur for their
hospitality during the course of the project.

## References

  * [1] N. Arkani-Hamed, Y. Bai, S. He and G. Yan, _Scattering Forms and the Positive Geometry of Kinematics, Color and the Worldsheet_ , _JHEP_ 05 (2018) 096, [1711.09102].
  * [2] D. Kreimer, M. Sars and W. D. van Suijlekom, _Quantization of gauge fields, graph polynomials and graph homology_ , _Annals Phys._ 336 (2013) 180–222, [1208.6477].
  * [3] N. Arkani-Hamed, H. Frost, G. Salvatori, P.-G. Plamondon and H. Thomas, _All Loop Scattering as a Counting Problem_ , 2309.15913.
  * [4] N. Arkani-Hamed, H. Frost, G. Salvatori, P.-G. Plamondon and H. Thomas, _All Loop Scattering For All Multiplicity_ , 2311.09284.
  * [5] N. Arkani-Hamed and J. Trnka, _The Amplituhedron_ , _JHEP_ 10 (2014) 030, [1312.2007].
  * [6] Z. Bern, E. Herrmann, S. Litsey, J. Stankowicz and J. Trnka, _Evidence for a Nonplanar Amplituhedron_ , _JHEP_ 06 (2016) 098, [1512.08591].
  * [7] J. D. Stasheff, _Homotopy associativity of h-spaces. ii_ , _Transactions of the American Mathematical Society_ 108 (1963) 293–312.
  * [8] T. Manneville and V. Pilaud, _Geometric realizations of the accordion complex of a dissection_ , _Discrete amp; Computational Geometry_ 61 (May, 2018) 507–540, [1703.09953].
  * [9] F. Chapoton, _Stokes posets and serpent nests_ , _Discrete Mathematics and Theoretical Computer Science_ 18 (2016) , [1505.05990].
  * [10] P. Banerjee, A. Laddha and P. Raman, _Stokes polytopes: the positive geometry for $\phi^{4}$ interactions_, _JHEP_ 08 (2019) 067, [1811.05904].
  * [11] P. B. Aneesh, M. Jagadale and N. Kalyanapuram, _Accordiohedra as positive geometries for generic scalar field theories_ , _Phys. Rev. D_ 100 (2019) 106013, [1906.12148].
  * [12] P. Raman, _The positive geometry for $\phi^{p}$ interactions_, _JHEP_ 10 (2019) 271, [1906.02985].
  * [13] P. B. Aneesh, P. Banerjee, M. Jagadale, R. Rajan, A. Laddha and S. Mahato, _On positive geometries of quartic interactions: Stokes polytopes, lower forms on associahedra and world-sheet forms_ , _JHEP_ 04 (2020) 149, [1911.06008].
  * [14] N. Arkani-Hamed, Y. Bai and T. Lam, _Positive Geometries and Canonical Forms_ , _JHEP_ 11 (2017) 039, [1703.04541].
  * [15] V. Bazier-Matte, N. Chapelier-Laget, G. Douville, K. Mousavand, H. Thomas and E. Yıldırım, _ABHY Associahedra and Newton polytopes of $F$-polynomials for finite type cluster algebras_, 1808.09986.
  * [16] A. Padrol, Y. Palu, V. Pilaud and P.-G. Plamondon, _Associahedra for finite type cluster algebras and minimal relations between $\mathbf{g}$-vectors_, _Proc. Lond. Math. Soc._ 127 (2023) 513–588, [1906.06861].
  * [17] L. Ferro and T. Lukowski, _Amplituhedra, and beyond_ , _J. Phys. A_ 54 (2021) 033001, [2007.04342].
  * [18] E. Herrmann and J. Trnka, _The SAGEX review on scattering amplitudes Chapter 7: Positive geometry of scattering amplitudes_ , _J. Phys. A_ 55 (2022) 443008, [2203.13018].
  * [19] M. Jagadale and A. Laddha, _Towards positive geometry of multi scalar field amplitudes. Accordiohedron and effective field theory_ , _JHEP_ 04 (2022) 100, [2104.04915].
  * [20] M. Jagadale and A. Laddha, _Towards Positive Geometries of Massive Scalar field theories_ , 2206.07979.
  * [21] N. Arkani-Hamed, Q. Cao, J. Dong, C. Figueiredo and S. He, _Hidden zeros for particle/string amplitudes and the unity of colored scalars, pions and gluons_ , 2312.16282.
  * [22] G. Salvatori and S. Stanojevic, _Scattering Amplitudes and Simple Canonical Forms for Simple Polytopes_ , _JHEP_ 03 (2021) 067, [1912.06125].
  * [23] N. Arkani-Hamed, S. He, G. Salvatori and H. Thomas, _Causal diamonds, cluster polytopes and scattering amplitudes_ , _JHEP_ 11 (2022) 049, [1912.12948].
  * [24] N. Arkani-Hamed, H. Frost, G. Salvatori, P.-G. Plamondon and H. Thomas, _Surfacehedra, to appear_ , .
  * [25] M. Jagadale and A. Laddha, _On positive geometries of quartic interactions: one loop integrands from polytopes_ , _JHEP_ 07 (2021) 136, [2007.12145].
  * [26] G. Salvatori, _1-loop Amplitudes from the Halohedron_ , _JHEP_ 12 (2019) 074, [1806.01842].
  * [27] J. Trnka, _Towards the Gravituhedron: New Expressions for NMHV Gravity Amplitudes_ , _JHEP_ 04 (2021) 253, [2012.15780].
  * [28] S. He, Y.-t. Huang and C.-K. Kuo, _The ABJM Amplituhedron_ , _JHEP_ 09 (2023) 165, [2306.00951].
  * [29] N. Arkani-Hamed, Q. Cao, J. Dong, C. Figueiredo and S. He, _Scalar-Scaffolded Gluons and the Combinatorial Origins of Yang-Mills Theory_ , 2401.00041.
  * [30] Z. Bern, J. J. M. Carrasco and H. Johansson, _New Relations for Gauge-Theory Amplitudes_ , _Phys. Rev. D_ 78 (2008) 085011, [0805.3993].
  * [31] R. H. Boels, R. S. Isermann, R. Monteiro and D. O’Connell, _Colour-Kinematics Duality for One-Loop Rational Amplitudes_ , _JHEP_ 04 (2013) 107, [1301.4165].
  * [32] Z. Bern, J. J. Carrasco, M. Chiodaroli, H. Johansson and R. Roiban, _The Duality Between Color and Kinematics and its Applications_ , 1909.01358.
  * [33] C. Cheung, C.-H. Shen and C. Wen, _Unifying Relations for Scattering Amplitudes_ , _JHEP_ 02 (2018) 095, [1705.03025].
  * [34] D. Damgaard, L. Ferro, T. Lukowski and R. Moerman, _Momentum amplituhedron meets kinematic associahedron_ , _JHEP_ 02 (2021) 041, [2010.15858].
  * [35] D. Kreimer and K. Yeats, _Properties of the corolla polynomial of a 3-regular graph_ , 1207.5460.
  * [36] J. Conant and K. Vogtmann, _On a theorem of kontsevich_ , _Algebraic amp; Geometric Topology_ 3 (Dec., 2003) 1167–1224.
  * [37] M. Berghoff and D. Kreimer, _Graph complexes and Feynman rules_ , _Commun. Num. Theor. Phys._ 17 (2023) 103–172, [2008.09540].
  * [38] D. Prinz, _The Corolla Polynomial for spontaneously broken Gauge Theories_ , _Math. Phys. Anal. Geom._ 19 (2016) 18, [1603.03321].
  * [39] M. Golz, _New graph polynomials in parametric QED Feynman integrals_ , _Annals Phys._ 385 (2017) 328–346, [1703.05134].
  * [40] C. Bogner and S. Weinzierl, _Feynman graph polynomials_ , _Int. J. Mod. Phys. A_ 25 (2010) 2585–2618, [1002.3458].
  * [41] S. Weinzierl, _Feynman Integrals_ , 2201.03593.
  * [42] V. A. Smirnov, _Analytic tools for Feynman integrals_ , vol. 250.  Springer, 2012, 10.1007/978-3-642-34886-0.
  * [43] V. A. Smirnov, _Evaluating feynman integrals_ , vol. 211.  Springer Science & Business Media, 2004.
  * [44] M. Jagadale and A. Laddha, _Positive Geometries of S-matrix without Color_ , 2304.04571.
  * [45] A. Hodges, _Eliminating spurious poles from gauge-theoretic amplitudes_ , _JHEP_ 05 (2013) 135, [0905.1473].
  * [46] N. Arkani-Hamed, J. Henn and J. Trnka, _Nonperturbative negative geometries: amplitudes at strong coupling and the amplituhedron_ , _JHEP_ 03 (2022) 108, [2112.06956].
  * [47] N. Arkani-Hamed, W. Flieger, J. M. Henn, A. Schreiber and J. Trnka, _Coulomb Branch Amplitudes from a Deformed Amplituhedron Geometry_ , 2311.10814.
  * [48] S. He and C. Zhang, _Notes on Scattering Amplitudes as Differential Forms_ , _JHEP_ 10 (2018) 054, [1807.11051].
  * [49] B. C. Ward, _Massey products for graph homology_ , 1903.12055.
  * [50] M. Batanin, M. Markl and J. Obradović, _Minimal models for graph-related (hyper)operads_ , 2002.06640.