# FiNER: Financial Numeric Entity Recognition for XBRL Tagging

Lefteris Loukas1,2, Manos Fergadiotis1,2, Ilias Chalkidis3, Eirini
Spyropoulou1,
Prodromos Malakasiotis1,2, Ion Androutsopoulos1,2, George Paliouras1
1Institute of Informatics and Telecommunications, NCSR “Demokritos”
2Department of Informatics, Athens University of Economics and Business
3Department of Computer Science, University of Copenhagen

###### Abstract

Publicly traded companies are required to submit periodic reports with
eXtensive Business Reporting Language (xbrl) word-level tags. Manually tagging
the reports is tedious and costly. We, therefore, introduce xbrl tagging as a
new entity extraction task for the financial domain and release finer-139, a
dataset of 1.1M sentences with gold xbrl tags. Unlike typical entity
extraction datasets, finer-139 uses a much larger label set of 139 entity
types. Most annotated tokens are numeric, with the correct tag per token
depending mostly on context, rather than the token itself. We show that
subword fragmentation of numeric expressions harms bert’s performance,
allowing word-level bilstms to perform better. To improve bert’s performance,
we propose two simple and effective solutions that replace numeric expressions
with pseudo-tokens reflecting original token shapes and numeric magnitudes. We
also experiment with fin-bert, an existing bert model for the financial
domain, and release our own bert (sec-bert), pre-trained on financial filings,
which performs best. Through data and error analysis, we finally identify
possible limitations to inspire future work on xbrl tagging.

##  1 Introduction

††Source code: https://github.com/nlpaueb/finer††Correspondence:
<EMAIL_ADDRESS>
Natural language processing (nlp) for finance is an emerging research area
(Hahn et al., 2019; Chen et al., 2020; El-Haj et al., 2020). Financial data
are mostly reported in tables,but substantial information can also be found in
textual form, e.g., in company filings, analyst reports, and economic news.
Such information is useful in numerous financial intelligence tasks, like
stock market prediction (Chen et al., 2019; Yang et al., 2019), financial
sentiment analysis (Malo et al., 2014; Wang et al., 2013; Akhtar et al.,
2017), economic event detection (Ein-Dor et al., 2019; Jacobs et al., 2018;
Zhai and Zhang, 2019), and causality analysis (Tabari et al., 2018; Izumi and
Sakaji, 2019). In this work, we study how financial reports can be
automatically enriched with word-level tags from the eXtensive Business
Reporting Language (xbrl), a tedious and costly task not considered so
far.111See https://www.xbrl.org/the-standard/what/an-introduction-to-xbrl/ for
an introduction to xbrl.

Figure 1: Sentences from finer-139, with xbrl tags on numeric and non-numeric
tokens. xbrl tags are actually xml-based and most tagged tokens are numeric.

To promote transparency among shareholders and potential investors, publicly
traded companies are required to file periodic financial reports. These
comprise multiple sections, including financial tables and text paragraphs,
called _text notes_. In addition, legislation in the us, the uk, the eu and
elsewhere requires the reports to be annotated with tags of xbrl, an xml-based
language, to facilitate the processing of financial information. The
annotation of tables can be easily achieved by using company-specific pre-
tagged table templates, since the structure and contents of the tables in the
reports of a particular company rarely change. On the other hand, the
unstructured and dynamic nature of text notes (Figure 1) makes adding xbrl
tags to them much more difficult. Hence, we focus on automatically tagging
text notes. Tackling this task could facilitate the annotation of new and old
reports (which may not include xbrl tags), e.g., by inspecting automatically
suggested tags.

Dataset | Domain | Entity Types
---|---|---
conll-2003 | Generic | 4
ontonotes-v5 | Generic | 18
ace-2005 | Generic | 7
genia | Biomedical | 36
Chalkidis et al. (2019) | Legal | 14
Francis et al. (2019) | Financial | 9
finer-139 (ours) | Financial | 139

Table 1: Examples of previous entity extraction datasets. Information about
the first four from Tjong Kim Sang and De Meulder (2003); Pradhan et al.
(2012); Doddington et al. (2004); Kim et al. (2003).

Towards this direction, we release finer-139, a new dataset of 1.1M sentences
with gold xbrl tags, from annual and quarterly reports of publicly traded
companies obtained from the us Securities and Exchange Commission (sec).
Unlike other entity extraction tasks, like named entity recognition (ner) or
contract element extraction (Table 1), which typically require identifying
entities of a small set of common types (e.g., persons, organizations), xbrl
defines approx. 6k entity types. As a first step, we consider the 139 most
frequent xbrl entity types, still a much larger label set than usual.

Another important difference from typical entity extraction is that most
tagged tokens ($\sim$91%) in the text notes we consider are numeric, with the
correct tag per token depending mostly on context, not the token itself
(Figure 1). The abundance of numeric tokens also leads to a very high ratio of
out-of-vocabulary (oov) tokens, approx. 10.4% when using a custom word2vec
Mikolov et al. (2013a) model trained on our corpus. When using subwords, e.g.,
in models like bert Devlin et al. (2019), there are no oov tokens, but numeric
expressions get excessively fragmented, making it difficult for the model to
gather information from the fragments and correctly tag them all. In our
experiments, this is evident by the slightly better performance of stacked
bilstms (Graves et al., 2013; Lample et al., 2016) operating on word
embeddings compared to bert. The latter improves when using a crf Lafferty et
al. (2001) layer, which helps avoid assigning nonsensical sequences of labels
to the fragments (subwords) of numeric expressions.

To further improve bert’s performance, we propose two simple and effective
solutions that replace numeric expressions with pseudo-tokens reflecting the
original token shapes and magnitudes. We also experiment with fin-bert (Yang
et al., 2020), an existing bert model for the financial domain, and release
our own family of bert models, pre-trained on 200k financial filings,
achieving the best overall performance.

Our key contributions are:

  1. 1.

We introduce xbrl tagging, a new financial nlp task for a real-world need, and
we release finer-139, the first xbrl tagging
dataset.222https://huggingface.co/datasets/nlpaueb/finer-139

  2. 2.

We provide extensive experiments using bilstms and bert with generic or in-
domain pre-training, which establish strong baseline results for future work
on finer-139.

  3. 3.

We show that replacing numeric tokens with pseudo-tokens reflecting token
shapes and magnitudes significantly boosts the performance of bert-based
models in this task.

  4. 4.

We release a new family of bert models (sec-bert, sec-bert-num, sec-bert-
shape) pre-trained on 200k financial filings that obtains the best results on
finer-139.333https://huggingface.co/nlpaueb/sec-bert-
base444https://huggingface.co/nlpaueb/sec-bert-
num555https://huggingface.co/nlpaueb/sec-bert-shape

##  2 Related Work

Entity extraction: xbrl tagging differs from ner and other previous entity
extraction tasks (Table 1), like contract element extraction Chalkidis et al.
(2019). Crucially, in xbrl tagging there is a much larger set of entity types
(6k in full xbrl, 139 in finer-139), most tagged tokens are numeric
($\sim$91%), and the correct tag highly depends on context. In most ner
datasets, numeric expressions are classified in generic entity types like
‘amount’ or ‘date’ (Bikel et al., 1999); this can often be achieved with
regular expressions that look for common formats of numeric expressions, and
the latter are often among the easiest entity types in ner datasets. By
contrast, although it is easy to figure out that the first three highlighted
expressions of Figure 1 are amounts, assigning them the correct xbrl tags
requires carefully considering their context. Contract element extraction
Chalkidis et al. (2019) also requires considering the context of dates,
amounts etc. to distinguish, for example, start dates from end dates, total
amounts from other mentioned amounts, but the number of entity types in
finer-139 is an order of magnitude larger (Table 1) and the full tag set of
xbrl is even larger (6k).

Financial ner: Previous financial ner applications use at most 9 (generic)
class labels. Salinas Alvarado et al. (2015) investigated ner in finance to
recognize organizations, persons, locations, and miscellaneous entities on 8
manually annotated sec financial agreements using crfs. Francis et al. (2019)
experimented with transfer learning by unfreezing different layers of a bilstm
with a crf layer, pre-trained on invoices, to extract 9 entity types with
distinct morphological patterns (e.g., iban, company name, date, total
amount). Also, Hampton et al. (2015, 2016) applied a Maximum Entropy
classifier, crfs, and handcrafted rules to London Stock Exchange filings to
detect 9 generic entity types (e.g., person, organization, location, money,
date, percentages). Finally, Kumar et al. (2016) extended the work of Finkel
et al. (2005) and built a financial entity recognizer of dates, numeric
values, economic terms in sec and non-sec documents, using numerous
handcrafted text features. By contrast, finer-139 uses a specialized set of
139 highly technical economic tags derived from the real-world need of xbrl
tagging, and we employ no handcrafted features.

Numerical reasoning: Neural numerical reasoning studies how to represent
numbers to solve numeracy tasks, e.g., compare numbers, understand
mathematical operations mentioned in a text etc. Zhang et al. (2020) released
numbert, a Transformer-based model that handles numerical reasoning tasks by
representing numbers by their scientific notation and applying subword
tokenization. On the other hand, genbert Geva et al. (2020) uses the decimal
notation and digit-by-digit tokenization of numbers. Both models attempt to
deal with the problem that word-level tokenization often turns numeric tokens
to oovs Thawani et al. (2021). This is important, because numerical reasoning
requires modeling the exact value of each numeric token. In finer-139, the
correct xbrl tags of numeric tokens depend much more on their contexts and
token shapes than on their exact numeric values (Fig. 1). Hence, these methods
are not directly relevant. genbert’s digit-by-digit tokenization would also
lead to excessive fragmentation, which we experimentally find to harm
performance.

##  3 Task and Dataset

Traditionally, business filings were simply rendered in plain text. Thus,
analysts and researchers needed to manually identify, copy, and paste each
amount of interest (e.g., from filings to spreadsheets). With xbrl-tagged
filings, identifying and extracting amounts of interest (e.g., to spreadsheets
or databases) can be automated. More generally, xbrl facilitates the machine
processing of financial documents. Hence, xbrl-tagged financial reports are
required in several countries, as already noted (Section 1). However, manually
tagging reports with xbrl tags is tedious and resource-intensive. Therefore,
we release finer-139 to foster research towards automating xbrl tagging.

Figure 2: Frequency distribution of the 139 xbrl tags used in this work over
the entire finer-139 dataset. Label indices shown instead of tag names to save
space.

finer-139 was compiled from approx. 10k annual and quarterly English reports
(filings) of publicly traded companies downloaded from sec’s edgar
system.666https://www.sec.gov/edgar/ The downloaded reports span a 5-year
period, from 2016 to 2020. They are annotated with xbrl tags by professional
auditors and describe the performance and projections of the companies. We
used regular expressions to extract the text notes from the Financial
Statements Item of each filing, which is the primary source of xbrl tags in
annual and quarterly reports.

xbrl taxonomies have many different attributes, making xbrl tagging
challenging even for humans Baldwin et al. (2006); Hoitash and Hoitash (2018).
Furthermore, each jurisdiction has its own xbrl taxonomy. Since we work with
us documents, our labels come from us-gaap.777www.xbrl.us/xbrl-
taxonomy/2020-us-gaap/ Since this is the first effort towards automatic xbrl
tagging, we chose to work with the most essential and informative attribute,
the tag names, which populate our label set. Also, since xbrl tags change
periodically, we selected the 139 (out of 6,008) most frequent xbrl tags with
at least 1,000 appearances in finer-139. The distribution of these tags seems
to follow a power law (Figure 2), hence most of the 6k xbrl tags that we did
not consider are very rare. We used the iob2 annotation scheme to distinguish
tokens at the beginning, inside, or outside of tagged expressions, which leads
to 279 possible token labels.

Subset | Sentences (S) | Avg. Tokens/S | Avg. Tags/S
---|---|---|---
Train | 900,384 | 44.7 $\pm$ 33.9 | 1.8 $\pm$ 1.2
Dev | 112,494 | 45.4 $\pm$ 35.9 | 1.7 $\pm$ 1.2
Test | 108,378 | 46.5 $\pm$ 38.9 | 1.7 $\pm$ 1.1

Table 2: finer-139 statistics, using spaCy’s tokenizer and the 139 tags of
this work ($\pm$ standard deviation).

We split the text notes into 1.8M sentences, the majority of which ($\sim$90%)
contained no tags.888We use nltk Bird et al. (2009) for sentence splitting.
The sentences are also html-stripped, normalized, and lower-cased. To avoid
conflating trivial and more difficult cases, we apply heuristic rules to
discard sentences that can be easily flagged as almost certainly requiring no
tagging; in a real-life setting, the heuristics, possibly further improved,
would discard sentences that do not need to be processed by the tagger. The
heuristic rules were created by inspecting the training subset and include
regular expressions that look for amounts and other expressions that are
typically annotated. Approx. 40% of the 1.8M sentences were removed,
discarding only 1% of tagged ones. We split chronologically the remaining
sentences into training, development, and test sets with an 80/10/10 ratio
(Table 2).

##  4 Baseline Models

spaCy (Honnibal et al., 2020) is an open-source nlp library.999We used spaCy
v.2.3; see https://spacy.io/. It includes an industrial ner that uses word-
level Bloom embeddings (Serrà and Karatzoglou, 2017) and residual
Convolutional Neural Networks (cnns) (He et al., 2016). We trained spaCy’s ner
from scratch on finer-139.

bilstm: This baseline uses a stacked bidirectional Long-Short Term Memory
(lstm) network (Graves et al., 2013; Lample et al., 2016) with residual
connections. Each token $t_{i}$ of a sentence $S$ is mapped to an embedding
and passed through the bilstm stack to extract the corresponding
contextualized embedding. A shared multinomial logistic regression (lr) layer
operates on top of each contextualized embedding to predict the correct label.
We use the word2vec embeddings (Mikolov et al., 2013a, b) of Loukas et al.
(2021).

bert: This is similar to bilstm, but now we fine-tune bert-base (Devlin et
al., 2019) to extract contextualized embeddings of subwords. Again, a
multinomial lr layer operates on top of the contextualized embeddings to
predict the correct label of the corresponding subword.

crfs: In this case, we replace the lr layer of the previous two models with a
Conditional Random Field (crf) layer (Lafferty et al., 2001), which has been
shown to be beneficial in several token labeling tasks (Huang et al., 2015;
Lample et al., 2016; Chalkidis et al., 2020b).101010We use a linear-chain crf
layer with log-likelihood optimization and Viterbi decoding.

##  5 Baseline Results

We report micro-$\mathrm{F_{1}}$ ($\mathrm{\muup}$-$\mathrm{F_{1}}$) and
macro-$\mathrm{F_{1}}$ ($\mathrm{m}$-$\mathrm{F_{1}}$) at the entity level,
i.e., if a gold tag annotates a multi-word span, a model gets credit only if
it tags the exact same span. This allows comparing more easily methods that
label words vs. subwords.

Table 3 shows that spaCy performs poorly, possibly due to the differences from
typical token labeling tasks, i.e., the large amount of entity types, the
abundance of numeric tokens, and the fact that in finer-139 the tagging
decisions depend mostly on context. Interestingly enough, bilstm (with word
embeddings) performs slightly better than bert. However, when a crf layer is
added, bert achieves the best results, while the performance of bilstm (with
word embeddings) deteriorates significantly, contradicting previous studies.

Baseline methods |  $\mathrm{\muup}$-$\mathrm{F_{1}}$ |  $\mathrm{m}$-$\mathrm{F_{1}}$
---|---|---
spaCy (words) | 48.6 $\pm$ 0.4 | 37.6 $\pm$ 0.2
bilstm (words) |  77.3 $\pm$ 0.6 |  73.8 $\pm$ 1.8
bilstm (subwords) | 71.3 $\pm$ 0.2 | 68.6 $\pm$ 0.2
bert (subwords) | 75.1 $\pm$ 1.1 | 72.6 $\pm$ 1.4
bilstm (words) + crf | 69.4 $\pm$ 1.2 | 67.3 $\pm$ 1.6
bilstm (subwords) + crf | 76.2 $\pm$ 0.2 | 73.4 $\pm$ 0.3
bert (subwords) + crf |  78.0 $\pm$ 0.5 |  75.2 $\pm$ 0.6

Table 3: Entity-level $\mathrm{\muup}$-$\mathrm{F_{1}}$ and
$\mathrm{m}$-$\mathrm{F_{1}}$ (%, avg. of 3 runs with different random seeds,
$\pm$ std. dev.) on test data.

We hypothesize that the inconsistent effect of crfs is due to tokenization
differences. When using bert’s subword tokenizer, there are more decisions
that need to be all correct for a tagged span to be correct (one decision per
subword) than when using word tokenization (one decision per word). Thus, it
becomes more difficult for subword models to avoid nonsensical sequences of
token labels, e.g., labeling two consecutive subwords as beginning and inside
of different entity types, especially given the large set of 279 labels (Table
1). The crf layer on top of subword models helps reduce the nonsensical
sequences of labels.

On the other side, when using words as tokens, there are fewer opportunities
for nonsensical label sequences, because there are fewer tokens. For instance,
the average number of subwords and words per gold span is 2.53 and 1.04,
respectively. Hence, it is easier for the bilstm to avoid predicting
nonsensical sequences of labels and the crf layer on top of the bilstm (with
word embeddings) has less room to contribute and mainly introduces noise
(e.g., it often assigns low probabilities to acceptable, but less frequent
label sequences). With the crf layer, the model tries to maximize both the
confidence of the bilstm for the predicted label of each word and the
probability that the predicted sequence of labels is frequent. When the bilstm
on its own rarely predicts nonsensical sequences of labels, adding the crf
layer rewards commonly seen sequences of labels, even if they are not the
correct labels, without reducing the already rare nonsensical sequences of
labels.

To further support our hypothesis, we repeated the bilstm experiments, but
with subword (instead of word) embeddings, trained on the same vocabulary with
bert. Without the crf, the subword bilstm performs much worse than the word
bilstm (6 p.p drop in $\mathrm{\muup}$-$\mathrm{F_{1}}$), because of the many
more decisions and opportunities to predict nonsensical label sequences. The
crf layer substantially improves the performance of the subword bilstm (4.9
p.p. increase in $\mathrm{\muup}$-$\mathrm{F_{1}}$), as expected, though the
word bilstm (without crf) is still better, because of the fewer opportunities
for nonsensical predictions. A drawback of crfs is that they significantly
slow down the models both during training and inference, especially when using
large label sets (Goldman and Goldberger, 2020), as in our case. Hence,
although bert with crf was the best model in Table 3, we wished to improve
bert’s performance further without employing crfs.

##  6 Fragmentation in bert

In finer-139, the majority (91.2%) of the gold tagged spans are numeric
expressions, which cannot all be included in bert’s finite vocabulary; e.g.,
the token ‘9,323.0’ is split into five subword units, [‘9’, ‘##,’, ‘##323’,
‘##.’, ‘##0’], while the token ‘12.78’ is split into [‘12’, ‘##.’, ‘##78’].
The excessive fragmentation of numeric expressions, when using subword
tokenization, harms the performance of the subword-based models (Table 3),
because it increases the probability of producing nonsensical sequences of
labels, as already discussed. We, therefore, propose two simple and effective
solutions to avoid the over-fragmentation of numbers.

Figure 3: xbrl tag predictions of bert (top), bert \+ [num] (middle), bert \+
[shape] (bottom) for the same sentence. bert tags incorrectly the amounts in
red. bert \+ [num] and bert \+ [shape] tag them more successfully (green
indicates correct tags).

bert \+ [num]: We detect numbers using regular expressions and replace each
one with a single [num] pseudo-token, which cannot be split. The pseudo-token
is added to the bert vocabulary, and its representation is learned during
fine-tuning. This allows handling all numeric expressions in a uniform manner,
disallowing their fragmentation.

bert \+ [shape]: We replace numbers with pseudo-tokens that cannot be split
and represent the number’s shape. For instance, ‘53.2’ becomes
‘$\mathrm{[XX.X]}$’, and ‘40,200.5’ becomes
‘$\mathrm{[XX}$,$\mathrm{XXX.X]}$’. We use 214 special tokens that cover all
the number shapes of the training set. Again, the representations of the
pseudo-tokens are fine-tuned, and numeric expressions (of known shapes) are no
longer fragmented. The shape pseudo-tokens also capture information about each
number’s magnitude; the intuition is that numeric tokens of similar magnitudes
may require similar xbrl tags. Figure 3 illustrates the use of [num] and
[shape] pseudo-tokens.

| development | test
---|---|---
|  $\mathrm{\muup}$-$\mathrm{P}$ |  $\mathrm{\muup}$-$\mathrm{R}$ |  $\mathrm{\muup}$-$\mathrm{F_{1}}$ |  $\mathrm{\muup}$-$\mathrm{P}$ |  $\mathrm{\muup}$-$\mathrm{R}$ |  $\mathrm{\muup}$-$\mathrm{F_{1}}$
bert | 74.9 $\pm$ 1.5 | 82.0 $\pm$ 1.3 | 78.2 $\pm$ 1.4 | 71.5 $\pm$ 1.1 | 79.6 $\pm$ 1.4 | 75.1 $\pm$ 1.1
bert \+ crf |  78.3 $\pm$ 0.8 |  83.6 $\pm$ 0.4 |  80.9 $\pm$ 0.3 |  75.0 $\pm$ 0.9 |  81.2 $\pm$ 0.2 |  78.0 $\pm$ 0.5
bert \+ [num] | 79.4 $\pm$ 0.8 |  83.0 $\pm$ 0.9 | 81.2 $\pm$ 0.9 | 76.0 $\pm$ 0.6 |  80.7 $\pm$ 0.8 | 78.3 $\pm$ 0.7
bert \+ [shape] |  82.1 $\pm$ 0.6 | 82.6 $\pm$ 0.4 |  82.3 $\pm$ 0.2 |  78.7 $\pm$ 0.5 | 80.1 $\pm$ 0.2 |  79.4 $\pm$ 0.2
fin-bert | 73.9 $\pm$ 1.3 | 81.4 $\pm$ 0.7 | 77.5 $\pm$ 1.0 | 70.2 $\pm$ 1.2 | 78.7 $\pm$ 0.7 | 74.0 $\pm$ 1.1
fin-bert \+ [num] | 81.1 $\pm$ 0.1 | 82.5 $\pm$ 1.2 | 81.8 $\pm$ 0.1 | 77.9 $\pm$ 0.1 | 79.9 $\pm$ 0.7 | 78.8 $\pm$ 0.3
fin-bert \+ [shape] |  82.3 $\pm$ 1.7 |  84.0 $\pm$ 1.2 |  83.2 $\pm$ 1.4 |  79.0 $\pm$ 1.6 |  81.2 $\pm$ 1.1 |  80.1 $\pm$ 1.4
sec-bert (ours) | 75.2 $\pm$ 0.4 | 82.7 $\pm$ 0.5 | 78.8 $\pm$ 0.1 | 71.6 $\pm$ 0.4 | 80.3 $\pm$ 0.5 | 75.7 $\pm$ 0.1
sec-bert-num (ours) | 82.5 $\pm$ 2.1 | 84.4 $\pm$ 1.2 | 83.7 $\pm$ 1.7 | 79.0 $\pm$ 1.9 | 82.0 $\pm$ 0.9 | 80.4 $\pm$ 1.4
sec-bert-shape (ours) |  84.8 $\pm$ 0.2 |  85.8 $\pm$ 0.2 |  85.3 $\pm$ 0.0 |  81.0 $\pm$ 0.2 |  83.2 $\pm$ 0.1 |  82.1 $\pm$ 0.1

Table 4: Entity-level micro-averaged $\mathrm{P}$, $\mathrm{R}$,
$\mathrm{F_{1}}$ $\pm$ std. dev. (3 runs) on the dev. and test data for bert-
based models.

##  7 In-domain Pre-training

Driven by the recent findings that pre-training language models on specialized
domains is beneficial for downstream tasks (Alsentzer et al., 2019; Beltagy et
al., 2019; Yang et al., 2020; Chalkidis et al., 2020b), we explore this
direction in our task which is derived from the financial domain.

fin-bert: We fine-tune fin-bert (Yang et al., 2020), which is pre-trained on a
financial corpus from sec documents, earnings call transcripts, and analyst
reports.111111We use the finbert-finvocab-uncased version from
https://github.com/yya518/FinBERT. The 30k subwords vocabulary of fin-bert is
built from scratch from its pre-training corpus. Again, we utilize fin-bert
with and without our numeric pseudo-tokens, whose representations are learned
during fine-tuning.

sec-bert: We also release our own family of bert models. Following the
original setup of Devlin et al. (2019), we pre-trained bert from scratch on
edgar-corpus, a collection of financial documents released by Loukas et al.
(2021). The resulting model, called sec-bert, has a newly created vocabulary
of 30k subwords. To further examine the impact of the proposed [num] and
[shape] special tokens, we also pre-trained two additional bert variants, sec-
bert-num and sec-bert-shape, on the same corpus, having replaced all numbers
by [num] or [shape] pseudo-tokens, respectively. In this case, the
representations of the pseudo-tokens are learned during pre-training and they
are updated during fine-tuning.

##  8 Improved bert Results

Table 4 reports micro-averaged precision, recall, and $\mathrm{F_{1}}$ on
development and test data. As with Table 3, a lr layer is used on top of each
embedding to predict the correct label, unless specified otherwise.

Focusing on the second zone, we observe that the [num] pseudo-token improves
bert’s results, as expected, since it does not allow numeric expressions to be
fragmented. The results of bert \+ [num] are now comparable to those of bert
\+ crf. Performance improves further when utilizing the shape pseudo-tokens
(bert \+ [shape]), yielding 79.4 $\mathrm{\muup}$-$\mathrm{F_{1}}$ and showing
that information about each number’s magnitude is valuable in xbrl tagging.

Interestingly, fin-bert (3rd zone) performs worse than bert despite its pre-
training on financial data. Similarly to bert, this can be attributed to the
fragmentation of numbers (2.5 subwords per gold tag span). Again, the proposed
pseudo-tokens ([num], [shape]) alleviate this problem and allow fin-bert to
leverage its in-domain pre-training in order to finally surpass the
corresponding bert variants, achieving an 80.1
$\mathrm{\muup}$-$\mathrm{F_{1}}$ test score.

Our new model, sec-bert (last zone), which is pre-trained on sec reports,
performs better than the existing bert and fin-bert models, when no numeric
pseudo-tokens are used. However, sec-bert is still worse than bert with
numeric pseudo-tokens (75.7 vs. 78.3 and 79.4 test
$\mathrm{\muup}$-$\mathrm{F_{1}}$), suffering from number fragmentation (2.4
subwords per gold tag span). sec-bert (without pseudo-tokens) also performs
worse than the bilstm with word embeddings (75.7 vs. 77.3
$\mathrm{\muup}$-$\mathrm{F_{1}}$, cf. Table 3). However, when the proposed
pseudo-tokens are used, sec-bert-num and sec-bert-shape achieve the best
overall performance, boosting the test $\mathrm{\muup}$-$\mathrm{F_{1}}$ to
80.4 and 82.1, respectively. This indicates that learning to handle numeric
expressions during model pre-training is a better strategy than trying to
acquire this knowledge only during fine-tuning.

##  9 Additional Experiments

###  9.1 Subword pooling

An alternative way to bypass word fragmentation is to use subword pooling for
each word. Ács et al. (2021) found that for ner tasks, it is better to use the
_first_ subword only, i.e., predict the label of an entire word from the
contextualized embedding of its first subword only; they compared to several
other methods, such as using only the _last_ subword of each word, or
combining the contextualized embeddings of all subwords with a self-attention
mechanism. Given this finding, we conducted an ablation study and compare (i)
our best model (sec-bert) with _first_ subword pooling (denoted sec-bert-
first) to (ii) sec-bert with our special tokens (sec-bert-num, sec-bert-
shape), which avoid segmenting numeric tokens.

Table 5 shows that, in xbrl tagging, using the proposed special tokens is
comparable (sec-bert-num) or better (sec-bert-shape) than performing _first_
pooling (sec-bert-first). It might be worth trying other pooling strategies as
well, like last-pooling or subword self-attention pooling. It’s worth noting,
however, that the latter will increase the training and inference times.

|  $\mathrm{\muup}$-$\mathrm{F_{1}}$ |  $\mathrm{m}$-$\mathrm{F_{1}}$
---|---|---
sec-bert | 78.8 $\pm$ 0.1 | 72.6 $\pm$ 0.4
sec-bert-first | 79.9 $\pm$ 1.2 | 77.1 $\pm$ 1.7
sec-bert-num | 80.4 $\pm$ 1.4 | 78.9 $\pm$ 1.3
sec-bert-shape |  82.1 $\pm$ 0.1 |  80.1 $\pm$ 0.2

Table 5: Entity-level $\mathrm{\muup}$-$\mathrm{F_{1}}$ and
$\mathrm{m}$-$\mathrm{F_{1}}$ (%, avg. of 3 runs with different random seeds,
$\pm$ std. dev.) on test data using different ways to alleviate fragmentation.

###  9.2 Subword bilstm with [num] and [shape]

To further investigate the effectiveness of our pseudo-tokens, we incorporated
them in the bilstm operating on subword embeddings (3rd model of Table 3).
Again, we replace each number by a single [num] pseudo-token or one of 214
[shape] pseudo-tokens, for the two approaches, respectively. These
replacements also happen when pre-training word2vec subword embeddings; hence,
an embedding is obtained for each pseudo-token.

Table 6 shows that bilstm-num outperforms the bilstm subword model. bilstm-
shape further improves performance and is the best bilstm subword model
overall, surpassing the subword bilstm with crf, which was the best subword
bilstm model in Table 3. These results further support our hypothesis that the
[num] and [shape] pseudo-tokens help subword models successfully generalize
over numeric expressions, with [shape] being the best of the two approaches,
while also avoiding the over-fragmentation of numbers.

|  $\mathrm{\muup}$-$\mathrm{F_{1}}$ |  $\mathrm{m}$-$\mathrm{F_{1}}$
---|---|---
bilstm (subwords) | 71.3 $\pm$ 0.2 | 68.6 $\pm$ 0.2
bilstm (subwords) + crf | 76.2 $\pm$ 0.2 | 73.4 $\pm$ 0.3
bilstm-num (subwords) | 75.6 $\pm$ 0.3 | 72.7 $\pm$ 0.4
bilstm-shape (subwords) |  76.8 $\pm$ 0.2 |  74.1 $\pm$ 0.3

Table 6: Entity-level $\mathrm{\muup}$-$\mathrm{F_{1}}$ and
$\mathrm{m}$-$\mathrm{F_{1}}$ (%, avg. of 3 runs with different random seeds,
$\pm$ std. dev.) on test data for bilstm models with [num] and [shape] tokens.

###  9.3 A Business Use Case

Since xbrl tagging is derived from a real-world need, it is crucial to analyze
the model’s performance in a business use case. After consulting with experts
of the financial domain, we concluded that one practical use case would be to
use an xbrl tagger as a recommendation engine that would propose the $k$ most
probable xbrl tags for a specific token selected by the user. The idea is that
an expert (e.g., accountant, auditor) knows beforehand the token(s) that
should be annotated and the tagger would assist by helping identify the
appropriate tags more quickly. Instead of having to select from several
hundreds of xbrl tags, the expert would only have to inspect a short list of
$k$ proposed tags.

Figure 4: $\mathrm{Hits@}k$ results (%, avg. of 3 runs with different random
seeds) on test data, for different $k$ values. Standard deviations were very
small and are omitted.

We evaluate our best model, sec-bert-shape, in this use case using
$\mathrm{Hits@}k$. We use the model to return the $k$ most probable xbrl tags
for each token that needs to be annotated, now assuming that the tokens to be
annotated are known. If the correct tag is among the top $k$, we increase the
number of hits by one. Finally, we divide by the number of tokens to be
annotated. Figure 4 shows the results for different values of $k$. The curve
is steep for $k=1$ to $5$ and saturates as $k$ approaches 10, where
$\mathrm{Hits@}k$ is nearly perfect (99.4%). In practice, this means that a
user would have to inspect 10 recommended xbrl tags instead of hundreds for
each token to be annotated; and in most cases, the correct tag would be among
the top 5 recommended ones.

###  9.4 Error Analysis

We also performed an exploratory data and error analysis to unveil the
peculiarities of finer-139, extract new insights about it, and discover the
limitations of our best model. Specifically, we manually inspected the errors
of sec-bert-shape in under-performing classes (where $\mathrm{F_{1}}<50\%$)
and identified three main sources of errors.

Specialized terminology: In this type of errors, the model is able to
understand the general financial semantics, but does not fully comprehend
highly technical details. For example, Operating Lease Expense amounts are
sometimes missclassified as Lease And Rental Expense, i.e., the model manages
to predict that these amounts are about expenses in general, but fails to
identify the specific details that distinguish operating lease expenses from
lease and rental expenses. Similarly, Payments to Acquire Businesses (Net of
Cash Acquired) amounts are mostly misclassified as Payments to Acquire
Businesses (Gross). In this case, the model understands the notion of business
acquisition, but fails to differentiate between net and gross payments.

Financial dates: Another interesting error type is the misclassification of
financial dates. For example, tokens of the class Debt Instrument Maturity
Date are mostly missclassified as not belonging to any entity at all (‘O’
tag). Given the previous type of errors, one would expect the model to
missclassify these tokens as a different type of financial date, but this is
not the case here. We suspect that errors of this type may be due to
annotation inconsistencies by the financial experts.

Annotation inconsistencies: Even though the gold xbrl tags of finer-139 come
from professional auditors, as required by the Securities & Exchange
Commission (sec) legislation, there are still some discrepancies. We provide
an illustrative example in Figure 5. We believe that such inconsistencies are
inevitable to occur and they are a part of the real-world nature of the
problem.

Figure 5: A manually inspected sentence from finer-139 showing some
inconsistencies in the gold xbrl tags of the auditors. The green ‘1’ is
correctly annotated with the xbrl tag Lessee Operating Lease Term Of Contract.
The red ‘16’ should have also been annotated with the same tag, but is not,
possibly because the annotator thought the (same) tag was obvious. The orange
numbers ‘0.1’ and ‘6’ lack xbrl annotations; they should have both been
annotated as Lessee Operating Lease Renewal Term. We can only speculate that
the auditor might not have been aware that there is an xbrl tag for lease
renewal terms, in which case the recommendation engine of Section 9.3 might
have helped.

We hope that this analysis inspires future work on xbrl tagging. For example,
the specialized terminology and financial date errors may be alleviated by
adopting hierarchical classifiers Chalkidis et al. (2020a); Manginas et al.
(2020), which would first detect entities in coarse classes (e.g., expenses,
dates) and would then try to classify the identified entities into finer
classes (e.g., lease vs. rent expenses, instrument maturity dates vs. other
types of dates). It would also be interesting to train classifiers towards
detecting wrong (or missing) gold annotations, in order to help in quality
assurance checks of xbrl-tagged documents.

##  10 Conclusions and Future Work

We introduced a new real-word nlp task from the financial domain, xbrl
tagging, required by regulatory commissions worldwide. We released finer-139,
a dataset of 1.1M sentences with xbrl tags. Unlike typical entity extraction
tasks, finer-139 uses a much larger label set (139 tags), most tokens to be
tagged are numeric, and the correct tag depends mostly on context rather than
the tagged token. We experimented with several neural classifiers, showing
that a bilstm outperforms bert due to the excessive numeric token
fragmentation of the latter. We proposed two simple and effective solutions
that use special tokens to generalize over the shapes and magnitudes of
numeric expressions. We also experimented with fin-bert, an existing bert
model for the financial domain, which also benefits from our special tokens.
Finally, we pre-trained and released our own domain-specific bert model, sec-
bert, both with and without the special tokens, which achieves the best
overall results with the special tokens, without costly crf layers.

In future work, one could hire experts to re-annotate a subset of the dataset
to measure human performance against the gold tags. Future work could also
consider less frequent xbrl tags (few- and zero-shot learning) and exploit the
hierarchical dependencies of xbrl tags, possibly with hierarchical
classifiers, building upon our error analysis.

## References

  * Ács et al. (2021) Judit Ács, Ákos Kádár, and Andras Kornai. 2021.  Subword pooling makes a difference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2284–2295, Online.
  * Akhtar et al. (2017) Md Shad Akhtar, Abhishek Kumar, Deepanway Ghosal, Asif Ekbal, and Pushpak Bhattacharyya. 2017.  A multilayer perceptron based ensemble technique for fine-grained financial sentiment analysis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 540–546, Copenhagen, Denmark.
  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, USA.
  * Baldwin et al. (2006) Amelia Annette Baldwin, Carol E. Brown, and Brad S. Trinkle. 2006.  Xbrl: An impacts framework and research challenge.  _Journal of Emerging Technologies in Accounting_ , 3:97–116.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China.
  * Bikel et al. (1999) Daniel M. Bikel, Richard Schwartz, and Ralph M. Weischedel. 1999.  An algorithm that learns what’s in a name.  _Machine Learning_ , 34(1-3):211–231.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural Language Processing with Python_ , 1st edition.  O’Reilly Media, Inc.
  * Chalkidis et al. (2020a) Ilias Chalkidis, Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020a.  An empirical study on large-scale multi-label text classification including few and zero-shot labels.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020_ , pages 7503–7515.
  * Chalkidis et al. (2020b) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020b.  LEGAL-BERT: The muppets straight out of law school.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904, Online.
  * Chalkidis et al. (2019) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019.  Neural contract element extraction revisited.  In _Workshop on Document Intelligence at NeurIPS 2019_.
  * Chen et al. (2020) Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen, editors. 2020\.  _Proceedings of the Second Workshop on Financial Technology and Natural Language Processing_. Kyoto, Japan.
  * Chen et al. (2019) Deli Chen, Shuming Ma, Keiko Harimoto, Ruihan Bao, Qi Su, and Xu Sun. 2019.  Group, extract and aggregate: Summarizing a large amount of finance news for forex movement prediction.  In _Proceedings of the Second Workshop on Economics and Natural Language Processing_ , pages 41–50, Hong Kong.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal.
  * Ein-Dor et al. (2019) Liat Ein-Dor, Ariel Gera, Orith Toledo-Ronen, Alon Halfon, Benjamin Sznajder, Lena Dankin, Yonatan Bilu, Yoav Katz, and Noam Slonim. 2019.  Financial event extraction using Wikipedia-based weak supervision.  In _Proceedings of the Second Workshop on Economics and Natural Language Processing_ , pages 10–15, Hong Kong.
  * El-Haj et al. (2020) Dr Mahmoud El-Haj, Dr Vasiliki Athanasakou, Dr Sira Ferradans, Dr Catherine Salzedo, Dr Ans Elhag, Dr Houda Bouamor, Dr Marina Litvak, Dr Paul Rayson, Dr George Giannakopoulos, and Nikiforos Pittaras, editors. 2020.  _Proceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation_. COLING, Barcelona, Spain (Online).
  * Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005.  Incorporating non-local information into information extraction systems by Gibbs sampling.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 363–370, Ann Arbor, Michigan.
  * Francis et al. (2019) Sumam Francis, Jordy Van Landeghem, and Marie-Francine Moens. 2019.  Transfer learning for named entity recognition in financial and biomedical documents.  _Information_ , 10(8):248.
  * Geva et al. (2020) Mor Geva, Ankit Gupta, and Jonathan Berant. 2020.  Injecting numerical reasoning skills into language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 946–958.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , volume 9 of _Proceedings of Machine Learning Research_ , pages 249–256, Chia Laguna Resort, Italy.
  * Goldman and Goldberger (2020) Eran Goldman and Jacob Goldberger. 2020.  Crf with deep class embedding for large scale classification.  _Computer Vision and Image Understanding_ , 191:102865.
  * Graves et al. (2013) Alex Graves, Abdel rahman Mohamed, and Geoffrey E. Hinton. 2013.  Speech recognition with deep recurrent neural networks.  _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 6645–6649.
  * Hahn et al. (2019) Udo Hahn, Véronique Hoste, and Zhu Zhang, editors. 2019.  _Proceedings of the Second Workshop on Economics and Natural Language Processing_. Association for Computational Linguistics, Hong Kong.
  * Hampton et al. (2016) Peter Hampton, Hui Wang, William Blackburn, and Zhiwei Lin. 2016.  Automated sequence tagging: Applications in financial hybrid systems.  In _Research and Development in Intelligent Systems XXXIII_ , pages 295–306. Springer.
  * Hampton et al. (2015) Peter John Hampton, Hui Wang, and William Blackburn. 2015.  A hybrid ensemble for classifying and repurposing financial entities.  In _Research and Development in Intelligent Systems XXXII_ , pages 197–202. Springer.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Hoitash and Hoitash (2018) Rani Hoitash and Udi Hoitash. 2018.  Measuring accounting reporting complexity with xbrl.  _The Accounting Review_ , 93:259–287.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _CoRR_ , abs/1508.01991.
  * Izumi and Sakaji (2019) Kiyoshi Izumi and Hiroki Sakaji. 2019.  Economic causal-chain search using text mining technology.  In _Proceedings of the First Workshop on Financial Technology and Natural Language Processing_ , pages 61–65, Macao, China.
  * Jacobs et al. (2018) Gilles Jacobs, Els Lefever, and Véronique Hoste. 2018.  Economic event detection in company-specific news text.  In _Proceedings of the First Workshop on Economics and Natural Language Processing_ , pages 1–10, Melbourne, Australia.
  * Kim et al. (2003) J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003.  Genia corpus–a semantically annotated corpus for bio-textmining.  _Bioinformatics_ , 19:i180–i182.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kumar et al. (2016) Aman Kumar, Hassan Alam, Tina Werner, and Manan Vyas. 2016.  Experiments in candidate phrase selection for financial named entity extraction - a demo.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations_ , pages 45–48, Osaka, Japan.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _Proceedings of the Eighteenth International Conference on Machine Learning_ , ICML ’01, page 282–289, San Francisco, USA.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270, San Diego, California.
  * Loukas et al. (2021) Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021.  EDGAR-CORPUS: Billions of tokens make the world go round.  In _Proceedings of the Third Workshop on Economics and Natural Language Processing_ , pages 13–18, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Malo et al. (2014) Pekka Malo, Ankur Sinha, Pekka J. Korhonen, Jyrki Wallenius, and Pyry Takala. 2014\.  Good debt or bad debt: Detecting semantic orientations in economic texts.  _J. Assoc. Inf. Sci. Technol._ , 65(4):782–796.
  * Manginas et al. (2020) Nikolaos Manginas, Ilias Chalkidis, and Prodromos Malakasiotis. 2020.  Layer-wise guided training for BERT: Learning incrementally refined document representations.  In _Proceedings of the Fourth Workshop on Structured Prediction for NLP_ , pages 53–61, Online. Association for Computational Linguistics.
  * Mikolov et al. (2013a) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013_ , Scottsdale, USA.
  * Mikolov et al. (2013b) Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013_ , pages 3111–3119, Lake Tahoe, United States.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes.  In _Joint Conference on EMNLP and CoNLL - Shared Task_ , pages 1–40, Jeju Island, Korea.
  * Salinas Alvarado et al. (2015) Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015.  Domain adaption of named entity recognition to support credit risk assessment.  In _Proceedings of the Australasian Language Technology Association Workshop 2015_ , pages 84–90, Parramatta, Australia.
  * Serrà and Karatzoglou (2017) Joan Serrà and Alexandros Karatzoglou. 2017.  Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks.  In _Proceedings of the Eleventh ACM Conference on Recommender Systems_ , page 279–287, New York, USA.
  * Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012.  Practical bayesian optimization of machine learning algorithms.  In _Advances in Neural Information Processing Systems_ , volume 25.
  * Tabari et al. (2018) Narges Tabari, Piyusha Biswas, Bhanu Praneeth, Armin Seyeditabari, Mirsad Hadzikadic, and Wlodek Zadrozny. 2018.  Causality analysis of Twitter sentiments and stock market returns.  In _Proceedings of the First Workshop on Economics and Natural Language Processing_ , pages 11–19, Melbourne, Australia.
  * Thawani et al. (2021) Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021.  Representing numbers in NLP: a survey and a vision.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–656, Online.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of CoNLL-2003_ , pages 142–147. Edmonton, Canada.
  * Wang et al. (2013) Chuan-Ju Wang, Ming-Feng Tsai, Tse Liu, and Chin-Ting Chang. 2013.  Financial sentiment analysis for risk prediction.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 802–808, Nagoya, Japan.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online.
  * Yang et al. (2019) Linyi Yang, Ruihai Dong, Tin Lok James Ng, and Yang Xu. 2019.  Leveraging BERT to improve the FEARS index for stock forecasting.  In _Proceedings of the First Workshop on Financial Technology and Natural Language Processing_ , pages 54–60, Macao, China.
  * Yang et al. (2020) Yi Yang, Mark Christopher Siy UY, and Allen Huang. 2020.  Finbert: A pretrained language model for financial communications.  _arXiv_ , abs/2006.08097.
  * Zhai and Zhang (2019) Shuang (Sophie) Zhai and Zhu (Drew) Zhang. 2019.  Forecasting firm material events from 8-k reports.  In _Proceedings of the Second Workshop on Economics and Natural Language Processing_ , pages 22–30, Hong Kong.
  * Zhang et al. (2020) Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020.  Do language embeddings capture scales? In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4889–4896, Online.

##  Appendix A Experimental Setup

For spaCy, we followed the recommended
practices.121212https://spacy.io/usage/v2-3. All other methods were
implemented in tensorflow.131313https://www.tensorflow.org/ Concerning bert
models, we used the implementation of huggingface (Wolf et al., 2020). We also
use Adam (Kingma and Ba, 2015), Glorot initialization (Glorot and Bengio,
2010), and the categorical cross-entropy loss.

Hyper-parameters were tuned on development data with Bayesian Optimization
Snoek et al. (2012) monitoring the development loss for 15 trials.141414We
used keras tuner (https://keras-team.github.io/keras-
tuner/documentation/tuners/) For the bilstm encoders, we searched for
$\\{1,2,3\\}$ hidden layers, $\\{128,200,256\\}$ hidden units,
$\\{1e$-$3,2e$-$3,3e$-$3,4e$-$3,5e$-$3\\}$ learning rate, and
$\\{0.1,0.2,0.3\\}$ dropout. We trained for $30$ epochs using early stopping
with patience $4$. For bert, we used grid-search to select the optimal
learning rate from $\\{1e$-$5,2e$-$5,3e$-$5,4e$-$5,5e$-$5\\}$, fine-tuning for
$10$ epochs, using early stopping with patience $2$. All final hyper-
parameters are shown in Table 7. Training was performed mainly on a dgx
station with 4 nvidia v100 gpus and an Intel Xeon cpu e5-2698 v4 @ 2.20ghz.

| $\mathit{Params}$ | $L$ | $U$ | $P_{\mathrm{drop}}$ | $\mathit{LR}$
---|---|---|---|---|---
bilstm (words) | 21M | 2 | 128 | 0.1 | 1e-3
bilstm (subwords) | 8M | 1 | 256 | 0.2 | 1e-3
bilstm (words) + crf | 21M | 2 | 128 | 0.1 | 1e-3
bilstm (subwords) + crf | 8M | 1 | 256 | 0.2 | 1e-3
bilstm-num (subwords) | 1M | 1 | 256 | 0.2 | 1e-3
bilstm-shape (subwords) | 0.8M | 2 | 128 | 0.1 | 1e-3
bert | 110M | - | - | - | 1e-5
bert \+ [num] | 110M | - | - | - | 1e-5
bert \+ [shape] | 110M | - | - | - | 1e-5
bert \+ crf | 110M | - | - | - | 1e-5
fin-bert | 110M | - | - | - | 2e-5
fin-bert \+ [num] | 110M | - | - | - | 2e-5
fin-bert \+ [shape] | 110M | - | - | - | 2e-5
sec-bert | 110M | - | - | - | 1e-5
sec-bert-num | 110M | - | - | - | 1e-5
sec-bert-shape | 110M | - | - | - | 1e-5

Table 7: Number of total parameters ($\mathit{Params}$) and the best hyper-
parameter values for each method; i.e., number of recurrent layers ($L$),
number of recurrent units ($U$), dropout probability $P_{\mathrm{drop}}$,
learning rate ($\mathit{LR}$).

##  Appendix B Additional Results

Table 9 shows micro-averaged Precision, Recall, and F1 for the development and
test data, using all baseline methods. The macro-averaged scores are similar
and we omit them for brevity. Using a logistic regression (lr) classification
layer, bilstm (words) surpasses bert both in Precision and F1 score. However,
when using a crf layer on top, bert outperforms bilstm (words) in all
measures.

Table 9 shows the micro-averaged Precision, Recall, and $\mathrm{F_{1}}$ for
the development and test data using the bilstm models operating on subwords
with the proposed tokenizations. [num] and [shape] tokens help the model to
bypass the word fragmentation problem, increasing its scores in all metrics.

| development | test
---|---|---
|  $\mathrm{\muup}$-$\mathrm{P}$ |  $\mathrm{\muup}$-$\mathrm{R}$ |  $\mathrm{\muup}$-$\mathrm{F_{1}}$ |  $\mathrm{\muup}$-$\mathrm{P}$ |  $\mathrm{\muup}$-$\mathrm{R}$ |  $\mathrm{\muup}$-$\mathrm{F_{1}}$
spaCy | 38.2 $\pm$ 0.4 | 58.2 $\pm$ 0.8 | 46.1 $\pm$ 0.1 | 40.8 $\pm$ 0.8 | 60.0 $\pm$ 0.4 | 48.6 $\pm$ 0.4
bilstm (words) | 78.6 $\pm$ 2.4 | 80.3 $\pm$ 1.2 | 79.4 $\pm$ 1.0 | 75.4 $\pm$ 1.9 | 78.0 $\pm$ 0.8 | 77.3 $\pm$ 0.6
bilstm (subwords) | 73.4 $\pm$ 0.1 | 77.2 $\pm$ 0.0 | 75.2 $\pm$ 0.1 | 68.8 $\pm$ 0.2 | 74.1 $\pm$ 0.2 | 71.3 $\pm$ 0.2
bert (subwords) | 74.9 $\pm$ 1.5 | 82.0 $\pm$ 1.3 | 78.2$\pm$ 1.4 | 71.5 $\pm$ 1.1 | 79.6 $\pm$ 1.4 | 75.1 $\pm$ 1.1
bilstm (words) + crf | 73.4 $\pm$ 2.0 | 69.3 $\pm$ 0.9 | 71.3 $\pm$ 1.2 | 70.9 $\pm$ 1.8 | 68.0 $\pm$ 0.9 | 69.4 $\pm$ 1.2
bilstm (subwords) + crf |  80.0 $\pm$ 0.3 | 78.7 $\pm$ 0.5 | 79.3 $\pm$ 0.4 |  76.5 $\pm$ 0.2 | 76.0 $\pm$ 0.2 | 76.2 $\pm$ 0.2
bert (subwords) + crf | 78.3 $\pm$ 0.8 |  83.6 $\pm$ 0.4 |  80.9 $\pm$ 0.3 | 75.0 $\pm$ 0.9 |  81.2 $\pm$ 0.2 |  78.0 $\pm$ 0.5

Table 8: Entity-level micro-averaged P, R, F1 $\pm$ std. dev. (3 runs) on the
dev. and test data for our baselines.

| development | test
---|---|---
|  $\mathrm{\muup}$-$\mathrm{P}$ |  $\mathrm{\muup}$-$\mathrm{R}$ |  $\mathrm{\muup}$-$\mathrm{F_{1}}$ |  $\mathrm{\muup}$-$\mathrm{P}$ |  $\mathrm{\muup}$-$\mathrm{R}$ |  $\mathrm{\muup}$-$\mathrm{F_{1}}$
bilstm (subwords) | 73.4 $\pm$ 0.1 | 77.2 $\pm$ 0.0 | 75.2 $\pm$ 0.1 | 68.8 $\pm$ 0.2 | 74.1 $\pm$ 0.2 | 71.3 $\pm$ 0.2
bilstm (subwords) + crf | 80.0 $\pm$ 0.3 | 78.7 $\pm$ 0.5 | 79.3 $\pm$ 0.4 | 76.5 $\pm$ 0.2 | 76.0 $\pm$ 0.2 | 76.2 $\pm$ 0.2
bilstm-num (subwords) | 77.9 $\pm$ 0.4 | 78.6 $\pm$ 0.7 | 78.2 $\pm$ 0.6 | 74.8 $\pm$ 0.2 | 76.5 $\pm$ 0.5 | 75.6 $\pm$ 0.3
bilstm-shape (subwords) |  81.1 $\pm$ 0.1 |  81.5 $\pm$ 0.3 |  81.3 $\pm$ 0.2 |  77.5 $\pm$ 0.3 |  78.7 $\pm$ 0.5 |  78.1 $\pm$ 0.4

Table 9: Entity-level micro-averaged P, R, F1 $\pm$ std. dev. (3 runs) on the
dev. and test data for the bilstm models using the [num] and [shape] tokens.