# An advanced combination of semi-supervised Normalizing Flow & Yolo (YoloNF)
to detect and recognize vehicle license plates.

Khalid OUBLAL${}^{1\corr}$ Xinyi DAI 2<EMAIL_ADDRESS>1Corresponding author: Department of Computer Science, Ecole Polytechnique -
IP Paris 2Department of Computer Science, Ecole Polytechnique - IP Paris

###### Abstract

Fully Automatic License Plate Recognition (ALPR) has been a frequent research
topic due to several practical applications. However, many of the current
solutions are still not robust enough in real situations, commonly depending
on many constraints. This paper presents a robust and efficient ALPR system
based on the state-of-the-art YOLO object detector and Normalizing flows. The
model uses two new strategies: Firstly, a two-stage network using YOLO and a
normalization flow-based model for normalization to detect Licenses Plates
(LP) and recognize the LP with numbers and Arabic characters. Secondly, Multi-
scale image transformations are implemented to provide a solution to the
problem of the YOLO cropped LP detection including significant background
noise. Furthermore, extensive experiments are led on a new dataset with
realistic scenarios, we introduce a larger public annotated dataset collected
from Moroccan plates. We demonstrate that our proposed model can learn on a
small number of samples free of single or multiple characters. The dataset
will also be made publicly available to encourage further studies and research
on plate detection and recognition.

Keywords: Normalizing Flows, Multi-scale detection, Semi-supervised training,
Automatic License Plate Recognition (ALPR)

##  1 Overview

Intelligent transportation is one of the main areas where artificial
intelligence has gained importance. Automatic License Plate Recognition (ALPR)
is an important task in smart transportation and surveillance, which has many
practical applications such as automatic traffic enforcement, such as the
detection of stolen vehicles or toll violations, traffic flow control, etc.

The ALPR problem can be divided into three sub-tasks: license plate detection,
license plate segmentation, and character recognition. These subtasks
constitute the common pipeline of ALPR systems found in the literature [11,
12, 32], and many works focus on only one or two of these subtasks [28, 2, 30,
3]. One of our main contributions is to introduce a new combination of
normalizing flows and YOLO as object detectors which supersedes the
segmentation and character recognition part, making the task becomes a simple
multi-scale labeling problem.

Our work is based on a normalizing flow model, proposed in 2020 by [14] [14],
this model uses a latent space of normalizing flow representing a normal
sample’s feature distribution. Unlike other generative models such as
variational autoencoder (VAE) [16] and GANs [7], the flow based generator
makes the bijective mapping between feature space and latent space assigned to
a likelihood.

Normalizing Flows (NF) are a network that can generate complex distributions
by transforming a probability density from a latent space through a series of
invertible affine transformations of <<flows>> and was introduced by [19][19].
Based on the change of variable rule, the bijective mappings between feature
space and latent space can be evaluated in both directions. The formula is
derived as Eq.1. Thus a scoring function can be derived to decide if an image
contains an anomaly or not. As a result, most common samples will have a high
likelihood, while uncommon images will have a lower likelihood.

$p_{i}(\textbf{z}_{i})=p_{i-1}(f_{i}^{-1}(\textbf{z}_{i}))\left|\det\dfrac{df_{i}^{-1}}{d\textbf{z}_{i}}\right|$
(1)

$\text{where }\textbf{z}_{i}=f_{i}(\textbf{z}_{i-1})\textbf{, thus
}\mathbf{z}_{i-1}=f_{i}^{-1}(\textbf{z}_{i})\sim p_{i-1}(\textbf{z}_{i-1})$

This model aims to find the best probability distribution $P_{z}(z)$ in the
latent space Z to maximize likelihoods for extracted features X. According to
the change-of-variables formula extended to the function 1, the following
steps are illustrated by the equation (Eq.2). After adding log function on
both sides, the loss function can be defined as (Eq.3).

$\begin{split}p_{i}(\textbf{z}_{i})&=p_{i-1}(f_{i}^{-1}(\textbf{z}_{i}))\left|\det\dfrac{df_{i}^{-1}}{d\textbf{z}_{i}}\right|\\\
&=p_{i-1}(\textbf{z}_{i-1})\left|\det{\Big{(}\dfrac{df_{i}}{d\textbf{z}_{i-1}}\Big{)}^{-1}}\right|\\\
&=p_{i-1}(\textbf{z}_{i-1}){\left|\det\dfrac{df_{i}}{d\textbf{z}_{i-1}}\right|^{-1}}\\\
\log p_{i}(\textbf{z}_{i})&=\log
p_{i-1}(\textbf{z}_{i-1})-\log\left|\det\dfrac{df_{i}}{d\textbf{z}_{i-1}}\right|\end{split}$
(2)

$\log p_{i}(\textbf{z}_{i})=\log
p_{i-1}(\textbf{z}_{i-1})-\log\left|\det\dfrac{df_{i}}{d\textbf{z}_{i-1}}\right|$
(3)

Figure 1: Composition schemes for affine coupling layers proposed by [9][9].

YOLO or You Only Look Once Unified and real-time object detection presented in
2016 by [27][27] reframes the object detection as a regression problem that
spatially separates bounding boxes and associates their class probabilities. A
single convolutional neural network is used to predict bounding boxes and
class probabilities in the YOLO system. With a given image as input, the
system first divides the image into an S x S grid. Each cell predicts B
bounding boxes and their corresponding confidence scores. The confidence score
is defined as $Pr(Class_{i})\times IOU^{truth}_{b-box}$, where the
intersection on the union (IOU) between the ground truth and the predicted
bounding box is calculated. Then, the conditional class probabilities are
multiplied by the confidence scores of each bounding box to obtain confidence
scores for a specific class. In this paper, we are interested in this YOLO
structure and we will detail all the main elements to combine this structure
with the other Normalizing Flows model structure which uses a coupling layer
architecture as proposed in Real-NVP [9]. The layout of a single block is
given in Figure 1.

In this paper, we focus on the YOLO neural network and we will detail all the
main elements to combine this network with the Normalizing Flows network. For
this purpose, this paper is organized as follows: firstly we briefly review
related work in Section.2. Then, the Moroccan ALPR dataset with the new
database is presented in Section.3. Section.4 presents the novel combination
of YOLO and flow normalization which builds the proposed ALPR system. We
present and discuss the results of our experiments in section.5. Conclusions
and future work are presented in Section.6.

##  2 Related works

Multiple ALPR methods have been tested recently on detection and recognition
of LPs, however the use of deep learning as a tool for ALPR targetting
Moroccan LPs has not been sufficiently explored. In [30], authors have
attempted a hybrid model based on morphological extraction and use an optical
character recognition OCR phase as [11] which is based on Tesseract [3].

For this purpose, we briefly review several recent works that use Deep
Learning approaches in the context of ALPR very similar to Moroccan licenses
plates. Specifically, we discuss work related to each stage of ALPR.

###  2.1 License Plates detection

End-to-end deep learning approaches for license plate recognition include [32,
28], which detect proposed license plate regions and perform the final
selection using a CNN as a binary classifier. As proposed by [5], the CNN is
trained for the entire character sequence to detect and recognize LPs with
ConvNet-RNN netowrk first[6].

Using such techniques, it is impossible to separate license plates from other
general alphanumeric text, such as when the vehicle image is taken in a severe
weather scenario or a simple area that is not centered on the image while
other text of the same shape, such as an advertisement, is in the scope of
primary view. In one instance, the fixed width of the bounding box limits the
application to standard license plates. [32] applied a visual attention model
to detect license plates containing blue and yellow regions of interest. The
eventual classification was completed by a CNN model that mapped the offset
using the Support Vector Machine (SVM) model. The results were not as expected
because the SVM model significantly limits some LP types and makes the method
more sensitive to illumination and scale variations. Another solution proposed
in [18] which consisted in generating the license plates with a vertical Sobel
filter and applying a binary CNN for the final verification. However, the
results of this approach failed to handle the actual negative cases caused by
Sobel due to the high noise on the plate images.

[12] proposed another method to meet the specific needs of Moroccan license
plates using the small version of YOLO, this method present a relevant results
in terms of detection but it remains very sensitive to the variation of class
type since it is trained directly on regular Moroccan plates while it fails on
the plates of trucks or service vehicles since the position of the LPs is not
on the large ground despite the fact that the content is the same. Moreover
when a several of vehicles appear in the same frame the network does not
detect all the plates and the ALPR interested only on the one which present in
the large ground, this is due probably to the Tiny-YOLOv3 Layers which reduces
the number of convolutional layers so the features are extracted only by using
a small number of 1×1 and 3×3 convolutional layers.

###  2.2 Licenses Plates Recognition & Segmentation

For the recognition process, the studies we consulted often use networks that
can convert a text-image into editable text. For the recognition process, we
used an attention sequence network, which takes the rectified image as input
and predicts the sequence of characters of the license plate. Their network
consists of an encoder and a decoder. The encoder uses a convolutional layer
stack (ConvNet) that extracts the characters to improve detection against
other test approaches.
[21][21]proposed using random CNNs to extract features. Using random CNNs to
extract features for character recognition achieved much better performance
than using image pixels or learning filter weights by backpropagation. On the
other hand, [4] proposed to perform china characters recognition only as a
sequence of Multi-scale labeling problem, which is not far from the approach
of [12] that also renders it as a character-level classification problem. A
recurrent neural network (RNN) with connectionist temporal classification
(CTC) is used to label the sequence data, thus recognizing the entire LP
without character-level segmentation. The latter, however, is sensitive to
variance, especially since it is a problem of labeling on images correlated
with the plates, so very small for which the neural network type RCNN[17] or
Fast RCNN[13], YOLO-tiny easily fails as a result of missing of more features
as well as the gathering of Arabic characters types diverges significantly
this problem.

###  2.3 Remarks

To summarize our remarks, these multiple methods cited their experience on
Moroccan LP datasets that present both Arabic and Latin alpha-numeric
characters. Making it difficult to accurately evaluate the methods presented.
Moreover, the one using Yolo-tiny [12] presents large difficulties, which make
the ALPR system fails in the global classification of characters, this is due
to the sensitivity of the model to the noise since this detection and
recognition is performed only on one-stage way which increases the chances of
losing the confidence rate. In addition to the uses of small CNN penalize the
detection of small objects, such as the characters. This problem can be
improved by using a normalizing flow approach if we adapt the density
estimation using REAL NVP approach proposed by [10][10] to our case.

To that effect we use an improved YOLO-v3 object detection at each step [31]
to create a robust and efficient end-to-end ALPR system. In addition, we
augment and transform the data for character recognition, as this step is the
main problem of some ALPR systems including that of [12].

##  3 The ALPR dataset

The public ALPR Moroccan dataset contains 657 images taken inside a vehicle in
irregular traffic or parking at the urban environment. We also add to this,
some samples from the public dataset of UPM6 [1] as well as the HDR dataset
dataset LPs [24] which is not far from the characteristics of the Moroccan LPs
for the new cars only (present of ww, M characters) and the Arabic characters
extracted from the Arabian open source database in order to perform the
characteristics engineering step with consistency. We have more than $1,200$
images and more than $1800$ images of sample characters generated by cropping
from the Moroccan dataset and other from HDR dataset and generated images
following the proposed method of [15] using GAN models . Approximately 85%
cars, 10% motorcycles with new and old plates and approximately 5% trucks. In
Morocco, the LPs have a slight variation in size and variations in color
(black, white, red) depending on the type of vehicle and its category e.g.
official or public LP service cars are often black and red. In general,LP cars
have a size of 40cm $\times$ 13cm, while motorcycles have 20cm $\times$ 17cm .
Figure.2 shows some of the different types of LPs in the dataset.

Figure 2: Samples from Dataset of vehicles Figure 3: Samples from Dataset of
characters

The data set is divided as follows: 40% for training, 40% for testing and 20%
for validation. The distribution of the dataset was done in such a way that
each split has the same number of images obtained, taking into account the
type and position of the vehicle, the color and characters of the PL vehicle’s
. In order to facilitate the interpretation, we associate to each character a
Latin label and for the numbers the numbers themselves, and if there is a
Latin character we took it itself.

Figure 4: Latin labels attributed to each class of the dataset characters
(Figure.3).

In Morocco, LPs is different from license plates which use Latin letters such
as the European ones. There is a small similarity between Moroccan plates and
some that use Arabic or Persian letters such as Algeria, Arabic Saudi,
Emirate..or Pakistanis plates.
Moroccan LPs consist of 3 parts separated by two verticals lines, in the first
part on the left there are five numbers that identify the vehicle
$XXXXX\in\\{0\to 99999\\}$, in the middle an Arabic characters and on the
right one or two numbers $YY\in\\{0\to 99\\}$ that represent the prefecture or
the province where the vehicle has been registered: For example, 38 for
<<Ouarzazate>> province, the plate can be typed as $XXXXX-a-38$ or for the
capital province <<Rabat>> $XXXXX-a-1$.

The rule that we have just explained is not satisfied across the entire
dataset, there are some older vehicles with plates that start with Arabic
characters, then the number that refers to the province of registration and
then the vehicle number, this example is shown in Figure 6 .

Therefore, the digits numbers {5,6,8} and the characters {a,b,d,h} have many
more samples than the others, as shown in Figure.5 which refer this problem of
ALPR to unbalanced data which must be taken into account during the
implementation of our approach.

Figure 5: Characters distribution in the new Morocco-ALPR dataset Enhanced by
other examples from the HDR LPs datasets[24] of the same type, compared to the
one provided to the challenge [23] and [22]. Figure 6: Illustration of the
different images License plates and their appearance ( the right image begin
with Arabic character while the other doesn’t, and the image of the motorcycle
has no character Arabic, and has only numbers.

##  4 Proposed ALPR approach

The proposed approach is based on 5 subsections: Plate detection, distribution
verification of the detected LPs, then we move to characters detection and
again to the cross-checking and linking two types of classification results of
the detected LPs and finally to character recognition. For visualization model
we also introduced to plot predicted bounding boxes for each plate and the
predicted characters result on each image (Figure.7).

  1. 1.

The proposed model takes vehicle images as input and then using YOLOv3 (Yolo-
LP), trained on our custom dataset, to detect and draw bounding boxes on each
plate.

  2. 2.

An extraction and Transformation block (CT-A) is created to crop the plate
images based on the bounding boxes drawn by the preview step. Both of the
cropped plates images and the original images will be saved. The
transformation model is further introduced to rotate and crop each plates
image’s edges to remove the background noises surrounding the plate.

  3. 3.

The processed plates images are then passed into the normalizing flow model
(NF-A) to generate a normal distribution by maximum likelihood training only
on plate (e.g semi-supervised training). A scoring function is used to compute
likelihood function to classify the input samples as plates or not this step
improves the precision of the detection. The unit (ST-A) allows to keep only
the detection with the largest confidence in cases where more than one Plate
is detected by introducing threshold value $\theta$. We have found that the
most optimal value of $\theta$ is $0.1$.

  4. 4.

A second block of Yolo-v3 (Yolo Characters) which trained only on character
dataset (this is not available from the ANRT, for that we have created own
labelling ) to detect and draw bounding boxes on each characters included
numbers. The second block (CT-B) provide provide samples of cropped and
transformed characters;

  5. 5.

The block (NF-B), will decide if a character is present on the distribution
built by the Normalization-Flow trained only on the 19 classes of characters,
The (CT-B) will decide according to the score result if there is a real
correlation between the character and the ones seen during the training step,
this is performed by the scoring function $\theta$ if this score is much
lower, this means that the character corresponds to a class, this one is the
class detected with the greatest probability, this promotes the use of the
Softmax function to retrieve the most likely class for a character combine
with the score results of Normalizing flow.

The robustness of this approach is not only due to the fact that the Flow
Normalizing can check the class detected by linking it to the scoring
function, but rather the possibility of transforming the samples avoid errors
in detection and classification, which by default is based only on confidence.

Figure 7: Global Proposed ALPR Approach (YoloFN)

###  4.1 Detect vehicles Plates by Yolo-v3 and Normalizing Flow

####  4.1.1 Block Yolo-V3

In order to use YOLO to detect LP for the first stage or character in the
fourth stage, we need to change the number of filters in the last
convolutional layer to match the number of classes. YOLO uses A anchor boxes
to predict bounding boxes, in our case we proposed $A=5$ with four coordinates
$(x,y,w,h)$, confidence and C class probabilities, so the number of filters is
given by:

$FILTERS=(C+5)\times A$ (4)

For the second time we also thought to mix two models, one that detects 2
classes (vehicle, motorcycles), which is already trained on coco dataset. For
training YOLOv3 and Fast-YoLo we used pretrained weights on ImageNet [26], and
another that only detects the plate, but most of our images the full vehicle
structure is not taken, which makes this idea useless. Therefore we will have
only one class which is the Licenses Plates, below the description of
filters=$(1+5)\times 5=30$ that we used.

Figure 8: The network used for LP detection. There are 30 filters in the last
layer to detect a single class.

####  4.1.2 Block Normalizing Flow & Transformation

We used model Different [14] which is a state-of-art model that utilizes a
latent space of normalizing flow to represent normal samples’ feature
distribution, which is in our case LPs, and for the second stage of our
pipeline, features are digits, Arabic characters. To improve the performance
of the model on the output images from Yolo, we propose an image
transformation model to rotate and crop each LP image. In training, various
scales of cropping on LPs images are performed to ease background noise
interference. Moreover, the range of rotation for input images is reduced from
180 degrees to 30 or 20 degrees for better computing performance. Then the
transformed images are fed into a pre-trained AlexNet[8] to extract the
feature. The extracted feature map is further passed into a normalizing flow-
based on coupling layer (Figure 1) to output a normal distribution of LPs by
maximum likelihood training.

As [14][14] proposed, we used the calculated likelihoods as a criterion to
classify a sample as an authentic plate (respectively character) if it fits
with the distribution generated from the learned distributors or it is not. To
get a robust score $\tau(x)$, we average the negative $log-likelihoods$ using
multiple transformations $T_{i}(X)\in T$ of an image $X$:

$\tau(X)=E_{T_{i\in T}}(-\log p_{Z}(f_{NF}(f_{ex}(T_{k}(X)))))$ (5)

Here, we have not provided for any adjustments,[14] has T chosen rotations and
manipulations of brightness and contrast (which is consistent with our color
image dataset and is sensitive to variations in contrast and brightness since
most images are taken during the day). An image is classified as LP
respectively characters if the score $\tau(X)$ is greater than the threshold
value $\theta$.

To classify if an input image is Licenses Plates (or character in the case of
Normalizing Flow Characters) or not, we will use the [14] structure which uses
a scoring function that compute the average of the negative log-likelihoods
using multiple transformations $T_{k}(X)$ of an image $X$.

$L(\mathcal{D})=-\frac{1}{|\mathcal{D}|}\sum_{\mathbf{X}\in\mathcal{D}}\log
p(\mathbf{X})$ (6)

Figure 9: A simple illustration of Flow Normalizing on a samples

###  4.2 Detection and Recognition of Characters

For the detection and recognition of characters, we built a new network
inspired on the YOLOv3 architecture (e.g as the first stage), with fundamental
technical differences to accommodate 19 classes: 10 numerical digits, 5 Arabic
characters, ($0,\to,9$) and (a, b, d, h, j, m, p, waw, ww), consequently, we
have to define filters=(19+5)x5=120. Please refer to Section 4.1.1 and
Table.4.2.

The input image is a 240 x 80 color LP patches, and the output of 30 x 10
guaranteed enough horizontal granularity for the 19 characters (digits and
Arabic characters).  Layer Filters Size Input Output conv 32 3 × 3 / 1 240 ×
80 × 3 240 × 80 × 32 max 2 × 2 / 2 240 × 80 × 32 120 × 40 × 32 conv 64 3 × 3 /
1 120 × 40 × 32 120 × 40 × 64 max 2 × 2 / 2 120 × 40 × 64 60 × 20 × 64 conv
128 3 × 3 / 1 60 × 20 × 64 60 × 20 × 128 conv 64 1 × 1 / 1 60 × 20 × 128 60 ×
20 × 64 conv 128 3 × 3 / 1 60 × 20 × 64 60 × 20 × 128 max 2 × 2 / 2 60 × 20 ×
128 30 × 10 × 128 conv 256 3 × 3 / 1 30 × 10 × 128 30 × 10 × 256 conv 128 1 ×
1 / 1 30 × 10 × 256 30 × 10 × 128 conv 256 3 × 3 / 1 30 × 10 × 128 30 × 10 ×
256 conv 512 3 × 3 / 1 30 × 10 × 256 30 × 10 × 512 conv 256 1 × 1 / 1 30 × 10
× 512 30 × 10 × 256 conv 512 3 × 3 / 1 30 × 10 × 256 30 × 10 × 512 conv 120 1
× 1 / 1 30 × 10 × 512 30 × 10 × 80 detection

Finally, we perform a post-processing step in order to swap highly confused
letters to digits and digits to letters, since the Brazilian license plate is
formed by exactly three letters followed by four digits.

As we mentioned before, a Moroccan LPs is different from license plates which
use Latin letters such as European. There is a small similarity between
Moroccan plates and some that use Arabic or Farsi letters such as Algeria,
Arabic Saudi, Emirate or Pakistanis LPs which we have refereed on section 2.

Since our evaluation only takes the new version of the name (XXXXXX-character-
XX) where X is digit number ($0\to 9$), there is an important post-processing
step in order to sort the plates correctly.

Understanding the specific layout of the Moroccan plate, we sort the detected
characters by their horizontal position for the vehicles. The left (four to
six characters) of the Arabic letter is the vehicle number ($0\to 99999$) as
format XXXXX, then the right of it is the prefecture/province number ($0\to
99$) as XX. Otherwise, if the left side is empty, then the number of the
prefecture/province is on the right side and all characters at the bottom of
the Arabic character correspond to the number of vehicles. When there is no
Arabic character, the sorting list is defined as default (from left to right
and from top to bottom).

##  5 Experiments & results

All experiments were performed on an NVIDIA Titan XP GPU (3,840 CUDA cores and
32 GB of RAM) using the Darknet [25] framework and different [14] implemented
on PyTorch.

We consider as correct only detection with IoU $\geq$ 0.5. This value was
chosen after several trials to improve the performance of our model on the
validation dataset. In addition, the following parameters were used for
training the networks: 80k iterations (max batches) and learning
rate$=[10^{-3},10^{-4},10{-5}]$ with batch size of 64 and 1000 iterations.

###  5.1 Evaluation metric

  * •

mAP & loss Likelihood metric: As we have well specified concerning the
Normalizing Flow network, its evaluation metric consists mainly on the
likelihood function, so to evaluate the training of Yolo in the two steps of
our ALPR system, we have considered Recall & precision curve which allows us
to have an access to the IOU and the threshold value that we must apply to
increase the precision.

The Mean Average Precision (mAP) is more accurate than F1-scores because it
takes into account the precision and recall relationship in a holistic manner.
The intuitive use of this metric is related to the official metric used in
[6].

  * •

Intersection over Union (IoU): To train an object detection model, usually,
there are 2 inputs:An image and ground-truth bounding boxes for each object in
the image. The model will predicts the bounding boxes of the detected objects.
It is expected that the predicted box will not match exactly the ground-truth
box

$IoU=\frac{\text{Intersection area}}{\text{Union area}}$ (7)

  * •

Recall & Precision

$\text{Recall}=\frac{TP}{TP+FN}\text{ , }\hskip
28.45274pt\text{Precision}=\frac{TP}{TP+FP}$ (8)

  * •

mAP formula:

$mAP=\frac{1}{N}\sum_{k=1}^{k=N}AP_{k}$ (9)

Where the $AP_{k}$ is the Average precision of class $k$ and $N$ is the number
of classes.

  * •

Levenshtein distance metric: Our last evaluation is mainly related to the
string similarity between string LP detected and the annotated label on the
validation dataset. In order to achieve this, we use Levenshtein distance like
Hamming distance, is the smallest number of edit operations required to
transform one string into the other. Unlike Hamming distance, the set of edit
operations also includes insertions and deletions, thus allowing us to compare
strings of different lengths. This metric appears robust and easy to
implement. In table we show the average error created by the proposed method
compared to other methods given in the state-of-the-art of the report.

###  5.2 Training and hyperparameters

Since we have a large number of trainable parameters, we must use techniques
to reduce overfitting. First, The Darknet model used an exponential decay of
the learning rate that we set to our desired values, to ensure that the change
in parameters will be small over time. In addition, we used L1 and L2
regularization in the dense layers, as they disperse the error terms across
all weights, leading to more accurate final models. To achieve this, the model
was based on DarkNet-53. Third, we used an early stop callback to stop the
training when the validation no longer improves based on the validation loss.

The training model converges after $80$ iterations and is stable for the rest
of the training phase. The average validation loss is $0.21$.

Figure 10: Training and validation loss of Yolo-LP (Improved version of
Yolo-v3). Figure 11: Training and validation loss of Yolo-Characters (Improved
version of Yolo-v3).

As the dataset we are using an imbalanced data some classes have a really
multiples samples (around of 20%) while more than 90% of the other classes
have a relatively low samples. To avoid this issue we using the label
smoothing which convert our one-hot encoded labels to a smooth probability
distribution [33].

Figure 12: The mAP and Average Levenshtein distance as a function of theta
threshold values tested to find the optimal value.

As well as training with the characters provided in the training set, we also
augment the data in two ways. Firstly, we create negative images to simulate
characters from other vehicle categories, and then we also verify which
characters can be rotated horizontally and vertically to create new cases.

###  5.3 Evaluation on the ALPR Dataset

For LP detection, we find that in the most challenging images, e.g.,
motorcycle images, the vehicle’s LP is not entirely within the predicted
bounding box, requiring a margin change of about 5% in the validation set for
the entire LP to be entirely within the predicted bounding box of the vehicle.
Therefore, we use a margin of 15% in the test set and in the training of the
LP detection Yolov3. The first stage, i.e. the YOLO model dedicated to LP
detection Yolo-LP at a recall rate obtained is 0.9813 which correspond to
1178/1,200 images. We were not able to detect the LP in a single vehicle in
only 22 images, because a false positive was predicted with a higher
confidence+score function $\tau(image)$ than the real LP.

Therefore, we used the Yolo-characters results to perform post-processing in
cases where more than one LP is detected, for example: evaluate on each
detected LP if there are at least $4$ characters or consider only the LP where
the confidence of the characters is higher and the likelihood scoring function
is minimal (e.g $\tau(X)$). However, since the actual LP can be detected with
very low confidence levels (i.e., $\leq$ 0.2 to 0.1), some false negatives
should be analyzed. To understand what the network is seeing we used
GradCAM[29] implemented at the output of Yolo and block Normalizing Flow.

Results obtained and the Levenshtein distance score in each ALPR stage in the
Moroccan APLR dataset (Validation set). The Recall stands for detection and
segmentation, and mAP stands for recognition.  ALPR Recall/mAP Average
Levenshtein distance Vehicle Detection. 1 License Plate Detection 0.9871 Tiny
Character Detection 0.9675 0.9654 Yolo Character Recognition 0.9683 Vehicle
Detection. - License Plate Detection 0.98 Open Character Detection(Seg) 94.75
1.4654 ALPR Character Recognition 92.83 Vehicle Detection. 1 YoloFN License
Plate Detection 0.9813 (our Character Detection 0.9863 0.4654 proposed)
Character Recognition 0.9871

Figure 13: Result of the Grad-Cam on license plate detection, on the
validation set id:439, 440, and 443. Figure 14: Result of the Grad-Cam on
license plate detection, on the validation set id:36_126 and 39_120

For the character detection and classification part (stage 2) In the
validation set, a margin of 20% is required for each cropped detection of LP
in order to contain all its characters. Although this margin adds background
noise in the LPs patches, the model is robust to this noise because at the
output of block flow (NF-B, Figure.7) normalizing with a threshold $\theta$ of
0.1 manages to detect the characters cleanly. The recall obtained is 0.9863
and 0.9854 considering the whole dataset test. This is much higher than the
[12] method, which is explained by the fact that block Flow normalizing cross-
checking class by class and the sum of the scores evokes a low error on the
classification. Moreover, the transformation at block (CT-B, Figure.7) allows
to better check the class of a given character, especially the similarity
between the class {h} and {d} in some cases as well as the class {1} and {a}.
Figure.15 illustrates the mean average when training the openALPR and the
Yolo-Tiny models for character detection. Our proposed model obtain a better
results in the validation set of APLR Morocco dataset, our model achieved good
score of recall, mAP, and also minimize the average Levenshtein distance (ALD)
compared to other methods, this but we note that the detection of the licenses
at the level of stage 1 there is a similarity between Yolo-tiny with a slight
advantage of the latter, but this is explained by our choice close to a theta
that reduces ALD, however the detection of the characters, our model is
significantly better than openAPLR and Yolotiny.

Figure.9 (Appendix 2) shows some LPs of different categories correctly
detected despite the tilt of the plate or the weather that degrades the
quality of the images.

Figure 15: Proposed System mAP and training batch curve in comparison to other
methods Tiniy-Yolo [12] and OpenALPR used on Saudi vehicles [2]

The table 5.3, gives the average error of this distance on which the APLR
system provides at the output a string of predict characters were it’s
compared to target label. Note that for our YoloFN model we changed several
values of theta at the threshold part ST-B of YoloFN $\theta\in[0.6to0.01]$ to
achieve an optimal score of ALD (Figure.12), and the corresponding value of
theta is $\theta=0.1$ this value was chosen to calculate the error on the
final ALD on test dataset.

##  6 Conclusions & future works

In this paper, we introduce a new combination in cascade between two neural
networks as well as a new dataset that contain the Moroccan license plates.
This dataset presents several challenges related to types of LPs, background
noise and the environment where the image was taken. Moreover, the method
proposed a detection of the LPs and then cross checking if it is the class LP
and not something else, then the network detects again the characters and the
recognition of the class is done by combining the confidence and the error on
the distribution with the normalizing flow model. The result showed that the
model increases more than 2.3% of its mAP and reduces more than 3% of the
errors on the predicted string of the LP if compared with a YOLO without the
normalizing flow blocks. We also compared our results on the same validation
set with the OpenAPLR method that uses OCR and the recognition improvement
exceeds 7.1%.

Several improvements of this model are also possible despite the chosen value
of $\theta$ (theta), which corresponds to the minimum of the Levenshtein
distance, this value does not correlate to the maximum mAP which means that we
have lost more than 1.2% of mAP to satisfy the criterion of Average
Levenshtein distance. Adding more layers to the bock flow normalization could
solve this particular issue and find a most efficient value of $\theta$.
Finally, we conclude that based on these extensive experiments this proposed
approach is robust for the detection of LPs and it can be extended to a real
time implementation with an FPS around $30Fps$ (using GPU).

This work can be further extended to tackle the detection of small overlapping
objects in real time in more challenging contexts. For example, the proposed
solution can be further tuned to detect pedestrians in challenging scenes
subject to extreme conditions such as storms and snow. It can be used in
search and rescue operations to save lives.

##  7 Acknowledgments

This work was supported by a prize from the National Telecommunications
Regulatory Agency (ANRT) and the Nvidia Deep Learning Institute. The author
would like to thank MoroccoAI conference data challenge team for the
coordination and organization.

<<Please note that this research study is a result of an ALPR framework that
we propose as a solution to the problem of LP detection and recognition for
vehicles in Morocco (Data challenge conference of December 2021), we have
tried to detail as much as possible each step, however due to constraints
related to the submission deadline we have not been able to explore in depth
some parts including the implementation of Grad-Cam on a YOlo, this will be
published as soon as possible. However, I am available on
<EMAIL_ADDRESS>to answer for any question related to this
framework. I would like to thank the MoroccoAI conference organizing team, the
ANRT and Nvidia for this real challenge.>>

## References

  * [1] Abdelkrim Alahyane, Mohamed El Fakir, Saad Benjelloun and Ikram Chairi  “Open data for Moroccan license plates for OCR applications: data collection, labeling, and model construction”  In _arXiv preprint arXiv:2104.08244_ , 2021
  * [2] Syed Talha Abid Ali et al.  “Mobile Registration Number Plate Recognition Using Artificial Intelligence”  In _2021 IEEE International Conference on Image Processing (ICIP)_ , 2021, pp. 944–948  IEEE
  * [3] Horacio Cachinho  “Reconnaissance d’éléments textuels ou graphiques dans les documents juridiques numérisés”, 2017
  * [4] Shyang-Lih Chang, Li-Shien Chen, Yun-Chung Chung and Sei-Wan Chen  “Automatic license plate recognition”  In _IEEE transactions on intelligent transportation systems_ 5.1 IEEE, 2004, pp. 42–53
  * [5] Teik Koon Cheang, Yong Shean Chong and Yong Haur Tay  “Segmentation-free vehicle license plate recognition using ConvNet-RNN”  In _arXiv preprint arXiv:1701.06439_ , 2017
  * [6] Teik Koon Cheang, Yong Shean Chong and Yong Haur Tay  “Segmentation-free vehicle license plate recognition using ConvNet-RNN”  In _arXiv preprint arXiv:1701.06439_ , 2017
  * [7] Antonia Creswell et al.  “Generative adversarial networks: An overview”  In _IEEE Signal Processing Magazine_ 35.1 IEEE, 2018, pp. 53–65
  * [8] Ling Ding et al.  “Alexnet feature extraction and multi-kernel learning for object-oriented classification”  In _Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci_ 42, 2018, pp. 277–281
  * [9] Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio  “Density estimation using real nvp”  In _arXiv preprint arXiv:1605.08803_ , 2016
  * [10] Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio  “Density estimation using real nvp”  In _arXiv preprint arXiv:1605.08803_ , 2016
  * [11] Shaimaa Ahmed Elsaid et al.  “Arabic Real-Time License Plate Recognition System”  In _Advances in Data Science, Cyber Security and IT Applications_ Cham: Springer International Publishing, 2019
  * [12] Abdelhak Fadili, Mohamed El Aroussi and Youssef Fakhri  “A one-stage modified Tiny-YOLOv3 method for Real time Moroccan license plate recognition”  In _International Journal of Computer Science and Information Security (IJCSIS)_ 19.7, 2021
  * [13] Ross Girshick  “Fast r-cnn”  In _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448
  * [14] Ulrika E Hallberg and Wilmar B Schaufeli  ““Same same” but different? Can work engagement be discriminated from job involvement and organizational commitment?”  In _European psychologist_ 11.2 Hogrefe & Huber Publishers, 2006, pp. 119–127  URL: https://arxiv.org/pdf/2008.12577.pdf
  * [15] Byung-Gil Han, Jong Taek Lee, Kil-Taek Lim and Doo-Hyun Choi  “License Plate Image Generation using Generative Adversarial Networks for End-To-End License Plate Character Recognition from a Small Set of Real Images”  In _Applied Sciences_ 10.8, 2020  DOI: 10.3390/app10082780
  * [16] Xianxu Hou, Linlin Shen, Ke Sun and Guoping Qiu  “Deep feature consistent variational autoencoder”  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2017, pp. 1133–1141  IEEE
  * [17] Zhongjie Huang et al.  “Recognition of vehicle-logo based on faster-RCNN”  In _International Conference On Signal And Information Processing, Networking And Computers_ , 2018, pp. 75–83  Springer
  * [18] Vishal Jain et al.  “Deep automatic license plate recognition system”  In _Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing_ , 2016, pp. 1–8
  * [19] Mahdi Karami et al.  “Invertible convolutional flow”, 2019
  * [20] HDR Khalid Oublal Moroccan Artificial Intelligence Society MSDA-UPM6 open dataset  “Dataset Morocco ALPR, Created on Data-challenge which addresses to resolve the problem of ANPR in Morocco licensed vehicles.”, http://malpr.ml/, 2021
  * [21] David Menotti, Giovani Chiachia, Alexandre X Falcao and VJ Oliveira Neto  “Vehicle license plate recognition with random convolutional networks”  In _2014 27th SIBGRAPI conference on graphics, patterns and images_ , 2014, pp. 298–303  IEEE
  * [22] MoroccoAI  “NARSA,The National Road Safety Agency”, 2021  URL: http://narsa.ma/fr/accueil
  * [23] MoroccoAI  “This data-challenge addresses the problem of ANPR in Morocco licensed vehicles.”, 2021  URL: https://www.morocco.ai/events/conferences/MoroccoAI-conf-2021/pages/datachallenge.html
  * [24] A. R.  “Holistic Recognition of Low Quality License Plates (HDR dataset)”  URL: https://medusa.fit.vutbr.cz/traffic/research-topics/general-traffic-analysis/holistic-recognition-of-low-quality-license-plates-by-cnn-using-track-annotated-data-iwt4s-avss-2017/
  * [25] Joseph Redmon  “Darknet: Open Source Neural Networks in C”, http://pjreddie.com/darknet/, 2016
  * [26] Joseph Redmon and Ali Farhadi  “YOLOv3: An Incremental Improvement”  In _arXiv_ , 2018
  * [27] Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi  “You only look once: Unified, real-time object detection”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788
  * [28] Muhammad Sarfraz, Mohammed Jameel Ahmed and Syed A Ghazi  “Saudi Arabian license plate recognition system”  In _2003 International Conference on Geometric Modeling and Graphics, 2003. Proceedings_ , 2003, pp. 36–41  IEEE
  * [29] Ramprasaath R Selvaraju et al.  “Grad-CAM: Why did you say that?”  In _arXiv preprint arXiv:1611.07450_ , 2016
  * [30] Fatimazahra TAKI, Abdelbaki ELBELRHITI ELALAOUI and Younes CHEDDAD  “Moroccan License Plate recognition using a hybrid method and license plate features”
  * [31] Yunong Tian et al.  “Apple detection during different growth stages in orchards using the improved YOLO-V3 model”  In _Computers and electronics in agriculture_ 157 Elsevier, 2019, pp. 417–426
  * [32] Umair Yousaf et al.  “A Deep Learning Based Approach for Localization and Recognition of Pakistani Vehicle License Plates”  In _Sensors_ 21.22 Multidisciplinary Digital Publishing Institute, 2021, pp. 7696
  * [33] Zhi Zhang et al.  “Bag of freebies for training object detection neural networks”  In _arXiv preprint arXiv:1902.04103_ , 2019

##  1 APPENDIX A: Annotation of new dataset Moroccan ALPR-Plus

We note that a new dataset [20] of annotated images has been introduced in
order to increase the dataset of the Moroccan ALPR, moreover this dataset
contains several levels of labeling (vehicles, plates care, and characters)
and is being performed using VOCA format. We have injected some images from
HDR dataset[24] and others was generated by using LP-GAN model proposed by
[15] [15].

  * •

The final injection conditions:

    * –

If all the characters in the plate are numbers or digits with a character
included in the list of classes available on the original Moroccan ALPR
dataset.

    * –

Respect the order and the colors of most of the plates by adding only white
(or black plates written in gray or white).

##  2 APPENDIX C: Examples of LPs detection and recognition success and
failures cases

Figure 16: Plates Recognize: 828h6, Plates target: 8428h8, Levenshtein
distance=2 & Ratio Levenshtein=0.73 Figure 17: Plates Recognize: 52547b1,
Plates target: 52547b1, Levenshtein distance=0 & Ratio Levenshtein=1.00 Figure
18: Plates Recognize: 56362a7, Plates target: 56362a7, Levenshtein distance=0
& Ratio Levenshtein=1.00 Figure 19: Plates Recognize: 11002571, Plates target:
11002577, Levenshtein distance =1 & Ratio Levenshtein=0.88 Figure 20: Plates
Recognize: 87004b8, Plates target: 87004b8 , Levenshtein distance =0 & Ratio
Levenshtein=1.00


*[enuminline,1]label=(),

# A Large-Scale Comparison of Historical Text Normalization Systems

Marcel Bollmann
Department of Computer Science
University of Copenhagen, Denmark
<EMAIL_ADDRESS>

###### Abstract

There is no consensus on the state-of-the-art approach to historical text
normalization. Many techniques have been proposed, including rule-based
methods, distance metrics, character-based statistical machine translation,
and neural encoder–decoder models, but studies have used different datasets,
different evaluation methods, and have come to different conclusions. This
paper presents the largest study of historical text normalization done so far.
We critically survey the existing literature and report experiments on eight
languages, comparing systems spanning all categories of proposed normalization
techniques, analysing the effect of training data quantity, and using
different evaluation methods. The datasets and scripts are made publicly
available.

##  1 Introduction111This work largely builds upon the author’s doctoral
thesis (Bollmann, 2018), the research for which was carried out at Ruhr-
Universität Bochum, Germany.

Spelling variation is one of the key challenges for NLP on historical texts,
affecting the performance of tools such as part-of-speech taggers or parsers
and complicating users’ search queries on a corpus. _Normalization_ is often
proposed as a solution; it is commonly defined as the mapping of historical
variant spellings to a single, contemporary “normal form” as exemplified in
Figure 1.

Automatic normalization of historical texts has a long history, going back to
at least Fix (1980). Earlier approaches often rely on hand-crafted algorithms
tailored to one specific language, while more recent approaches have focused
on supervised machine learning, particularly character-based statistical
machine translation (SMT) and its neural equivalent (NMT). However, no clear
consensus has emerged about the state of the art for this task, with papers
either reporting an advantage for NMT (Hämäläinen et al., 2018), SMT (Domingo
and Casacuberta, 2018), or language-specific algorithms (Schneider et al.,
2017). Moreover, the quantity of annotated training data varies considerably
between studies, making it difficult to obtain practical recommendations for
new projects seeking to use normalization techniques.

theirthairthairethearthayrtheairetheiaretheireþerþereþairtheyrthirtherthar
Figure 1: Historical text normalization exemplified: mapping variant spellings
from historical English texts to their normalization ‘their’

#### Contributions

This paper aims to provide the most comprehensive evaluation and analysis of
historical text normalization systems so far. Motivated by a systematic review
of previous work on this topic (Sec. 2), only publicly available normalization
systems covering a wide range of proposed techniques are selected (Sec. 3) and
evaluated across a diverse collection of historical datasets covering eight
languages (Sec. 4). This is followed by a detailed analysis of the effect of
training data quantity and a critical discussion of evaluation methods for
assessing normalization quality (Sec. 5).

The datasets and code are made freely available whenever
possible,333https://github.com/coastalcph/histnorm; one dataset could not be
included due to licensing restrictions. along with detailed instructions on
how to reproduce the experiments.

##  2 A Brief Survey of Automatic Historical Text Normalization

The following overview is broadly organized by categories that each represent
a conceptually or methodically different approach.

###  2.1 Substitution Lists

The conceptually simplest form of normalization is to look up each historical
variant in a pre-compiled list that maps it to its intended normalization.
This approach can go by many names, such as _lexical substitution_ ,
_dictionary lookup_ , _wordlist mapping_ , or _memorization_. While it does
not generalize in any way to variants that are not covered by the list, it has
proven highly effective as a component in several normalization systems, such
as the semi-automatic VARD tool (Rayson et al., 2005; Baron and Rayson, 2008)
or the fully automatic Norma tool (Bollmann, 2012).

###  2.2 Rule-based Methods

Rule-based approaches try to encode regularities in spelling variants—e.g.,
historical $\langle$v$\rangle$ often representing modern
$\langle$u$\rangle$—in the form of replacement rules, typically including
context information to discriminate between different usages of a character.
Some of the earliest approaches to normalization are rule-based, with rules
being created manually for one particular language, such as Old Icelandic
(Fix, 1980) or Old German (Koller, 1983).

VARD 2 uses “letter replacement rules” to construct normalization candidates,
but is not necessarily concerned with precision due to its interactive nature
(Baron and Rayson, 2008). Bollmann et al. (2011) describe a supervised
learning algorithm to automatically derive context-aware replacement rules
from training data, including “identity rules” that leave a character
unchanged, then apply one rule to each character of a historical word form to
produce a normalization. Porta et al. (2013) model phonological sound change
rules for Old Spanish using finite-state transducers; Etxeberria et al. (2016)
describe a similarly motivated model that can be trained in a supervised
manner.

Rule-based methods are also commonly found when the goal is not to produce a
single best normalization, but to cluster a group of spelling variants (Giusti
et al., 2007) or to retrieve occurrences of variant spellings given a modern
form in an information retrieval (IR) scenario (Ernst-Gerlach and Fuhr, 2006;
Koolen et al., 2006).

###  2.3 Distance-based Methods

Approaches using edit distance measures (such as Levenshtein distance;
Levenshtein, 1966) are most commonly found in an IR context, since measures
that compare two word forms are a natural fit for matching a search term with
relevant word forms in a historical document (e.g., Robertson and Willett,
1993). Weighted variants of distance measures can be used to assign lower
costs to more likely edit operations (Kempken et al., 2006; Hauser and Schulz,
2007).

In a normalization context, distance measures can be used to compare
historical variants to entries in a contemporary full-form lexicon (Kestemont
et al., 2010; Jurish, 2010a). Norma includes a distance-based component whose
edit weights can be learned from a training set of normalizations (Bollmann,
2012). Pettersson et al. (2013a) find a similar approach to be more effective
than hand-crafted rules on Swedish. Sometimes, the line between distance-based
and rule-based methods get blurred; Adesam et al. (2012) use the Levenshtein
algorithm to derive “substitution rules” from training data, which are then
used to link up historical Swedish forms with lexicon entries; van Halteren
and Rem (2013) describe a comparable approach for Dutch.

Furthermore, distance measures also lend themselves to unsupervised approaches
for clustering historical variants of the same modern form, where identifying
the precise modern form is not necessarily required (Amoia and Martínez, 2013;
Barteld et al., 2015).

###  2.4 Statistical Models

In a probabilistic view of the normalization task, the goal is to optimize the
probability $p(t|s)$ that a contemporary word form $t$ is the normalization of
a historical word form $s$. This can be seen as a _noisy channel model_ ,
which has been used for normalization by, e.g., Oravecz et al. (2010) and
Etxeberria et al. (2016).

More commonly, _character-based statistical machine translation (CSMT)_ has
been applied to the normalization task. Instead of translating a sentence as a
sequence of tokens, these approaches “translate” a historical word form as a
sequence of characters. This has been found to be very effective for a variety
of historical languages, such as Spanish (Sánchez-Martínez et al., 2013),
Icelandic and Swedish (Pettersson et al., 2013b), Slovene (Scherrer and
Erjavec, 2013, 2016; Ljubešić et al., 2016), as well as Hungarian, German, and
English (Pettersson, 2016), where it is usually found to outperform previous
approaches.

Pettersson et al. (2014) find that a CSMT system often performs best in a
comparison with a filtering method and a distance-based approach on five
different languages. Schneider et al. (2017) compare VARD 2 to CSMT on English
and find that VARD 2 performs slightly better. Domingo and Casacuberta (2018)
evaluate both word-based and character-based models and find that SMT
outperforms a neural network model.

###  2.5 Neural Models

Neural network architectures have become popular for a variety of NLP tasks,
and historical normalization is no exception. _Character-based neural machine
translation (CNMT)_ is the logical neural equivalent to the CSMT approach, and
has first been used for normalization of historical German (Bollmann et al.,
2017; Korchagina, 2017) using encoder–decoder models with long short-term
memory (LSTM) units. Robertson and Goldwater (2018) present a more detailed
evaluation of this architecture on five different languages. Hämäläinen et al.
(2018) evaluate SMT, NMT, an edit-distance approach, and a rule-based finite-
state transducer, and advocate for a combination of these approaches to make
use of their individual strengths; however, they restrict their evaluation to
English.

Other neural architectures have rarely been used for normalization so far. Al
Azawi et al. (2013) and Bollmann and Søgaard (2016) frame the normalization
task as a sequence labelling problem, labelling each character in the
historical word form with its normalized equivalent. Kestemont et al. (2016)
use convolutional networks for lemmatization of historical Dutch. Overall,
though, the encoder–decoder model with recurrent layers is the dominant
approach.

###  2.6 Beyond Token-Level Normalization

The presented literature almost exclusively focuses on models where the input
is a single token. In theory, it would be desirable to include context from
the surrounding tokens, as some historical spellings can have more than one
modern equivalent depending on the context in which they are used (e.g.,
historical ther could represent their or there). Remarkably few studies have
attempted this so far: Jurish (2010b) uses hidden Markov models to select
between normalization candidates; Mitankin et al. (2014) use a language model
in a similar vein; Ljubešić et al. (2016) experiment with “segment-level”
input, i.e., a string of several historical tokens as input to a normalizer.
Since this area is currently very underexplored, it warrants a deeper
investigation that goes beyond the scope of this paper.

Dataset/Language | Time Period | Genre | Tokens
---|---|---|---
|  |  |  | Train | Dev | Test
DEA | German (Anselm) | 14th–16th c. | Religious | 233,947 | 45,996 | 45,999
DER | German (RIDGES) | 1482–1652 | Science | 41,857 | 9,712 | 9,587
EN | English | 1386–1698 | Letters | 147,826 | 16,334 | 17,644
ES | Spanish | 15th–19th c. | Letters | 97,320 | 11,650 | 12,479
HU | Hungarian | 1440–1541 | Religious | 134,028 | 16,707 | 16,779
IS | Icelandic | 15th c. | Religious | 49,633 | 6,109 | 6,037
PT | Portuguese | 15th–19th c. | Letters | 222,525 | 26,749 | 27,078
SLB | Slovene (Bohorič) | 1750–1840s | Diverse | 50,023 | 5,841 | 5,969
SLG | Slovene (Gaj) | 1840s–1899 | Diverse | 161,211 | 20,878 | 21,493
SV | Swedish | 1527–1812 | Diverse | 24,458 | 2,245 | 29,184
Table 1: Historical datasets used in the experiments

##  3 Experimental Setup

#### Systems

The selection of normalization systems follows two goals:  1 to include at
least one system for each major category as identified in Sec. 2; and  2 to
use only freely available tools in order to facilitate reproduction and
application of the described methods.  To that effect, this study compares the
following approaches:

  * •

Norma444https://github.com/comphist/norma (Bollmann, 2012), which combines
substitution lists, a rule-based normalizer, and a distance-based algorithm,
with the option of running them separately or combined. Importantly, it
implements supervised learning algorithms for all of these components and is
not restricted to a particular language.

  * •

cSMTiser555https://github.com/clarinsi/csmtiser (Ljubešić et al., 2016;
Scherrer and Ljubešić, 2016), which implements a normalization pipeline using
character-based statistical machine translation (CSMT) using the Moses toolkit
(Koehn et al., 2007).

  * •

Neural machine translation (NMT), in the form of two publicly available
implementations:  1 the model by Bollmann (2018), also used in Bollmann et al.
(2018);666I reimplemented the model here using the XNMT toolkit (Neubig et
al., 2018). and  2 the model by Tang et al.
(2018).777https://github.com/tanggongbo/normalization-NMT; their model uses
the deep transition architecture of Sennrich et al. (2017, Sec. 2.3.1) as
implemented by Marian (Junczys-Dowmunt et al., 2018).

Two systems were chosen for the NMT approach as they use very different
hyperparameters, despite both using comparable neural encoder–decoder models:
Bollmann (2018) uses a single LSTM layer with dimensionality 300 in the
encoder and decoder, while Tang et al. (2018) use six vanilla RNN cells with
dimensionality 1024.

#### Datasets

Table 1 gives an overview of the historical datasets. They are taken from
Bollmann (2018) and represent the largest and most varied collection of
datasets used for historical text normalization so far, covering eight
languages from different language families—English, German, Hungarian,
Icelandic, Spanish, Portuguese, Slovene, and Swedish—as well as different text
genres and time periods. Furthermore, most of these have also been used in
previous work, such as the English, Hungarian, Icelandic, and Swedish datasets
(e.g., Pettersson et al., 2014; Pettersson, 2016; Robertson and Goldwater,
2018; Tang et al., 2018) and the Slovene datasets (e.g., Ljubešić et al.,
2016; Scherrer and Erjavec, 2016; Etxeberria et al., 2016; Domingo and
Casacuberta, 2018).

Additionally, contemporary datasets are required for the rule-based and
distance-based components of Norma, as they expect a list of valid target word
forms to function properly. For this, we want to choose resources that are
readily available for many languages and are reliable, i.e., consist of
carefully edited text. Here, I choose a combination of three
sources:888Detailed descriptions of the data extraction procedure can be found
in the Supplementary Material. 1 the normalizations in the training sets,  2
the Europarl corpus (Koehn, 2005), and  3 the parallel Bible corpus by
Christodouloupoulos and Steedman (2015).  The only exception is Icelandic,
which is not covered by Europarl; here, we can follow Pettersson (2016)
instead by using data from two specialized resources, the BÍN database
(Bjarnadóttir, 2012) and the MÍM corpus (Helgadóttir et al., 2012). This way,
we obtain full-form lexica of 12k–64k word types from the Bible corpus,
55k–268k types from Europarl, and 2.8M types from the Icelandic resources.

#### Preprocessing

The most important preprocessing decisions999The full preprocessing steps can
be found in the Supplementary Material. are  1 to lowercase all characters and
2 to remove all punctuation-only tokens.  Both capitalization and punctuation
often cannot be handled correctly without considering token context, which all
current normalization models do not do. Furthermore, their usage can be very
erratic in historical texts, potentially distorting the evaluation; e.g., when
a text uses punctuation marks according to modern conventions, their
normalization is usually trivial, resulting in artificial gains in
normalization accuracy that other texts do not get. At the same time, most
previous work has not followed these same preprocessing guidelines, making a
direct comparison more difficult. This work tries to make up for this by
evaluating many different systems, effectively reproducing some of these
previous results instead.

Method | Dataset
---|---
| DEA | DER | EN | ES | HU | IS | PT | SLB | SLG | SV
Identity | 30.63 | 44.36 | 75.29 | 73.40 | 17.53 | 47.62 | 65.19 | 40.74 | 85.38 | 58.59
Maximum | 94.64 | 96.46 | 98.57 | 97.40 | 98.70 | 93.46 | 97.65 | 98.71 | 98.96 | 98.97
Norma, Lookup | 83.86 | 82.15 | 92.45 | 92.51 | 74.58 | 82.84 | 91.67 | 81.76 | 93.90 | 83.80
Norma, Rule-based | 76.48 | 82.52 | 90.85 | 88.59 | 78.73 | 83.72 | 86.33 | 86.09 | 91.63 | 85.23
Norma, Distance-based | 58.92 | 73.30 | 83.92 | 84.41 | 62.38 | 69.95 | 77.28 | 71.02 | 88.20 | 76.03
Norma (Combined) | 88.02 | 86.55 | 94.60 | 94.41 | 86.83 |  *86.85 | 94.19 | 89.45 | 91.44 | 87.12
cSMTiser | 88.82 |  *88.06 |  *95.21 |  *95.01 |  *91.63 |  *87.10 |  *95.09 |  *93.18 |  *95.99 | 91.13
cSMTiser+LM | 86.69 |  *88.19 | 95.24 | 95.02 | 91.70 |  *86.83 | 95.18 | 93.30 | 96.01 |  *91.11
NMT (Bollmann, 2018) | 89.16 |  *88.07 | 94.80 |  *94.83 | 91.17 | 86.45 | 94.64 | 91.61 | 95.19 | 90.27
NMT (Tang et al., 2018) | 89.64 | 88.22 | 94.95 |  *94.84 |  *91.65 | 87.31 | 94.51 | 92.60 |  *95.85 | 90.39
$\dagger$SMT (Pettersson et al., 2014) | – | – | 94.3– | – | 80.1– | 71.8– | – | – | – | 92.9–
$\dagger$NMT (Tang et al., 2018) | – | – | 94.69 | – | 91.69 | 87.59 | – | – | – | 91.56

Table 2: Word accuracy of different normalization methods on the test sets of
the historical datasets, in percent; best result for each dataset in bold;
results marked with an asterisk (*) are not significantly different from the
best result using McNemar’s test at $p<0.05$. $\dagger$ indicates scores that
were not (re)produced here, but reported in previous work; they might not be
strictly comparable due to differences in data preprocessing (cf. Sec. 3).
Additionally, Identity shows the accuracy when leaving all word forms
unchanged, while Maximum gives the theoretical maximum accuracy with purely
token-level methods.

##  4 Evaluation

All models are trained and evaluated separately for each dataset by
calculating word accuracy over all tokens. In particular, there is no step to
discriminate between tokens that require normalization and those that do not;
all word forms in the datasets are treated equally.

For Norma, all components are evaluated both separately and combined, as the
former gives us insight into the performance of each individual component,
while the latter is reported to produce the best results (Bollmann, 2012). For
cSMTiser, the authors suggest using additional monolingual data to improve the
language model; the contemporary datasets are used for this purpose and the
model is trained both without and with this additional data; the latter is
denoted cSMTiser+LM. For NMT, the model by Bollmann (2018) is evaluated using
an ensemble of five models; the model by Tang et al. (2018) is trained on
character-level input using the default settings provided by their
implementation.101010This is the “Att-RNN” setting reported in their paper;
due to the high computational demands of the model, it was not feasible to run
experiments with multiple configurations.

To illustrate how challenging the normalization task is on different datasets,
we can additionally look at the identity baseline—i.e., the percentage of
tokens that do _not_ need to be normalized—as well as the maximum accuracy
obtainable if each word type was mapped to its most frequently occurring
normalization. The latter gives an indication of the extent of ambiguity in
the datasets and the disadvantage of not considering token context (cf. Sec.
2.6).

Method | Dataset
---|---
| DEA | DER | EN | ES | HU | IS | PT | SLB | SLG | SV
Norma, Lookup | 0.41 | 0.31 | 0.38 | 0.35 | 0.43 | 0.38 | 0.39 | 0.44 | 0.44 | 0.29
Norma, Rule-based | 0.40 | 0.33 | 0.43 | 0.39 | 0.38 | 0.40 | 0.45 | 0.47 | 0.46 | 0.32
Norma, Distance-based | 0.42 | 0.34 | 0.46 | 0.44 | 0.41 | 0.44 | 0.50 | 0.52 | 0.39 | 0.38
Norma (Combined) | 0.41 | 0.33 | 0.45 | 0.42 | 0.34 | 0.42 | 0.51 | 0.51 | 0.42 | 0.31
cSMTiser | 0.37 | 0.26 | 0.39 | 0.41 | 0.26 | 0.40 | 0.50 | 0.53 | 0.56 | 0.24
cSMTiser+LM | 0.39 | 0.27 | 0.39 | 0.42 | 0.27 | 0.41 | 0.50 | 0.53 | 0.56 | 0.24
NMT (Bollmann, 2018) | 0.38 | 0.26 | 0.39 | 0.43 | 0.27 | 0.40 | 0.48 | 0.47 | 0.51 | 0.23
NMT (Tang et al., 2018) | 0.38 | 0.27 | 0.38 | 0.42 | 0.26 | 0.41 | 0.46 | 0.50 | 0.56 | 0.24

(a) CERI: character error rate on the subset of incorrect normalizations
(lower is better)

Norma, Lookup | 08.47 | 16.72 | 08.33 | 27.81 | 02.86 | – | 04.57 | – | – | 07.42
---|---|---|---|---|---|---|---|---|---|---
Norma, Rule-based | 12.82 | 26.73 | 08.48 | 26.33 | 12.44 | – | 05.65 | – | – | 15.38
Norma, Distance-based | 07.71 | 19.10 | 07.26 | 28.62 | 10.93 | – | 05.58 | – | – | 12.18
Norma (Combined) | 16.97 | 28.94 | 09.86 | 43.55 | 20.00 | – | 11.13 | – | – | 20.75
cSMTiser | 17.92 | 34.67 | 08.27 | 43.82 | 20.44 | – | 06.09 | – | – | 17.73
cSMTiser+LM | 12.23 | 33.83 | 08.46 | 42.93 | 20.40 | – | 06.60 | – | – | 17.58
NMT (Bollmann, 2018) | 17.24 | 33.65 | 07.96 | 39.22 | 18.30 | – | 05.92 | – | – | 16.65
NMT (Tang et al., 2018) | 16.34 | 34.19 | 09.43 | 40.99 | 19.84 | – | 06.46 | – | – | 18.93

(b) Stemming accuracy: percentage of incorrect normalizations with correct
word stems (higher is better)

Table 3: Evaluations on the subset of incorrect normalizations only; best
results for each dataset in bold. Note that this subset is different for each
system, so for comparisons between systems, these numbers should be considered
in conjunction with word accuracy scores from Table 2.

#### Results

Table 2 shows the results of this evaluation. The extent of spelling variation
varies greatly between datasets, with less than 15% of tokens requiring
normalization (SLG) to more than 80% (HU). The maximum accuracy is above 97%
for most datasets, suggesting that we can obtain high normalization accuracy
in principle even without considering token context.

For the normalization systems, we observe significantly better word accuracy
with SMT than NMT on four of the datasets, and non-significant differences on
five others. There is only one dataset (DEA) where the NMT system by Tang et
al. (2018) gets significantly better word accuracy than other systems. This
somewhat contradicts the results from Tang et al. (2018), who find NMT to
usually outperform the SMT baseline by Pettersson et al. (2014). However, note
that the results for the cSMTiser system are often significantly better than
reported in previous work: e.g., on Hungarian, cSMTiser obtains 91.7%
accuracy, but only 80.1% with the SMT system from Pettersson et al. (2014).

Overall, the deep NMT model by Tang et al. (2018) consistently outperforms the
shallow one by Bollmann (2018). cSMTiser seems to benefit from the added
contemporary data for language modelling, though the effect is not significant
on any individual dataset. Finally, while Norma does produce competitive
results on several datasets (particularly in the “combined” setting), it is
generally significantly behind the SMT and NMT methods.

##  5 Analysis

###  5.1 Measuring Normalization Quality

While word accuracy is easily interpretable, it is also a very crude measure,
as it classifies predictions as correct/incorrect without considering the type
of error(s) made by the model. _Character error rate (CER)_ has sometimes been
suggested as a complement to address this issue, but I believe this is not
very insightful: For any normalization system that achieves a reasonably high
word accuracy, CER will highly correlate with accuracy simply because CER
equals zero for any word that is accurately normalized.111111When comparing
word accuracy scores in Table 2 with the same configurations evaluated using
CER, they correlate with Pearson’s $r\approx-0.96$. At the same time, there is
a need for a more fine-grained way to assess the normalization quality.
Consider the following example from the Hungarian dataset with its predicted
normalization from the NMT system by Bollmann (2018):

. .[Orig] yduewzewlendewk .̱[Gold] üdvözülendőek .̱[Pred] üdvözülendők

Here, the prediction matches the correct target form almost perfectly, but
would be counted as incorrect since it misses an insertion of the letter
$\langle$e$\rangle$ towards the end. In this vein, it will be treated the same
by the word accuracy measure as a prediction that, e.g., had left the original
form unchanged.

#### CERI

One alternative is to consider character error rate _on the subset of
incorrect normalizations only._ This way, CER becomes a true complement to
word accuracy by assessing the _magnitude of error_ that a normalization model
makes when it is not perfectly accurate. The results of this measure, denoted
CERI, are shown in Table 3(a). The lowest CERI score is often achieved by
Norma’s lookup module, which leaves historical word forms unchanged if they
are not in its lookup wordlist learned during training. This suggests that the
incorrect predictions made by other systems are often _worse_ than just
leaving the historical spelling unchanged.

#### Stemming

Another problem of CER is that all types of errors are treated the same: a
one-letter difference in inflection, such as king – kings or came – come,
would be treated identically to an error that changes the meaning of the word
(bids – beds) or results in a non-word (creature – cryature). I propose an
approach that, to the best of my knowledge, has not been used in normalization
evaluation before: measure accuracy on _word stems,_ i.e., process both the
reference normalization and the prediction with an automatic stemming
algorithm and check if both stems match. For this evaluation, I choose the
Snowball stemmer (Porter, 2001) as it contains stemming algorithms for many
languages (including the ones represented here except for Icelandic and
Slovene) and is publicly available.121212http://snowballstem.org/

Table 3(b) shows the accuracy on word stems, again only evaluated on the
subset of incorrect normalizations, as this better highlights the differences
between settings. This evaluation reveals some notable differences between
_datasets_ : For example, while the English and Spanish datasets have very
comparable accuracy scores overall (cf. Tab. 2), they show very different
characteristics in the stemming evaluation; for English, only up to 9.86% of
incorrect predictions show the correct word stem, while for Spanish the number
is up to 43.82%. Examining predictions on the dev set, many of the incorrectly
predicted cases in Spanish result from mistakes in placement of diacritics,
such as ésta – está or envíe – envié; the stemming algorithm removes
diacritics and can therefore match these instances. Overall, this gives an
indication that the errors made on the Spanish dataset are less severe than
those on English, despite comparable word accuracy scores and a usually higher
CERI for Spanish.

This case study shows that stemming can be a useful tool for error analysis in
normalization models and reveal characteristics that neither word accuracy nor
CER alone can show.

###  5.2 Effect of Training Data Quantity

Supervised methods for historical text normalization have been evaluated with
highly varying amounts of training data: e.g., Domingo and Casacuberta (2018)
train a normalizer for 17th century Spanish on 436k tokens; Etxeberria et al.
(2016) use only 8k tokens to train a normalizer for Basque. Even in the
evaluation in Sec. 4, training set sizes varied between 24k and 234k tokens,
depending on the dataset. Furthermore, many research projects seeking to use
automatic normalization techniques cannot afford to produce training data in
high quantity. All of this raises the question how different normalization
systems perform with varying amounts of training data, and whether reasonable
normalization results can be achieved in a low-resource scenario.

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (a) English

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (b) Hungarian

Figure 2: Word accuracy on the development sets for different amounts of
training data (note that the $x$-axis is log-scaled); NMT-1 is the model by
Bollmann (2018), NMT-2 is the model by Tang et al. (2018).

#### Methodology

All models are retrained on varying subsets of the training data, with sizes
ranging from 100 tokens to 50,000 tokens. However, the lower the training set
size is, the higher the potential variance when training on it, since random
factors such as the covered spelling variants or vocabulary are more likely to
impact the results. Therefore, I choose the following approach: For each
dataset and training size, up to ten different training splits are
extracted,131313The ten training splits consist of chunks of $n$ tokens that
are spaced equidistantly across the full training set; for larger $n$, the
number of chunks is reduced so that no splits overlap to more than 50%. and a
separate model is trained on each one. Each model is then evaluated on the
respective development dataset, and only the average accuracy across all
splits is considered.

Method | Dataset
---|---
|  DEA |  DER | EN | ES | HU | IS | PT |  SLB |  SLG | SV
Norma, Lookup/Combined | 92.36 | 93.66 | 97.46 | 96.59 | 96.81 | 89.51 | 97.04 | 97.15 | 98.17 | 97.61
Norma, Rule-based | 80.34 | 89.26 | 93.17 | 89.98 | 88.42 | 86.21 | 88.77 | 93.61 | 96.10 | 92.10
Norma, Distance-based | 60.79 | 78.85 | 86.77 | 85.89 | 67.04 | 70.99 | 78.93 | 77.49 | 94.54 | 84.17
cSMTiser | 92.18 | 93.25 | 97.10 | 96.33 | 96.33 | 89.27 | 96.80 | 96.73 | 98.09 | 97.66
cSMTiser+LM | 90.52 | 93.45 | 97.15 | 96.42 | 96.33 | 88.99 | 96.82 | 96.90 | 98.07 | 97.66
NMT (Bollmann, 2018) | 91.91 | 93.29 | 97.19 | 96.37 | 96.18 | 88.67 | 96.72 | 95.92 | 97.56 | 97.01
NMT (Tang et al., 2018) | 92.25 | 93.41 | 97.19 | 96.27 | 96.43 | 89.34 | 96.60 | 96.84 | 97.89 | 96.90

(a) In-vocabulary/seen tokens

Norma, Lookup | 3.91 | 19.92 | 30.42 | 46.72 | 3.28 | 29.40 | 28.25 | 18.07 | 68.07 | 39.85
---|---|---|---|---|---|---|---|---|---|---
Norma, Rule-based | 40.27 | 46.06 | 62.06 | 72.97 | 47.66 | 63.73 | 57.50 | 54.99 | 64.59 | 63.37
Norma, Distance-based | 41.33 | 43.25 | 48.63 | 67.78 | 47.43 | 61.64 | 57.79 | 44.23 | 49.82 | 50.15
Norma (Combined) | 47.25 | 48.13 | 59.18 | 69.93 | 54.83 | 65.52 | 60.62 | 57.57 | 50.71 | 53.73
cSMTiser | 57.25 | 59.96 | 71.78 | 80.22 | 76.59 | 69.70 | 74.69 | 78.49 | 83.30 | 70.35
cSMTiser+LM | 50.69 | 59.76 | 71.70 | 79.24 | 76.86 | 69.55 | 75.91 | 78.40 | 83.56 | 70.26
NMT (Bollmann, 2018) | 63.24 | 59.83 | 65.17 | 77.57 | 75.13 | 68.66 | 70.00 | 73.75 | 80.87 | 68.78
NMT (Tang et al., 2018) | 65.09 | 60.16 | 67.15 | 78.75 | 76.33 | 71.04 | 69.90 | 75.04 | 83.46 | 69.66

(b) Out-of-vocabulary/unseen tokens

Table 4: Word accuracy for seen/unseen tokens separately (cf. Sec. 5.3); best
results for each dataset in bold.

Method | Dataset
---|---
|  DEA |  DER | EN | ES | HU | IS | PT |  SLB |  SLG | SV
Best without lookup |  *89.64 |  *88.22 | 95.24 | 95.02 | 91.70 |  *87.31 | 95.18 | 93.30 |  *96.01 | 91.13
cSMTiser | 88.98 |  *88.41 | 95.54 | 95.25 |  *92.00 |  *87.31 |  *95.31 | 93.52 |  *96.06 |  *91.09
cSMTiser+LM | 88.35 |  *88.37 |  *95.53 |  *95.17 | 92.07 |  *87.30 | 95.39 |  *93.50 | 96.10 |  *91.07
NMT (Bollmann, 2018) | 89.56 |  *88.38 | 95.05 | 95.03 | 91.66 |  *87.20 | 94.93 | 92.60 | 95.71 | 90.72
NMT (Tang et al., 2018) | 89.74 | 88.45 | 95.19 |  *95.13 |  *91.94 | 87.46 | 94.92 | 92.85 |  *96.08 |  *90.93

Table 5: Word accuracy for the “lookup on seen tokens, learned models on
unseen tokens” strategy, following Robertson and Goldwater (2018) (cf. Sec.
5.3), compared to the best result without this strategy (according to Table
2). Best result for each dataset in bold; results marked with an asterisk (*)
are not significantly different from the best result using McNemar’s test at
$p<0.05$.

#### Results

Figure 2 shows two learning curves that are representative for most of the
datasets.141414Plots for all datasets can be found in the Appendix. They
reveal that Norma (in the “combined” setting) performs best in extremely low-
resource scenarios, but is overtaken by the SMT approach as more training data
becomes available; usually already around 500–1000 tokens. The NMT models have
a steeper learning curve, needing more training data to become competitive.
Extrapolating this trend, it is conceivable that the NMT models would simply
need more training data than our current datasets provide in order to
consistently outperform the SMT approach. On the other hand, there appears to
be no correlation between the size of the training set (cf. Tab. 1) and the
relative performance of NMT vs. SMT (cf. Tab. 2) in the experiments. Since I
am not aware of larger datasets for the historical normalization task, this
remains an open question for now.

A remarkable result is that very small amounts of training data can already be
helpful for the normalization task. The English dataset has comparatively
little spelling variation to begin with: leaving all words unnormalized
already results in an accuracy of 75.5%. Still, with as little as 100 tokens
for training, applying the Norma tool raises the accuracy above 83%. For
Hungarian, the same amount of training data raises the accuracy from 17.8%
(unnormalized) to around 50%. It would be interesting to further compare these
results with fully unsupervised methods.

###  5.3 Out-of-Vocabulary Words

Robertson and Goldwater (2018) highlight the importance of evaluating
separately on seen vs. unseen tokens, i.e., tokens that have also been in the
training set (in-vocabulary) and those that have not (out-of-vocabulary), as
well as comparing to a naive memorization baseline. These numbers are
presented in Table 5. For unseen tokens (Tab. 4(b)), the accuracy scores
follow generally the same trend as in the full evaluation of Tab. 2; i.e., SMT
performs best in most cases. For seen tokens (Tab. 4(a)), however, Norma’s
lookup component—which implements naive memorization—obtains the highest score
on nine datasets.

These observations suggest a new normalization strategy: apply the naive
lookup on the subset of in-vocabulary tokens and the SMT/NMT models on the
subset of out-of-vocabulary tokens only. Table 5 shows the results of this
strategy.151515The non-lookup components of Norma are not included in this
evaluation since “Norma (Combined)” effectively implements such a strategy
already. On nine datasets, it performs better than always using the learned
models (as in Tab. 2), and this difference is statistically significant on
five of them. These results support the claim from Robertson and Goldwater
(2018) that “learned models should typically only be applied to _unseen_
tokens.”

##  6 Conclusion

This paper presented a large study of historical text normalization. Starting
with a systematic survey of the existing literature, four different systems
(based on supervised learning) were evaluated and compared on datasets from
eight different languages. On the basis of these results, we can extract some
practical recommendations for projects seeking to employ normalization
techniques:

  1. 1.

to use the Norma tool when only little training data ($<$500 tokens) is
available;

  2. 2.

to use cSMTiser otherwise, ideally with additional data for language
modelling; and

  3. 3.

to make use of the naive memorization/lookup technique for in-vocabulary
tokens when possible.

Furthermore, the qualitative analysis (in Sec. 5.1) should encourage authors
evaluating normalization systems to use task-motivated approaches, such as
evaluation on word stems, to provide deeper insight into the properties of
their models and datasets.

Detailed information on how to train and apply all of the evaluated techniques
is also made available online at https://github.com/coastalcph/histnorm.

## Acknowledgments

I would like to thank my PhD supervisor, Stefanie Dipper, for her continuous
support over many years that culminated in the doctoral thesis on which this
paper is based; the acknowledgments in that thesis largely extend to this
paper as well. Many thanks to Anders Søgaard for many helpful discussions and
for supporting the follow-up experiments conducted for this paper. Further
thanks go to the anonymous reviewers whose helpful suggestions have largely
been incorporated here.

I gratefully acknowledge the donation of a Titan Xp GPU by the NVIDIA
Corporation that was used for a substantial part of this research.

## References

  * Adesam et al. (2012) Yvonne Adesam, Malin Ahlberg, and Gerlof Bouma. 2012.  _bokstaffua, bokstaffwa, bokstafwa, bokstaua, bokstawa…_ Towards lexical link-up for a corpus of Old Swedish.  In _Proceedings of the 11th Conference on Natural Language Processing (KONVENS 2012), LThist 2012 workshop_ , pages 365–369, Vienna, Austria.
  * Al Azawi et al. (2013) Mayce Al Azawi, Muhammad Zeshan Afzal, and Thomas M. Breuel. 2013.  Normalizing historical orthography for OCR historical documents using LSTM.  In _Proceedings of the 2nd International Workshop on Historical Document Imaging and Processing (HIP ’13)_ , pages 80–85, Washington, DC.
  * Amoia and Martínez (2013) Marilisa Amoia and José Manuel Martínez. 2013.  Using comparable collections of historical texts for building a diachronic dictionary for spelling normalization.  In _Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH)_ , pages 84–89. Association for Computational Linguistics.
  * Baron and Rayson (2008) Alistair Baron and Paul Rayson. 2008.  VARD 2: A tool for dealing with spelling variation in historical corpora.  In _Proceedings of the Postgraduate Conference in Corpus Linguistics_.
  * Barteld et al. (2015) Fabian Barteld, Ingrid Schröder, and Heike Zinsmeister. 2015.  Unsupervised regularization of historical texts for POS tagging.  In _Proceedings of the Workshop on Corpus-Based Research in the Humanities (CRH)_ , pages 3–12, Warsaw, Poland.
  * Bjarnadóttir (2012) Kristín Bjarnadóttir. 2012.  The database of modern Icelandic inflection (Beygingarlýsing íslensks nútímamáls).  In _Proceedings of the Workshop on Language Technology for Normalisation of Less-Resourced Languages_ , pages 13–18.
  * Bollmann (2012) Marcel Bollmann. 2012.  (Semi-)automatic normalization of historical texts using distance measures and the Norma tool.  In _Proceedings of the Second Workshop on Annotation of Corpora for Research in the Humanities (ACRH-2)_ , pages 3–12, Lisbon, Portugal.
  * Bollmann (2018) Marcel Bollmann. 2018.  Normalization of historical texts with neural network models.  _Bochumer Linguistische Arbeitsberichte_ , 22.
  * Bollmann et al. (2017) Marcel Bollmann, Joachim Bingel, and Anders Søgaard. 2017.  Learning attention for historical text normalization by learning to pronounce.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 332–344. Association for Computational Linguistics.
  * Bollmann et al. (2011) Marcel Bollmann, Florian Petran, and Stefanie Dipper. 2011.  Rule-based normalization of historical texts.  In _Proceedings of the International Workshop on Language Technologies for Digital Humanities and Cultural Heritage_ , pages 34–42, Hissar, Bulgaria.
  * Bollmann and Søgaard (2016) Marcel Bollmann and Anders Søgaard. 2016.  Improving historical spelling normalization with bi-directional LSTMs and multi-task learning.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 131–139. The COLING 2016 Organizing Committee.
  * Bollmann et al. (2018) Marcel Bollmann, Anders Søgaard, and Joachim Bingel. 2018.  Multi-task learning for historical text normalization: Size matters.  In _Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP_ , pages 19–24. Association for Computational Linguistics.
  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: the Bible in 100 languages.  _Language Resources and Evaluation_ , 49(2):375–395.
  * Domingo and Casacuberta (2018) Miguel Domingo and Francisco Casacuberta. 2018.  Spelling normalization of historical documents by using a machine translation approach.  In _Proceedings of the 21st Annual Conference of the European Association for Machine Translation_ , pages 129–137.
  * Ernst-Gerlach and Fuhr (2006) Andrea Ernst-Gerlach and Norbert Fuhr. 2006.  Generating search term variants for text collections with historic spellings.  In _Proceedings of the 28th European Conference on Information Retrieval Research (ECIR 2006)_ , Lecture Notes in Computer Science, pages 49–60, Berlin. Springer.
  * Etxeberria et al. (2016) Izaskun Etxeberria, Iñaki Alegria, Larraitz Uria, and Mans Hulden. 2016.  Evaluating the noisy channel model for the normalization of historical texts: Basque, Spanish and Slovene.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , pages 1064–1069, Paris, France. European Language Resources Association (ELRA).
  * Fix (1980) Hans Fix. 1980.  Automatische Normalisierung – Vorarbeit zur Lemmatisierung eines diplomatischen altisländischen Textes.  In Paul Sappler and Erich Straßner, editors, _Maschinelle Verarbeitung altdeutscher Texte. Beiträge zum dritten Symposion, Tübingen 17.–19. Februar 1977_ , pages 92–100. Niemeyer, Tübingen.
  * Giusti et al. (2007) Rafael Giusti, Arnaldo Candido Jr, Marcelo Muniz, Lívia Cucatto, and Sandra Aluísio. 2007.  Automatic detection of spelling variation in historical corpus: An application to build a Brazilian Portuguese spelling variants dictionary.  In _Proceedings of the Corpus Linguistics Conference (CL2007)_ , Birmingham, UK.
  * van Halteren and Rem (2013) Hans van Halteren and Margit Rem. 2013.  Dealing with orthographic variation in a tagger-lemmatizer for fourteenth century Dutch charters.  _Language Resources and Evaluation_ , 47(4):1233–1259.
  * Hämäläinen et al. (2018) Mika Hämäläinen, Tanja Säily, Jack Rueter, Jörg Tiedemann, and Eetu Mäkelä. 2018.  Normalizing Early English letters to present-day English spelling.  In _Proceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature_ , pages 87–96. Association for Computational Linguistics.
  * Hauser and Schulz (2007) Andreas W. Hauser and Klaus U. Schulz. 2007.  Unsupervised learning of edit distance weights for retrieving historical spelling variations.  In _Proceedings of the First Workshop on Finite-State Techniques and Approximate Search (FSTAS 2007)_ , pages 1–6, Borovets, Bulgaria.
  * Helgadóttir et al. (2012) Sigrún Helgadóttir, Ásta Svavarsdóttir, Eiríkur Rögnvaldsson, Kristín Bjarnadóttir, and Hrafn Loftsson. 2012.  The tagged Icelandic corpus (MíM).  In _Proceedings of the Workshop on Language Technology for Normalisation of Less-Resourced Languages_ , pages 67–72.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Jurish (2010a) Bryan Jurish. 2010a.  Comparing canonicalizations of historical German text.  In _Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology_ , pages 72–77, Uppsala, Sweden. Association for Computational Linguistics.
  * Jurish (2010b) Bryan Jurish. 2010b.  More than words: using token context to improve canonicalization of historical German.  _Journal for Language Technology and Computational Linguistics_ , 25(1):23–39.
  * Kempken et al. (2006) Sebastian Kempken, Wolfram Luther, and Thomas Pilz. 2006.  Comparison of distance measures for historical spelling variants.  In Max Bramer, editor, _Artificial Intelligence in Theory and Practice_ , pages 295–304. Springer, Boston, MA.
  * Kestemont et al. (2010) Mike Kestemont, Walter Daelemans, and Guy De Pauw. 2010.  Weigh your words—memory-based lemmatization for Middle Dutch.  _Literary and Linguistic Computing_ , 25(3):287–301.
  * Kestemont et al. (2016) Mike Kestemont, Guy de Pauw, Renske van Nie, and Walter Daelemans. 2016.  Lemmatization for variation-rich languages using deep learning.  _Digital Scholarship in the Humanities_.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of MT Summit_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Mercello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the ACL 2007 Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic.
  * Koller (1983) Gerhard Koller. 1983.  Ein maschinelles Verfahren zur Normalisierung altdeutscher Texte.  In Dietmar Peschel, editor, _Germanistik in Erlangen_ , volume 31 of _Erlanger Forschungen_ , pages 611–620. Universitätsbund Erlangen-Nürnberg, Erlangen.
  * Koolen et al. (2006) Marijn Koolen, Frans Adriaans, Jaap Kamps, and Maarten de Rijke. 2006.  A cross-language approach to historic document retrieval.  In _Proceedings of the 28th European Conference on Information Retrieval Research (ECIR 2006)_ , Lecture Notes in Computer Science, pages 407–419, Berlin. Springer.
  * Korchagina (2017) Natalia Korchagina. 2017.  Normalizing medieval German texts: from rules to deep learning.  In _Proceedings of the NoDaLiDa 2017 Workshop on Processing Historical Language_ , pages 12–17. Linköping University Electronic Press.
  * Levenshtein (1966) Vladimir I. Levenshtein. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  _Soviet Physics Doklady_ , 10(8):707–710.
  * Ljubešić et al. (2016) Nikola Ljubešić, Katja Zupan, Darja Fišer, and Tomaž Erjavec. 2016\.  Normalising Slovene data: historical texts vs. user-generated content.  In _Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016)_ , volume 16 of _Bochumer Linguistische Arbeitsberichte_ , pages 146–155, Bochum, Germany.
  * Mitankin et al. (2014) Petar Mitankin, Stefan Gerdjikov, and Stoyan Mihov. 2014.  An approach to unsupervised historical text normalisation.  In _Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage (DATeCH ’14)_ , Madrid, Spain.
  * Neubig et al. (2018) Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, John Hewitt, Rachid Riad, and Liming Wang. 2018.  XNMT: The extensible neural machine translation toolkit.  In _Conference of the Association for Machine Translation in the Americas (AMTA) Open Source Software Showcase_ , Boston.
  * Oravecz et al. (2010) Csaba Oravecz, Bálint Sass, and Eszter Simon. 2010.  Semi-automatic normalization of Old Hungarian codices.  In _Proceedings of the ECAI 2010 Workshop on Language Technology for Cultural Heritage_ , pages 55–59.
  * Pettersson (2016) Eva Pettersson. 2016.  _Spelling Normalisation and Linguistic Analysis of Historical Text for Information Extraction_.  Doctoral dissertation, Uppsala University, Department of Linguistics and Philology, Uppsala.
  * Pettersson et al. (2013a) Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2013a.  Normalisation of historical text using context-sensitive weighted Levenshtein distance and compound splitting.  In _Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013)_ , pages 163–179. Linköping University Electronic Press.
  * Pettersson et al. (2014) Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2014.  A multilingual evaluation of three spelling normalisation methods for historical text.  In _Proceedings of the 8th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH)_ , pages 32–41. Association for Computational Linguistics.
  * Pettersson et al. (2013b) Eva Pettersson, Beáta Megyesi, and Jörg Tiedemann. 2013b.  An SMT approach to automatic annotation of historical text.  In _Proceedings of the Workshop on Computational Historical Linguistics at NODALIDA 2013_ , NEALT Proceedings Series 18/Linköping Electronic Conference Proceedings 87, pages 54–69. Linköping University Electronic Press.
  * Porta et al. (2013) Jordi Porta, José-Luis Sancho, and Javier Gómez. 2013.  Edit transducers for spelling variation in Old Spanish.  In _Proceedings of the Workshop on Computational Historical Linguistics at NODALIDA 2013_ , NEALT Proceedings Series 18/Linköping Electronic Conference Proceedings 87, pages 70–79. Linköping University Electronic Press.
  * Porter (2001) Martin Porter. 2001.  Snowball: A language for stemming algorithms.
  * Rayson et al. (2005) Paul Rayson, Dawn Archer, and Nicholas Smith. 2005.  VARD versus Word: A comparison of the UCREL variant detector and modern spell checkers on English historical corpora.  In _Proceedings of the Corpus Linguistics Conference CL2005_ , Birmingham, UK. University of Birmingham.
  * Robertson and Goldwater (2018) Alexander Robertson and Sharon Goldwater. 2018.  Evaluating historical text normalization systems: How well do they generalize? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 720–725. Association for Computational Linguistics.
  * Robertson and Willett (1993) Alexander M. Robertson and Peter Willett. 1993.  A comparison of spelling-correction methods for the identification of word forms in historical text databases.  _Literary and Linguistic Computing_ , 8(3):143–152.
  * Sánchez-Martínez et al. (2013) Felipe Sánchez-Martínez, Isabel Martínez-Sempere, Xavier Ivars-Ribes, and Rafael C. Carrasco. 2013.  An open diachronic corpus of historical Spanish: annotation criteria and automatic modernisation of spelling.  _CoRR_ , abs/1306.3692.
  * Scherrer and Erjavec (2013) Yves Scherrer and Tomaž Erjavec. 2013.  Modernizing historical slovene words with character-based SMT.  In _Proceedings of the 4th Biennial International Workshop on Balto-Slavic Natural Language Processing_ , pages 58–62. Association for Computational Linguistics.
  * Scherrer and Erjavec (2016) Yves Scherrer and Tomaž Erjavec. 2016.  Modernising historical Slovene words.  _Natural Language Engineering_ , 22(6):881–905.
  * Scherrer and Ljubešić (2016) Yves Scherrer and Nikola Ljubešić. 2016.  Automatic normalisation of the Swiss German ArchiMob corpus using character-level machine translation.  In _Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016)_ , volume 16 of _Bochumer Linguistische Arbeitsberichte_ , pages 248–255, Bochum, Germany.
  * Schneider et al. (2017) Gerold Schneider, Eva Pettersson, and Michael Percillier. 2017.  Comparing rule-based and SMT-based spelling normalisation for English historical texts.  In _Proceedings of the NoDaLiDa 2017 Workshop on Processing Historical Language_ , pages 40–46. Linköping University Electronic Press.
  * Sennrich et al. (2017) Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017.  The University of Edinburgh’s neural MT systems for WMT17.  In _Proceedings of the Second Conference on Machine Translation_ , pages 389–399. Association for Computational Linguistics.
  * Tang et al. (2018) Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018.  An evaluation of neural machine translation models on historical spelling normalization.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1320–1331. Association for Computational Linguistics.

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (a) German
(Anselm)

1002505001k2.5k5k10k25k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (b) German
(RIDGES)

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (c) English

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (d) Hungarian

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (e) Spanish

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (f)
Portuguese

Figure 3: Word accuracy on the development sets for different amounts of
training data (note that the $x$-axis is log-scaled); NMT-1 is the model by
Bollmann (2018), NMT-2 is the model by Tang et al. (2018).

1002505001k2.5k5k10k25k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (a) Icelandic

1002505001k2.5k5k10k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (b) Swedish

1002505001k2.5k5k10k25k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (c) Slovene
(Bohorič)

1002505001k2.5k5k10k25k50k20%40%60%80%NormacSMTiser+LMNMT-1NMT-2 (d) Slovene
(Gaj)

Figure 4: Word accuracy on the development sets (continued); NMT-1 is the
model by Bollmann (2018), NMT-2 is the model by Tang et al. (2018).

##  Appendix A Appendix

Figures 3 and 4 show plots of the learning curves for all of the datasets.

###  A.1 Preprocessing

The full preprocessing steps of all datasets (both historical and
contemporary) comprise of:

  1. 1.

lowercasing all tokens;

  2. 2.

filtering out pairs where either the historical token or the reference
normalization is empty;

  3. 3.

filtering out pairs where either the historical token or the reference
normalization consists _only_ of punctuation marks, defined as characters that
belong to one of the Unicode “Punctuation” categories;

  4. 4.

replacing all digits with zeroes _iff_ the digits in the historical token and
the reference normalization match;

  5. 5.

replacing actual space characters in either the historical token or the
reference normalization with a special symbol that does not otherwise occur in
the dataset; and

  6. 6.

performing Unicode normalization according to the NFC standard.

Additionally, the preprocessing script can also be found in the Supplementary
Material or at https://github.com/coastalcph/histnorm.