# A tailored Handwritten-Text-Recognition System for Medieval Latin

Philipp Koch1$\diamondsuit$ Gilary Vera Nuñez1$\diamondsuit$ Esteban Garces
Arias1$\spadesuit$ Christian Heumann1$\spadesuit$ Matthias
Schöffel2$\clubsuit$ Alexander Häberlin2,3$\heartsuit$ Matthias
Aßenmacher1,4$\spadesuit$

1 Department of Statistics, LMU, Munich, Germany
2 Bavarian Academy of Sciences, BAdW, Munich, Germany
3 Universität Zürich, Zurich, Switzerland
4 Munich Center for Machine Learning (MCML), LMU, Munich, Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The Bavarian Academy of Sciences and Humanities aims to digitize its Medieval
Latin Dictionary. This dictionary entails record cards referring to lemmas in
medieval Latin, a low-resource language. A crucial step of the digitization
process is the Handwritten Text Recognition (HTR) of the handwritten lemmas
found on these record cards. In our work, we introduce an end-to-end pipeline,
tailored to the medieval Latin dictionary, for locating, extracting, and
transcribing the lemmas. We employ two state-of-the-art (SOTA) image
segmentation models to prepare the initial data set for the HTR task.
Furthermore, we experiment with different transformer-based models and conduct
a set of experiments to explore the capabilities of different combinations of
vision encoders with a GPT-2 decoder. Additionally, we also apply extensive
data augmentation resulting in a highly competitive model. The best-performing
setup achieved a Character Error Rate (CER) of 0.015, which is even superior
to the commercial Google Cloud Vision model, and shows more stable
performance.

##  1 Introduction

††This paper has been accepted at the First Workshop on Ancient Language
Processing, co-located with RANLP 2023.

The Medieval Latin Dictionary (MLW)111In German: Mittellateinisches Wörterbuch
(MLW), located at the Bavarian Academy of Sciences, deals with Latin texts
that were created between 500 and 1280 in the German-speaking region. The
foundations for this project have been developed from 1948 onwards and since
then, the dictionary has been continuously published in individual partial
editions since 1959. Currently, the letter S is being worked on in particular.
The basis of the dictionary consists of 50 selected texts that have been fully
transcribed onto DIN-A6 sheets (record cards) constituting about 40% of the
note material. Later, another 2,500 texts were excerpted and transcribed
manually onto DIN-A6 record cards, using a typewriter (cf. Fig. 1). In
addition, there are so-called "index cards", a type of record card, that helps
to uncover often hundreds of additional references. In total, it is estimated
that 1.3 million reference points have been recorded for the MLW. These record
cards were sorted alphabetically by the first letter of the keyword (lemma),
and serve as the foundation for creating a dictionary. By 2025, at least half
of the note material is planned to be scanned and recorded in a database.

Figure 1: Record card from the MLW data set.

To digitize the material, the lemmas - always found in the upper left corner
of the record cards, either hand- or machine-written - must be extracted from
the cards and recognized using an Optical Character Recognition (OCR) or HTR
procedure. Around 200,000 record cards have been scanned (cf. Fig. 1), and
annotated with their respective lemma. The accurate extraction and
transcription of the lemma present a challenge, which is further compounded by
the limited resources available for medieval Latin. To address this, we
develop an end-to-end pipeline that begins by extracting the lemma from the
record cards and subsequently utilizes an elaborated HTR system to recognize
the text.

#### Contributions

  1. 1.

We present a novel end-to-end HTR pipeline specifically designed for detecting
and transcribing handwritten medieval Latin text. Notably, it surpasses
commercial applications currently considered SOTA for related tasks.

  2. 2.

We successfully train a detection model without relying on human-annotated
bounding boxes for the lemmas.

  3. 3.

We conduct extensive experiments to compare various vision encoders and
evaluate the effectiveness of data augmentation techniques.

  4. 4.

We make our codebase, models, and data sets publicly available.

##  2 Related Work

We provide an overview of the field of HTR, which is the main challenge of
this work. We also deal with an instance of object detection to prepare the
training data. However, since this problem is only an intermediate step and
not the aim of this work, we do not cover it extensively. We refer to the
survey of Zaidi et al. (2021) for a detailed overview.

The recognition of handwritten text differs from OCR insofar as it needs to
deal with less standardized data. Previous approaches have focused on applying
deep learning to tackle these tasks. Here, the objective of Connectionist
Temporal Classification (CTC) (Graves et al., 2006) comes into play. CTC is a
technique in which a neural network – initially a Recurrent Neural Network
(RNN) but other networks might also be used (Chaudhary and Bali, 2022) – is
trained to predict a matrix of conditional transition probabilities. The input
image, represented as a vector representation through a Convolutional Neural
Network (CNN), is fed to the network, and for each input (i.e. the activation
maps of the CNN) the network predicts the character. After obtaining the
probabilities, a matrix of conditional transition probabilities can be
constructed. A unique void character is introduced to avoid false repetitions,
and the final sequence can be obtained and compared with the ground truth.
Since many sequences can be obtained from the matrix, the network is trained
to maximize the correct conditional transition probabilities. During inference
time, the model cannot compute the path of all likely sequences but instead
needs to predict the class just in time. For this purpose, search algorithms
like beam search or infix search are used.

CTC, combined with CNNs and RNNs, often yielded competitive results, such as
shown by Puigcerver (2017) and Bluche and Messina (2017). Furthermore,
approaches applying only CNNs and CTC also exist (Chaudhary and Bali, 2021,
2022). The model Easter2.0 achieved competitive results on the IAM data set
(Marti and Bunke, 2002), a data set consisting of English handwritten text and
being widely used for HTR.

A recent work that achieved SOTA results on the IAM data set is the TrOCR
model (Li et al., 2022), based on the transformer (Vaswani et al., 2017). The
model consists of a vision encoder and a text decoder, deviating from previous
approaches in which CNNs and RNNs have been primarily used. The input is
processed through the encoder and represented in vector space. A language
model for decoding subsequently produces the text to be predicted. However,
with the emergence of the transformer in the vision domain (Dosovitskiy et
al., 2021; Bao et al., 2022), end-to-end modeling has become possible. In the
work of Barrere et al. (2022), another transformer-based model is applied for
HTR. The main difference to TrOCR is a different embedding technique for
visual features based on a CNN. Furthermore, the model also applies CTC during
training. The results have also been shown to be competitive on the IAM data
set. Diaz et al. (2021) compared different encoder-decoder models’ performance
on HTR. In their study, they used different models in the encoder and decoder
parts, so a transformer encoder is used before using a CTC-based decoder.
Furthermore, they found that a transformer encoder and a CTC-trained decoder
enriched with a language model achieved SOTA results on the IAM data set.

The TrOCR framework has been successfully applied to historical data akin to
our task. In the work of Ströbel et al. (2022), a TrOCR instance was fine-
tuned to handwritten Latin from the 16th century (Stotz and Ströbel, 2021,
referred to as Gwalther), achieving competitive results.

##  3 Data

Our data set comprises 114,653 images (18,9 GB), corresponding to 3,507
distinct lemmas. All images are in RGB, but not uniform in size, i.e. height,
and width differ from image to image. Additionally, the information on the
corresponding lemma (i.e., the ground truth) is available for each image as
well as the dictionary’s vocabulary.

#### Image data

Figure 1 shows one (arbitrarily chosen) sample from the data set. Most record
cards follow the same structure being composed of three main parts,
highlighted via green boxes. The first one (1), and the one we deem most
challenging, is the lemma, which is always located in the upper left corner of
the record card. The second part (2) is the index of the text where the lemma
is found. The third part (3) contains a text extract in which the word
(corresponding to the lemma) occurs in context.

#### Lemma Annotation

Figure 2: Distribution of the first letters of the lemmas. Figure 3: Length
distribution of the lemmas.

Our analysis is based on lemma annotations on an image level, i.e. which lemma
is on the corresponding record cards. There is a total of 17 different first
letters, eight of which are each upper- and lowercase, as well as one special
character. The capitalization of a word plays a crucial role since a word’s
meaning changes depending on capitalization. Since the majority of our data
stems from the S-series of the dictionary, most lemmas start with the letter
"s". Likewise, we found a large number of lemmas starting with the letters
"m", "v", "t", "u", "l", and "n" (cf. Fig. 2). We also analyzed the number of
record cards available per lemma. In this analysis, we found that some lemmas
are under-represented in the data set, while a few constitute a large chunk of
the data.

A total of 2,420 lemmas (69%) were found to have ten record cards or less; 854
lemmas (24.4%), between 10 and 100 record cards, and just 233 lemmas (6.6%),
more than 100 record cards. It is worth mentioning that 1,123 lemmas (approx.
36.7%) had only one record card.

Finally, we analyze the length of the lemmas (cf. Fig. 3). We observe lemmas
from a length of one character up to a maximum of 19 characters. The average
length of the lemmas lies between five and six characters. The presence of
such long lemmas motivated the decision of additionally using a weighted
metric for model evaluation, as will be explained in Section 5.3.

##  4 Lemma Extraction Pipeline

In this section, we delve into the details of the custom-designed pipeline for
the extraction of the lemma from the record cards.

Figure 4: Visualization of the designed pipeline, encompassing three building
blocks: (1) the visual detection of the lemma, followed by (2) the encoding in
the latent space, and (3) the decoding into plain text.

###  4.1 Visual Detection

Due to the data structure, we are confronted with the problem of finding
suitable bounding boxes to extract the lemmas from the upper left of the
record cards. When using the entire record cards for the recognition task, the
majority of the image is noise, making model training significantly more
difficult. Since the lemmas are not annotated with their exact locations,
training a custom object detection model for extraction is not feasible. In
order to still retrieve the locations of the bounding boxes for some lemmas,
we transform the problem into an instance of visual grounding by providing a
model with an image and the description of an object in the image, upon which
it is expected to return the object’s location. We use the One For All (OFA)
transformer (Wang et al., 2022), fine-tuned on RefCOCO (Kazemzadeh et al.,
2014). To ensure the quality of the extracted lemma, we experiment with
multiple prompts and examine their results (cf. Appendix A).

After obtaining a training data set of 20,000 instances, each of them
annotated with bounding boxes, we train a YOLOv8 model (Jocher et al., 2023)
based on the You Only Look Once (YOLO) architecture (Redmon et al., 2016). The
model predictions from our YOLO model, are then subject to two post-processing
steps (described in the following) to ensure the quality of the images.

#### Multiple Bounding Boxes:

For 17,674 images (15.42% of the data), the model predicted more than just one
bounding box. We visually examined the cases and found that other handwritten
text was often recognized as a lemma, sometimes scattered throughout the
record cards (e.g. upper or lower right). The distribution of the bounding
boxes throughout the record cards is displayed in Figure 12 (Appendix B).

#### Missing Bounding Box:

We visually examined the 202 cases where no bounding box was detected, some
stemming from machine writing (instead of cursive handwriting) or scanning
errors. For some images that follow the standard layout of the record cards,
the model also failed. We disregard this set constituting less than 0.2% of
the entire data set.

#### Determining the Bounding Box

Taking all aspects into account, we introduce two rules to determine the
appropriate bounding box: (1) choose the largest bounding box, and (2) the
bounding box has to be in the upper left quarter of the entire image. The
result after applying these rules is displayed in Figure 13 (Appendix B). The
final data set consists of 114,451 samples, exhibiting a difference of the 202
samples to the initial 114,653 image-label pairs. We make our data available
on HuggingFace.222https://huggingface.co/misoda

###  4.2 HTR Model

We use a transformer as the main model akin to TrOCR. For the encoder, we
consider three different architectures, while we use GPT-2 (Radford et al.,
2019) as a decoder model for all setups. All models are trained from scratch,
although we use pre-trained image processors for the encoder models and train
a tokenizer for our custom alphabet.

#### Tokenizer

We use a customized byte-level BPE (Sennrich et al., 2016) tokenizer for the
dictionary’s vocabulary. The tokenizer is trained on the labels from our data
set.

#### Vision Encoders

We consider three different encoder architectures, namely Vision Transformer
(ViT) (Dosovitskiy et al., 2021), Bidirectional Encoder representation for
Image Transformers (BEiT) (Bao et al., 2022), and Shifted Window Transformer
(Swin) (Liu et al., 2021).

ViT is a transformer-encoder-based model employing 16 x 16 image patching for
transforming images into sequences. Additionally, a class patch is
concatenated to the sequence of patches, which is used for classification
tasks and entails general information about the sequence, similar to the CLS
token in BERT Devlin et al. (2019). For training, a feed-forward neural
network is stacked on top of the encoder, serving as an adapter between the
encoder and the targets during pre-training.

Akin to ViT, BEiT builds on image patching and an image vocabulary. For pre-
training, masked image modeling is introduced, inspired by masked language
modeling from natural language processing (Devlin et al., 2019). Further, the
encoder exhibits a visual vocabulary, and a Variational Auto Encoder (Kingma
and Welling, 2022) is trained in advance to encode an image to a lower
dimension. The decoder reproduces the image from the latent codes which can be
used as a visual vocabulary, and based on this vocabulary, an image can be
represented through a sequence of visual tokens. For our purpose, we train the
model from scratch and only re-use the pre-trained image processor.

A problem in the vision domain is the high dimensionality and the often
spatially related information in the data. Splitting the image into large
patches might break the often fine-grained relations of entities on the
images. To overcome this issue, self-attention can be applied more fine-
grained, however, this results in higher computational cost. In Swin, this
issue is tackled using a new encoder block structure, differing substantially
from the other transformers. To overcome the above-mentioned problems, the
self-attention mechanism is applied differently to account for different
aspects of the image. In the lower layers, the image is divided into small
patches, and the self-attention mechanism is applied to the small patches in
windows. These windows are shifted in upper layers to connect the different
patches. Furthermore, the windows are enlarged in upper layers producing a
hierarchical representation. Our model uses a newly initialized Swin
transformer alongside a pre-trained image processor.

#### Text Decoder

GPT-2 (Radford et al., 2019) is a decoder-only transformer that has shown
competitive capabilities in text generation. A decoder can only be trained to
predict the next token based on the previous sequence while relying on encoded
information from the encoder. Since the problem of predicting the next token
is a classification task, the Cross-Entropy Loss is used. GPT-2 has been shown
to be able to capture the underlying patterns and structures of natural
language, making it capable of generating coherent and contextually
appropriate text. Due to the overall strong performance of GPT-2, we chose to
use it as a decoder for our model. We train it from scratch, i.e., we do not
use the pre-trained weights since we deal with a specific task in a low-
resource language setting.

#### Implementation Details

We use the HuggingFace transformers library (Wolf et al., 2020) and PyTorch
(Paszke et al., 2019) to train the HTR pipeline. Our codebase, containing all
scripts (experiments and training) is available via
GitHub333https://github.com/slds-lmu/mlw-htr, and the final model is on
pypi.444https://pypi.org/project/mlw-lectiomat/ All the experiments were
conducted using a Tesla V100 GPU (16 GB).

##  5 Experiments

###  5.1 Standard Training

After shuffling the data, we randomly split it into a train (85% – 97,283
samples) and a test (15% – 17,168 samples) set. In the train split, 94.53%
(3,315) of the lemmas are present. For all training procedures, we use the
AdamW optimizer (Loshchilov and Hutter, 2019) and did not engage in
hyperparameter tuning. Further details are reported in Appendix C. For
standard training, the model is trained using a data set that includes the cut
images from the record cards as input and their respective lemmas as the
labels to be predicted. We train each of the models for a total of 5 epochs.

###  5.2 Data Augmentation

Augmentation is a common technique used in deep learning to diversify the
training data by applying different modifications without changing the
underlying semantics of the data. The goal of augmentation is to provide the
model with a diverse set of examples, helping it generalize better and improve
performance. Yang et al. (2022) show that augmentation can notably improve the
results of deep learning models. We apply on-the-fly augmentation to our data
due to the large data set size.

To provide maximal modification and increase the diversity of the training
data, different augmentation techniques are applied at random on-the-fly.
These techniques include random rotation, blurring, or modifications related
to color perception. Since the augmentation is applied on-the-fly, it is
necessary to increase the number of epochs so that the model has also enough
opportunities to observe the original, unmodified data and the augmented
variations. We increased the number of epochs to 20 (compared to 5 for the
standard training).

We use three different augmentation pipelines, one of which is randomly chosen
with $p=\frac{1}{3}$. In the following, we will illustrate each of them at the
example of the lemmas shown in Figure 5.

Figure 5: Original lemmas without any modification.

#### Pipeline A

For the first pipeline, blurring and modifications to sharpness are applied to
the data. The intensity of these modifications is determined randomly and can
range from no modification to higher intensity (cf. Fig. 6).

Figure 6: Exemplary samples from pipeline A.

#### Pipeline B

For the second pipeline, various modifications are applied to alter
brightness, contrast, saturation, sharpness, and hue. The specific alterations
for each instance are again determined randomly, also including the
possibility of no modifications at all (cf. Fig. 7).

Figure 7: Exemplary samples from pipeline B.

#### Pipeline C

The third pipeline combines the modifications from the previous two (cf. Fig.
8).

Figure 8: Exemplary samples from pipeline C.

In addition to the described techniques, all augmentation pipelines include
random masking, where rectangles of the images are blackened, and random
rotation within a range of -10 to 10 degrees.

#### Decoder Pre-Training

We experiment with pre-training the decoder in order to incorporate prior
knowledge about the vocabulary we want to predict in the medieval Latin
language. After pre-training the decoder on a corpus of the concatenated
lemmas, we combine it with the encoder and continue training as described in
Section 5.1. While pre-training is performed for a total of 10 epochs, the
training of the entire transformer is conducted for 20 epochs. In this
approach, the same augmentation techniques, as outlined before, are applied to
the training data.

###  5.3 Performance metrics

We assess the model performance using the CER, which is computed by summing up
edit operations and dividing by the length of the lemma.

$CER=\frac{S+D+I}{N}=\frac{S+D+I}{S+D+C},\\\ $ (1)

where S is the number of substitutions, D is the number of deletions, I is
number of insertions, C is the number of correct characters, and N is number
of characters in the label. To account for the varying length of the lemmas,
we further utilize the weighted CER.

$WeightedCER=\frac{\sum_{i=1}^{n}l_{i}*CER_{i}}{\sum_{i=1}^{n}l_{i}},$ (2)

where $l_{i}=$ is the number of characters of label $i$, and $CER_{i}$ is the
CER for example $i,i=1,\ldots,n$.

###  5.4 Experimental Results

The main results of our work are reported in Table 1. The BEiT+GPT-2
architecture achieved the best results in case of the standard training
regime, exhibiting a CER of 0.258, followed by Swin+GPT-2 (0.349) and
ViT+GPT-2 (0.418).

Applying the augmentation pipelines, as described in Section 5.2, notably
improves model performance compared to the standard training for all three
models. The best model with augmentation is Swin+GPT-2, achieving a CER of
0.017. As for the other two models, the CER is 0.073 for ViT+GPT-2 and 0.110
for BEiT+GPT-2.

| ViT | Swin | BEiT
---|---|---|---
Standard | 0.418 | 0.349 | 0.258
\+ Data Augmentation | 0.073 | 0.017 | 0.110
\+ Decoder Pre-Training | 0.049 | 0.018 | 0.114

Table 1: CER-Results for different encoder configurations.

Pre-training of the decoder does, on average, not lead to further improvement.
ViT+GPT-2 is the exception, for which the CER drops to 0.049. We observe no
improvements for the other models (BEiT+GPT-2: 0.114, Swin+GPT-2: 0.018).

To summarize, the best results are achieved when using a Swin+GPT-2 model with
data augmentations, reaching a CER value of 0.017.

###  5.5 Ablation Study

To investigate the impact of the data augmentation, we perform three
ablations, removing individual steps from the augmentation pipelines. Our
ablations include applying modifications to the image regarding sharpness,
brightness, color, and blurring (cf. Sec. 5.2). We also apply random rotation
and random erasing of some image parts, resulting in black rectangles
(masking). To investigate the individual effects of each augmentation
technique, we train the model without a specific augmentation method and
report the resulting CER.

Swin+GPT-2 (Full augmentation pipelines) | 0.017
---|---
w/o masking augmentation | 0.015
w/o rotation augmentation | 0.021
w/o color augmentation | 0.017

Table 2: CER-Results of different model configurations.

The results of the ablation study can be seen in Table 2. Excluding the
masking step from the pipeline leads to an actual improvement of model
performance, such that the CER improves to 0.015. However, excluding random
rotations of the images leads to an increase in CER to 0.021, while
augmentation without applying the color-related augmentations results in a CER
of 0.017, equal to the initial model trained with all augmentation techniques.
Please note, that only the specific technique was left out at a time, while
the other modifications were still in use. From the results of these
ablations, we can conclude that adding rotation seems to be a major
contribution to prediction quality. Random masking decreases the performance,
while color does not seem to have an impact on the performance.

###  5.6 Google Cloud Vision Comparison

Figure 9: Violin plots for the comparison of our Swin+GPT-2 model (left) to
Google Cloud Vision (right).

To compare the results of our model, we decided to use a highly competitive
model for HTR, the  Gloogle Cloud Vision (GCV) model. It is capable of
recognizing handwritten text and has been proven performative in practical
applications (Thammarak et al., 2022). As already mentioned, some of the cut
record cards which contain the lemmas in our data set contain extra characters
and/or suffixes that are not part of the true lemma. We observe that GCV often
predicts these extra characters as well. Considering this issue, we decided to
post-process the predictions by GCV for a fair comparison. This work consisted
of deleting extra characters and words after the first word or after a ’-’ or
a ’(’. Nevertheless, it was not possible to remove all artifacts from the
predictions. In some cases, separating them was impossible since the
characters were predicted altogether. Figure 9 shows the comparison of our
model with GCV.

The violin plots of the (unweighted) CERs show a concentration of the CER
values around 0 (= correct prediction) for both models. For our model, the
most extreme values are at a CER of 3, for GCV the maximum is nearly twice as
high and we observe an overall higher standard deviation compared to our
model. Note, that these extreme values originate from the problem that the
models sometimes predict too many characters, which are not part of the true
(annotated) lemma. To conclude, our best model exhibits a weighted CER of
0.0153, while GCV only reaches 0.1045. Overall, our model correctly predicts
97,09% of all lemmas, while GCV only does so for 78.26%.

###  5.7 Performance of other HTR systems

Table 3 illustrates the CERs of other systems on different HTR data sets.
Ströbel et al. (2022) use the Rudolph Gwalther data set, while all other
papers evaluate their systems on the IAM data set. Our model achieves the
lowest CER. However, it must be considered that we did not evaluate the same
data set, which makes a direct comparison impossible. In contrast to the other
transformer-based models, our best model uses Swin as an encoder which we have
not found in other work.

Model | CER | Data set | Architecture
---|---|---|---
Ours (Best) | 0.0153 | MLW | Transformer
TrOCR Large (Ströbel et al., 2022) | 0.0255 | Gwalther | Transformer
TrOCR Large (Li et al., 2022) | 0.0289 | IAM | Transformer
EASTER2.0 (Chaudhary and Bali, 2022) | 0.0621 | IAM | CNN+CTC
Light Transformer (Barrere et al., 2022) | 0.0570 | IAM | CNN+Transformer
Self-Att.+CTC+LM (Diaz et al., 2021) | 0.0275 | IAM | Trf.+CTC\+ LM

Table 3: Performance of contemporary HTR systems evaluated on different data
sets.

##  6 Discussion and Outlook

Due to the focus on recognizing the lemma, we did not experiment with other
object detection or image segmentation techniques. Since the record cards
include much more information than the one we extracted, we recommend further
research into various extraction techniques. With the recent publication of
Segment Anything Model, Kirillov et al. (2023) introduce a model that might be
able to extract features from the record cards with much higher accuracy. The
next objective could be to extract the inflected lemmas (cf. Sec. 3).

We did neither experiment with the initial TrOCR architecture nor did we fine-
tune a pre-trained TrOCR instance for this task. However, the results of
Ströbel et al. (2022) suggest a strong performance of TrOCR. Thus we also
recommend training it on the MLW data set. On the other hand, the results of
using the Swin encoder indicate a powerful performance compared to the other
models we have used. Thus, we also suggest more research into investigating
the usage of Swin as an encoder for this task.

##  7 Conclusion

We present a novel end-to-end pipeline for the Medieval Latin dictionary. Our
library includes an image-detection-based model for lemma extraction and a
tailored HTR model. We experiment with training different configurations of
transformers using the ViT, BEiT, and Swin encoders while using a GPT-2
decoder. Employing data augmentation, our best model (Swin+GPT-2) achieves a
CER of 0.015. The evaluation of the results exhibits a weaker performance on
longer lemmas and on lemmas that appear less frequently in the training data.
Further experiments with generative models to produce synthetic data (not
reported in the paper) were not successful, however, we recommend further
research into the direction of creating synthetic data. To conclude, our
approach presents a promising HTR solution for Medieval Latin. Future research
can build upon our work, and explore its generalizability to other languages
and data sets by making use of our pip-installable Python package:
https://pypi.org/project/mlw-lectiomat/

## Limitations

Our approach has several limitations that can be addressed to improve its
efficiency further. There are issues regarding the data set (cf. Sec. 3) that
might be reflected in the model’s performance. As discussed in Section 3, some
lemmas are stroked out partially or entirely, introducing a notable noise to
the data. Further, handwritten comments or other annotations have been added
to some of the record cards, and some images are not correctly labeled, which
might have distorted the recognition capabilities of our model.

Since our pipeline was mostly trained on data from the S-series of the
dictionary, many words starting with other letters were not seen by the model
during training. Therefore, the performance of the proposed approach, when
applied to other series, remains somewhat uncertain. As elaborated in section
7, the model tends to perform weaker on unseen lemmas. Further, there are
indications that the model might perform worse on longer lemmas.

The lemma-detection model (YOLOv8) is not guaranteed to predict the correct
bounding box for the lemma consistently. Errors at this early stage of the
pipeline may severely impact the result. Although the failure rate for the
training dataset in which no bounding box was predicted is close to zero, the
problem can still appear during inference.

## Ethics Statement

We affirm that our research adheres to the ACL Ethics Policy. This work
involves the use of publicly available data sets and does not involve human
subjects or any personally identifiable information. We declare that we have
no conflicts of interest that could potentially influence the outcomes,
interpretations, or conclusions of this research. All funding sources
supporting this study are acknowledged. We have made our best effort to
document our methodology, experiments, and results accurately and are
committed to sharing our code, data, and other relevant resources to foster
reproducibility and further advancements in research.

## Acknowledgements

We wish to thank the Bavarian Academy of Sciences for providing us with the
guidance and required access to the handwritten material. This work has been
partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) as part of BERD@NFDI - grant number 460037581.

## References

  * Bao et al. (2022) Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022.  Beit: Bert pre-training of image transformers.
  * Barrere et al. (2022) Killian Barrere, Yann Soullard, Aurélie Lemaitre, and Bertrand Coüasnon. 2022.  A light transformer-based architecture for handwritten text recognition.  In _Document Analysis Systems_ , pages 275–290, Cham. Springer International Publishing.
  * Bluche and Messina (2017) Théodore Bluche and Ronaldo Messina. 2017.  Gated convolutional recurrent neural networks for multilingual handwriting recognition.  In _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , volume 01, pages 646–651.
  * Chaudhary and Bali (2021) Kartik Chaudhary and Raghav Bali. 2021.  Easter: Simplifying Text Recognition using only 1d Convolutions.  _Proceedings of the Canadian Conference on Artificial Intelligence_.  Https://caiac.pubpub.org/pub/fm5sy88o.
  * Chaudhary and Bali (2022) Kartik Chaudhary and Raghav Bali. 2022.  Easter2.0: Improving convolutional models for handwritten text recognition.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Diaz et al. (2021) Daniel Hernandez Diaz, Siyang Qin, Reeve Ingle, Yasuhisa Fujii, and Alessandro Bissacco. 2021.  Rethinking text line recognition models.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks.  In _Proceedings of the 23rd International Conference on Machine Learning_ , ICML ’06, page 369–376, New York, NY, USA. Association for Computing Machinery.
  * Jocher et al. (2023) Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023.  YOLO by Ultralytics.
  * Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014.  ReferItGame: Referring to objects in photographs of natural scenes.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 787–798, Doha, Qatar. Association for Computational Linguistics.
  * Kingma and Welling (2022) Diederik P Kingma and Max Welling. 2022.  Auto-encoding variational bayes.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023.  Segment anything.
  * Li et al. (2022) Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2022.  Trocr: Transformer-based optical character recognition with pre-trained models.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.
  * Marti and Bunke (2002) Urs-Viktor Marti and H. Bunke. 2002.  The iam-database: An english sentence database for offline handwriting recognition.  _International Journal on Document Analysis and Recognition_ , 5:39–46.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.
  * Puigcerver (2017) Joan Puigcerver. 2017.  Are multidimensional recurrent layers really necessary for handwritten text recognition? In _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , volume 01, pages 67–72.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016.  You only look once: Unified, real-time object detection.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.
  * Stotz and Ströbel (2021) Peter Stotz and Phillip Ströbel. 2021.  bullinger-digital/gwalther-handwriting-ground- truth: Initial release.
  * Ströbel et al. (2022) Phillip Benjamin Ströbel, Simon Clematide, Martin Volk, and Tobias Hodel. 2022\.  Transformer-based htr for historical documents.
  * Thammarak et al. (2022) Karanrat Thammarak, Prateep Kongkla, Yaowarat Sirisathitkul, and Sarun Intakosum. 2022.  Comparative analysis of tesseract and google cloud vision for thai vehicle registration certificate.  _International Journal of Electrical and Computer Engineering_ , 22:1849–1858.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Wang et al. (2022) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Huggingface’s transformers: State-of-the-art natural language processing.
  * Yang et al. (2022) Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. 2022.  Image data augmentation for deep learning: A survey.
  * Zaidi et al. (2021) Syed Sahil Abbas Zaidi, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Asghar, and Brian Lee. 2021.  A survey of modern deep learning based object detection models.

## Appendix

##  Appendix A Annotatong the Bounding Boxes

This Appendix holds the details of the Visual Detection part of the pipeline,
described in Section 4.1, and the challenges we were confronted with.

###  A.1 The Task

To annotate the bounding boxes, the model is provided with a prompt describing
the lemma and the image. The model then returns a bounding box for the
requested object, which is the lemma in our case. Different prompts are
described in Table 4.

Prompt 1 |  | Cursive text upper left
---|---|---
Prompt 2 |  | Handwritten cursive word upper left
Prompt 3 | Length: 1-5: | Blue drawing in the upper left
Other: | Handwritten cursive word upper left
Prompt 4 | Length: 1-6: | Blue drawing in the upper left
Other: | Handwritten cursive word upper left

Table 4: Different prompts used for OFA.

###  A.2 Assumption about Bounding Boxes

Since we do not have any ground truth about the bounding boxes, we rely on
heuristics to verify the correctness of the boxes. One such heuristic is the
assumed linear relationship between the lemma length and the bounding box’s
width. While the height of the boxes is assumed to be similar across
instances, the lemma length must significantly impact the bounding box’s
width. To verify the results of the annotation process, we use box plots to
visualize the relationship between lemma length and width (cf. Fig. 10(a) –
10(d)).

###  A.3 Initial Implementation and Results

We use the RefCOCO-OFA model555Huggingface: OFA-Base-RefCOCO and modify it
four our purposes. Prompt one (cf. Tab. 4) is used to obtain the lemmas for
all images.

After running the model on the first instances with Prompt 1, we find that the
relationship between the box’s width and the lemma length does not look as
expected. Figure 10 illustrates this problem. Investigating the short lemmas,
we observe that the model often fails to annotate the record cards
appropriately. Often other textual objects are annotated, or the bounding box
stretches throughout the entire record card.

(a) First Prompt

(b) Second Prompt

(c) Third Prompt

(d) Fourth and final Prompt

Figure 10: Box-Plots for the width of the bounding boxes based on the lemma’s
length.

###  A.4 Two Different Prompts for Shorter and Longer Lemmata

After different experiments, Prompt 2 turned out to work appropriately for
shorter lemmas, but was, however, not suitable for longer ones. To combine the
strength of both prompts, we apply a conditional prompt based on the length of
the lemma using different cut-offs (5 or 6 characters). We find that using
Prompt 4 is the best-suited approach. The analysis of the relationship between
the bounding box widths and the length of the lemma for different prompts can
be seen in Figure 10.

##  Appendix B YOLO: Training and Inference

###  B.1 Training Results

Figure 11: YOLO Training Results.

###  B.2 Multiple Lemmas Detected by YOLO

Figure 12: All bounding boxes from instances where YOLO has detected more than
one bounding box. Figure 13: Bounding boxes of all instances to which the rule
largest bounding box in the upper left corner was applied to.

##  Appendix C Training details

We used the defaults from transformers (4.26.1), if not reported otherwise.

###  C.1 Standard Training

Parameter | Value
---|---
Seed | 42
Optimizer | AdamW
Epochs | 5
Decoder | GPT-2
Encoder | {BEIT, Swin, ViT}
Batch Size (Train & Test) | 64
Table 5: Parameters for the standard training.

###  C.2 Training with Augmentation

Parameter | Value
---|---
Seed | 42
Optimizer | AdamW
Epochs | {5, 20}
Decoder | GPT-2
Encoder | {BEIT, Swin, ViT}
Batch Size (Train & Test) | 64
Table 6: Parameters for training with augmentation.

###  C.3 Natural Language Generation

Parameter | Value
---|---
Max Length | 32
Early Stopping | True
No Repeat Ngram Size | 3
Length Penalty | 2.0
Number of Beams | 4
Table 7: Parameters for natural language generation.

###  C.4 Decoder Pre-Training

Parameter | Value
---|---
Seed | 42
Epochs | 10
Batch Size (Train & Test) | 192
Table 8: Parameters for pre-training of the decoder.

BEiT

    Bidirectional Encoder representation for Image Transformers
CER

    Character Error Rate
CNN

    Convolutional Neural Network
CTC

    Connectionist Temporal Classification
GCV

    Gloogle Cloud Vision
HTR

    Handwritten Text Recognition
MLW

    Medieval Latin Dictionary
OCR

    Optical Character Recognition
OFA

    One For All
RNN

    Recurrent Neural Network
SOTA

    state-of-the-art
Swin

    Shifted Window Transformer
ViT

    Vision Transformer
YOLO

    You Only Look Once
  *[HTR]: Handwritten Text Recognition
  *[SOTA]: state-of-the-art
  *[CER]: Character Error Rate
  *[MLW]: Medieval Latin Dictionary
  *[OCR]: Optical Character Recognition
  *[CTC]: Connectionist Temporal Classification
  *[RNN]: Recurrent Neural Network
  *[CNN]: Convolutional Neural Network
  *[OFA]: One For All
  *[YOLO]: You Only Look Once
  *[ViT]: Vision Transformer
  *[BEiT]: Bidirectional Encoder representation for Image Transformers
  *[Swin]: Shifted Window Transformer
  *[GCV]: Gloogle Cloud Vision