11institutetext:  Computer Vision Center, Universitat Autonoma de Barcelona,
Barcelona, Spain 22institutetext: AWS AI Labs 33institutetext: AllRead MLT
33email<EMAIL_ADDRESS>
33email<EMAIL_ADDRESS>
# Out-of-Vocabulary Challenge Report

Sergi Garcia-Bordils$\dagger$1,3 Andrés Mafla$\dagger$ 11 Ali Furkan
Biten$\dagger$ 11
Oren Nuriel* 22 Aviad Aberdam* 22 Shai Mazor* 22 Ron Litman* 22
Dimosthenis Karatzas 11

###### Abstract

This paper presents final results of the Out-Of-Vocabulary 2022 (OOV)
challenge. The OOV contest introduces an important aspect that is not commonly
studied by Optical Character Recognition (OCR) models, namely, the recognition
of unseen scene text instances at training time. The competition compiles a
collection of public scene text datasets comprising of 326,385 images with
4,864,405 scene text instances, thus covering a wide range of data
distributions. A new and independent validation and test set is formed with
scene text instances that are out of vocabulary at training time. The
competition was structured in two tasks, end-to-end and cropped scene text
recognition respectively. A thorough analysis of results from baselines and
different participants is presented. Interestingly, current state-of-the-art
models show a significant performance gap under the newly studied setting. We
conclude that the OOV dataset proposed in this challenge will be an essential
area to be explored in order to develop scene text models that achieve more
robust and generalized predictions.

$\dagger$$\dagger$footnotetext: Equal contribution$*$$*$footnotetext: Work
does not relate to Amazon position

##  1 Introduction

Scene-text detection and recognition plays a key role in a multitude of vision
and language tasks, such as visual question answering [5, 34], image
captioning [captioning] or image retrieval [26]. Performance on classic
benchmarks, such as ICDAR13 [13] or ICDAR15 [12] has noticeably increased
thanks to the surge of sophisticated deep learning models. Interest in this
field has gained traction in the last few years and, as a consequence,
multiple new datasets have appeared. Some of them have introduced diverse new
challenges, such as irregular text detection and recognition [6, 41] or
complex layout analysis [24]. At the same time, the scale of new datasets has
also noticeably increased, reducing the reliance on synthetic data [35, 16].

However, none of the existing benchmarks makes a distinction between out-of-
vocabulary (OOV) words and in-vocabulary (IV) words. By OOV word we refer to
text instances that have never been seen in the training sets of the most
common Scene Text understanding datasets to date. Recent research suggests
that current OCR systems over-rely on language priors to recognize text [38],
by exploiting their explicit or implicit language model. As a consequence,
while the performance on IV text is high, recognition performance on unseen
vocabulary is lower, showing poor generalization. Since OOV words can convey
important high-level information about the scene (such as prices, dates,
toponyms, URLs, etc.), performance of OCR systems on unseen vocabulary should
also be seen as an important characteristic.

With this motivation in mind we present the Out-of-Vocabulary Challenge, a
competition on scene text understanding where the focus is put on unseen
vocabulary words. This challenge is formed by two different tasks; an End-to-
End Text Recognition task and a Cropped Word Text Recognition task. In the
End-To-End task participants were provided with images and were expected to
localize and recognize all the text in the image at word granularity. In the
Cropped Word task the participants were presented with the cropped word
instances of the test set, and were asked to provide a transcription. In order
to be able to compare the performance of the submissions on seen and unseen
vocabulary, we decided to include both types of instances on the test sets and
report the results separately.

The dataset used for this competition is a collection of multiple existing
datasets. Some of the featured datasets were collected with text in mind,
while others used sources where the text is incidental. We have created our
own validation and test splits of the End-to-End challenge to contain at least
one OOV word per image. In the validation and test sets of the Cropped Word
Recognition task we include the cropped OOV and IV words from the End-to-End
dataset.

##  2 Related Work

The field of scene text recognition can be divided into two main tasks - text
detection and text recognition. The OOV challenge addresses methods which
either perform end-to-end text recognition [18, 23, 31, 9, 4, 42, 25, 15, 32]
and thus solve both tasks, or methods that tackle just text recognition [33,
3, 22, 36, 30, 40, 29, 2, 8, 1], and thus assume the words are already
extracted.

The problem of vocabulary reliance in scene text recognition was first
revealed by [38]. Through extensive experiments, they found that state-of-the-
art methods perform well on previously seen, in vocabulary, word images yet
generalize poorly to images with out of vocabulary words, never seen during
training. In addition, the authors proposed a mutual learning approach which
jointly optimizes two different decoder types, showing that this can alleviate
some of the problems of vocabulary reliance. [43] suggested a context-based
supervised contrastive learning framework, which pulls together clusters of
identical characters within various contexts and pushes apart clusters of
different characters in an embedding space. In this way they are able to
mitigate some of the gaps between in and out of vocabulary words.

##  3 Competition Protocol

The OOV Challenge took place from May to July of 2022. A training set was
given to participants at the beginning of May, but the images for the test set
were only made accessible for a window between June 15 and July 22.
Participants were asked to submit results obtained on the public test set
images rather than model executables. We rest on the scientific integrity of
the participants to adhere to the challenge’s specified guidelines.

The Robust Reading Competition (RRC) portal111https://rrc.cvc.uab.es/ served
as the Challenge’s host. The RRC site was created in 2011 to host the first
rigorous reading contests including text detection and identification from
scene photographs and born-digital images, and it has since expanded into a
fully-fledged platform for organizing academic competitions. The portal now
hosts 19 distinct challenges, with different tasks mostly related to scene
text detection and recognition, Scene-Text Visual Question Answering (ST-VQA)
and Document VQA. The RRC portal has more than 35,000 registered users from
more than 148 countries, and more than 77,000 submitted results have already
been evaluated. The findings in this report are an accurate reflection of the
submissions’ status at the end of the formal challenge period. The RRC portal
should be viewed as an archive of results, where any new findings contributed
after the compilation of this report will also be included. All submitted
findings are automatically analyzed, and the site provides per-task ranking
tables and visualization tools to examine the results.

##  4 The OOV Dataset

The OOV (Out Of Vocabulary) dataset encompasses a collection of images from 7
public datasets, namely: HierText [24], TextOCR [35], ICDAR13 [13], ICDAR15
[12], MLT19 [28], Coco-text [37] and OpenImages [16]. This dataset selection
aims to generalize the performance of models overcoming existing biases in
datasets [14].

The validation and test splits of the OOV dataset were defined to measure the
performance of models on unseen words at training time. To do this we
extracted all the words that appear at least once in the training and
validation splits of the datasets and, jointly with the 90k word dictionary
introduced by Jaderberg et al. [11], we created an in-vocabulary dictionary of
words. To create the test set of the OOV dataset we picked those images from
the original test sets which contained, at least, one word outside of this
vocabulary. Images in the validation dataset were picked from the training and
validation splits of the original datasets, and we only kept images that
contained words that appear once (and therefore do not appear in the training
split). We limited the number of images in the validation set to 5,000 images.
The rest of the images were used in the training split.

For this first iteration of the competition we focus on text instances in
which characters come from a limited alphabet. This alphabet is formed by the
Latin alphabet, numbers and a few punctuation signs 222See
https://rrc.cvc.uab.es/?ch=19&com=tasks for the full alphabet.. Words that
contain out of alphabet characters are not considered for the final evaluation
(in the End-to-End task they are treated as “don’t care”).

| # of Images | # of Cropped Words
---|---|---
Dataset | Train | Validation | Test | Train | Validation | Test
ICDAR13 | 229 | 0 | 233 | 795 | 54 | 0
ICDAR15 | 1000 | 0 | 500 | 4350 | 118 | 0
ICDAR MLT19 | 10000 | 0 | 10000 | 80937 | 8499 | 103297
MSRA-TD500 | 300 | 0 | 200 | - | - | -
COCO-Text | 43686 | 10000 | 10000 | 80549 | 6571 | 11688
TextOCR | 24902 | 0 | 3232 | 1155320 | 47019 | 96082
HierText | 8281 | 1724 | 1634 | 981329 | 59630 | 187190
Open Images V5 Text | 191059 | 16731 | 0 | 2066451 | 7788 | 0
OOV Dataset | 312612 | 5000 | 8773 | 4369731 | 128832 | 365842
Table 1: Dataset size comparison Figure 1: Number of words per image and
dataset of origin. Images that contain more than 150 words have been counted
in the last bin.

###  4.1 Dataset Analysis

Given that this dataset is a collection of datasets from multiple sources, it
has the benefit of featuring data that come from different sources and
annotation settings. Some of the featured datasets were originally collected
with text in mind, while in others the text is more incidental. COCO-Text is
an example of a dataset with purely incidental text. Since the source of the
images is the MS COCO [21] dataset, the images are not text biased like in
other datasets. In Figure 1 we can see the distribution of the number of words
per image and dataset of origin in the test set. Some datasets contain, in
general, a small number of instances per image (such as the aforementioned
COCO-Text), while others are more prone to contain images with tens or even
hundreds of words (HierText being the most prominent one). Consequently, our
test set is less biased towards specific distributions of words.

Table LABEL:tab:crops_stats shows the dataset of origin of the cropped words
for each one of the splits. The test split features a balanced number of
instances coming from 4 different sources, avoiding relying too much on a
specific dataset. The spatial distribution of the instances of the test set
can be seen in the Figure 2. Each dataset featured in the test set appears to
have different spatial distributions, a consequence of the original source of
the images. For example, COCO-Text and TextOCR originally come from datasets
that were not collected with text in mind (MS COCO [37] and Open Images V4
[17]), consequently the word instances are incidental and distributed over all
the image. On the other hand, the text featured on MLT19 is more focused and
more clustered around the center of the images. HierText was collected with
text in mind but contains much more text instances per image (as seen in the
Figure 1, which distributes the text uniformly over the images. Combining
datasets that contain images from different origins gives us a more varied and
rich test set. Finally, Figure 3 shows the distribution of the lengths of the
OOV words of the test set, separated per dataset. The distribution of the
lengths appears to be similar for the featured datasets.

(a) COCO-Text

(b) HierText

(c) MLT19

(d) TextOCR

(e) OOV

Figure 2: Text spatial distribution of the different datasets featured in the
test set. Figure 3: Character length of OOV words per dataset.

##  5 The OOV Challenge

The Out of Vocabulary challenge aims to evaluate the ability of text
extraction models to deal with words that they have never seen before. Our
motivation for organizing this challenge is the apparent over-reliance of
modern OCR systems on previously seen vocabulary. We argue that generalizing
well on in and out of vocabulary text should be considered as important as
generalizing well on text with different visual appearances or fonts. OOV text
can convey important semantic information about the scene, failing to
recognize a proper noun (such as the name of a street in the context of
autonomous driving) or a random string of numbers (such as a telephone number)
can result in unfortunate consequences.

Wan et. al. [38] call the phenomenon of memorizing the words the in the
training set “vocabulary reliance”. To prove this behaviour, the authors train
diverse text recognition models using the same data and the same backbone. The
results are reported using the IIIT-5k[27] dataset and they provide results on
both in-vocabulary and out-of-vocabulary text. Like us, they consider OOV
whichever words are not present in the test set, including the synthetic data.
Their results show a gap as small as 15.3% using CA-FCN [20] and as high as
22.5% using CRNN [33], proving how much OCR systems rely on learned
vocabulary.

Therefore, in this challenge we evaluate the entries putting special emphasis
on words that the models have never seen before. We hope that our curated
dataset and our evaluation protocol can be useful to the community to develop
more robust and unbiased OCR systems.

###  5.1 Task 1

The End-to-End task aims to evaluate the performance of the models in both
detection and recognition. Unlike some previous competitions in the RRC, we do
not provide any vocabulary. For each correct detection we look for a perfect
match between the proposed and the ground truth transcriptions. Evaluation is
case-sensitive, and punctuation signs are taken into account. The evaluation
procedure displays results on both IV and OOV text for each of the methods.
Below, we give brief descriptions for some of the submitted methods, as
provided by their authors.

CLOVA OCR DEER. An end-to-end scene text spotter based on a CNN backbone, a
Deformable Transformer Encoder [44], location decoder and text decoder. The
location decoder, based on the segmentation method (Differentiable
Binarization [20]), detects text regions, and the text decoder based on the
deformable transformer decoder recognizes each instance from image features
and detected location information. They use both the training data provided by
the challenge, as well as synthetic data.

Detector Free E2E. It is a detection free end-to-end text recognizer, where a
CNN with Deformable Encoder & Decoder is used. The models are trained with
data from the challenge and additional SynthText data synthesized with MJSynth
90k dictionary.

oCLIP & oCLIP_v2. For detection, they first pre-train their Deformable
ResNet-101 by using oCLIP on the provided training set. Then they train TESTR
[42], PAN and Mask TextSpotter with different backbones using the pre-trained
model. Finally, they combine results from different methods, different
backbones, and different scales together while for recognition, they adopt
SCATTER [22].

DB_threshold2_TRBA. The detector is based on Differentiable Binarization (DB)
[19]. The recognizer is TRBA from WIW [3]. TRBA denotes TPS + ResNet Backbone
+ BiLSTM + Attention. The models were not jointly trained. Since DB does not
output an up-vector, they rotated the detected region according to the aspect
ratio. CocoText has label noises (not case sensitive), and thus, they cleaned
the dataset using the teacher model. They use synthetic data (ST) as well as
challenge-provided data.

E2E_Mask. They only use the OOV dataset to train their model. In the detection
stage, they follow TBNet and Mask2Former as the base model with a multi-scale
training strategy. To combine the final detection results, they ensemble
different detectors with different backbones and different testing sizes. In
the recognition stage, they use a vision transformer model that consists of
ViT encoder and query-based decoder to generate the recognition results in
parallel.

###  5.2 Task 2

On the Cropped Word Recognition task the participants had to predict the
recognition for all the cropped words of the test set. There is a total of
313,751 cropped words in the test set, 271,664 of these words are in-
vocabulary and 42,087 are out of vocabulary. Like in Task 1, the evaluation is
case-sensitive, and punctuation signs are taken into account. Below, we
provide a brief description for some of the submitted methods.

OCRFLY_V2. They design a new text recognition framework for OOV-ST, named
Character level Adaptive Mutual Decoder (CAMD), where both multi-arch and
multi-direction autoregressive seq2seq heads are jointly used during training
and testing. CAMD adopts a CNN-ViT Backbone as encoder, and two different
vision-language adaptively balanced decoders: an LSTM and a Transformer
decoder, are built upon the aforementioned encoder. Only Syn90k and the
training splits given by the challenge are used for training.

OOV3decode. Three models are combined by voting. An Encoder-Decoder Framework
with a 12-ViT-based encoder, a CTC Combined Decoder, a CTC-Attention Combined
Decoder, and a Mix-CTC-Position-Attn Combined Decoder. They use 300w+
generated images and the challenge training data.

Vision Transformer Based Method (VTBM). They train several models with the
same Vision Transformer based backbone and various decoders (CTC and
Attention), and they ensemble them based on confidence. They first pre-trained
their models on nearly 10 million synthetic images and fine-tuned them on the
official training set. Common augmentations such as rotation, blur, etc. are
adopted; especially the image concatenate augmentation is used to mine textual
context information.

DAT. An Encoder-Decoder transformer-based encoder with 12 layers of VIT-based
block and $4\times 4$ patch size is used. An ensemble strategy is used to fuse
the results from three decoder types: a CTC-based decoder, an attention-based
decoder and a CTC+attention-based decoder.

OCRFLY. This is a simple baseline based on the seq2seq algorithm, where they
adopt a CNN-ViT Backbone as encoder and a 6-layer transformer as decoder. Only
Syn90k and the training splits are used for training.

###  5.3 Baselines

For Task 1, we evaluate the recent state of the art methods, TESTR [42],
TextTranSpotter[15], GLASS[32].

TESTR. We provide results for their pretrained model released on their
official code package333https://github.com/mlpc-ucsd/testr using the default
configurations.

TextTranSpotter. TextTranSpotter was trained following the fully-supervised
training protocol in the paper with the following datasets: pretrained on
SynthText, then fine-tuned on a mix of SynthText, ICDAR13, ICDAR15, TotalText
and TextOCR. The model weights were then frozen and the mask branch was
trained on SynthText and then on the mix of datasets.

GLASS. Training was performed on the following train datasets: SynthText,
ICDAR13, ICDAR15, TotalText and TextOCR. The model was pretrained for 250k
iterations with a batch size of 24, and then fine tuned for another 100k
iterations specifically on TextOCR with a batch size of 8 images. The
architecture and parameters chosen for the detection, recognition and fusion
branches, are detailed in [32].

For Task 2, we evaluate the models in two types of settings, the first when
trained on synthetic data and the second when trained on the real data. The
real data consists of the word crops introduced in the OOV dataset.

SCATTER. In both settings SCATTER was trained for 600k iterations from
scratch. We employ the exact same training procedure as described in the [22].
We use two selective-contextual refinement blocks and take the output of the
last one.

Baek et al. For the synthetic setting, we use the case-sensitive model
released in the official repository444https://github.com/clovaai/deep-text-
recognition-benchmark. For the real data setting the model was trained from
scratch using the same training procedure as published in the repository.

ABINET. We used the official
codebase555https://github.com/FangShancheng/ABINet and trained a case-
sensitive model on the MJ and ST synthetic datasets for the synthetic setting.
For the real setting, we also utilized the OOV dataset. In both cases, we only
trained the network end-to-end without the pretraining stages.

###  5.4 Evaluation Metrics

For Task 1 (End-to-End text detection and recognition) we use a modified
version of the evaluation method proposed by Wang et. al. [39]. This method
considers a correct match when one of the proposed detections overlaps with a
ground truth bounding box by more than 50% and their transcriptions match
(again, caring about the letter case and punctuation signs). Correctly matched
proposals count as true positives, while unmatched proposals count as false
positives. Unmatched ground truth annotations count as false negatives. Most
of the annotations of the datasets used to form the validation and test sets
have annotations with some sort of “unreadable” attribute. These words are
treated as “don’t care”, and do not affect (positively or negatively) the
results. As discussed earlier, words that contain characters that are out of
alphabet are also considered as “don’t care”. Jointly with the precision and
recall, we also report the harmonic mean (or F-score) of each method:

$Hmean=\frac{2*Recall*Precision}{Recall+Precision}$ (1)

Since our evaluation protocol has to distinguish between OOV and IV words, we
modified the evaluation procedure to ignore the opposite split during the
evaluation. For example, when we are evaluating on OOV words, in-vocabulary
words are treated as “don’t care”. This way, matched ground truth IV words do
not count as false positives, and unmatched words do not count as false
negatives. The opposite applies when evaluating for IV words. The number of
false positives is the same in both cases, since matched ground truth
annotations are either true positives or treated as “don’t care”.

Finally, we report the unweighted average of the OOV and IV Hmean, as a
balanced metric for this task. Our motivation behind this metric is to give
equal importance to both distributions, as ideally, increasing performance on
OOV words should not undermine performance on IV words.

For the Task 2 (Cropped Word Recognition) we report two metrics. The first
metric is the total edit distance between each predicted word and its ground
truth, considering equal costs for insertions, deletions and substitutions.
The second metric is word accuracy, which is calculated as the sum of
correctly recognized words divided by the total amount of text instances. We
provide both metrics for OOV and IV words. The final, balanced metric reported
for the Task 2 is the unweighted average of the OOV and IV accuracy. Similarly
to Task 1, we consider performance on both subsets equally important.

##  6 Results

In this section, we provide the results for the submitted methods on both
tasks. We also present baselines to compare to the submitted methods.

Method | Average | All | OOV | IV
---|---|---|---|---
Hmean | P | R | Hmean | P | R | Hmean | P | R | Hmean
TESTR[42] | 15.9 | 31.4 | 20.1 | 25.1 | 4.4 | 17.8 | 7.1 | 29.2 | 21.4 | 24.6
TextTranSpotter [15] | 18.6 | 37.4 | 25.0 | 29.9 | 4.5 | 16.4 | 7.0 | 35.5 | 26.2 | 30.1
GLASS[32] | 34.9 | 75.8 | 30.6 | 43.6 | 24.9 | 27.2 | 26.0 | 73.7 | 31.1 | 43.7
CLOVA OCR DEER | 42.39 | 67.17 | 52.04 | 58.64 | 18.58 | 48.72 | 26.9 | 64.51 | 52.49 | 57.88
Detector Free E2E | 42.01 | 66.15 | 52.44 | 58.5 | 17.97 | 49.35 | 26.35 | 63.44 | 52.86 | 57.67
oCLIP_v2 | 41.33 | 67.37 | 46.82 | 55.24 | 20.28 | 48.42 | 28.59 | 64.41 | 46.6 | 54.08
DB threshold2 TRBA | 39.1 | 64.08 | 49.93 | 56.13 | 15.26 | 42.29 | 22.43 | 61.6 | 50.96 | 55.78
E2E_Mask | 32.13 | 47.9 | 54.14 | 50.83 | 8.64 | 46.73 | 14.58 | 45.2 | 55.14 | 49.68
YYDS | 28.68 | 51.53 | 35.54 | 42.07 | 10.63 | 33.36 | 16.12 | 48.57 | 35.83 | 41.24
sudokill-9 | 28.34 | 51.62 | 34.08 | 41.06 | 11.03 | 33.22 | 16.56 | 48.54 | 34.2 | 40.12
PAN | 28.13 | 50.5 | 34.81 | 41.21 | 10.5 | 33.58 | 16.0 | 47.45 | 34.98 | 40.27
oCLIP | 24.04 | 47.72 | 7.51 | 12.98 | 41.21 | 48.42 | 44.52 | 17.46 | 1.98 | 3.55
DBNetpp | 20.34 | 39.42 | 27.0 | 32.05 | 5.62 | 20.75 | 8.85 | 37.15 | 27.84 | 31.83
TH-DL | 9.32 | 18.39 | 13.23 | 15.39 | 2.16 | 10.87 | 3.6 | 16.89 | 13.55 | 15.04

Table 2: Harmonic mean, precision and recall for the entire dataset (All), in
vocabulary (IV) and out of vocabulary (OOV) across different baseline methods.
The average Hmean is the unweighted average of IV and OOV.

###  6.1 Task 1

We provide all the results in Table 2 where the top part represents the
baselines while the bottom part is for the submitted methods. We present
results on two different subsets, namely in-vocabulary (IV) and out-of-
vocabulary (OOV). Moreover, we rank the methods according to the balanced
Average Hmean metric.

As can be appreciated from Table 2, CLOVA OCR DEER is the winning method in
terms of Average Hmean. The best method in terms of OOV Hmean is oCLIP,
surpassing the second best method by +17.6 points. However, we see that
oCLIP’s performance on IV set is the lowest between the participated methods.
This observation makes us wonder if there is a trade-off between the
performance of IV and OOV in the model.

Method | Train Set | Total | IV | OOV
---|---|---|---|---
Word Acc$\uparrow$ | Word Acc$\uparrow$ | ED$\downarrow$ | Word Acc$\uparrow$ | ED$\downarrow$
ABINet [8] | Syn | 38.01 | 50.29 | 342,552 | 25.73 | 115536
Baek et al [3] | Syn | 44.47 | 52.61 | 365,566 | 36.34 | 114,101
SCATTER [22] | Syn | 47.79 | 56.85 | 321,101 | 38.74 | 103,928
ABINet [8] | Real | 59.84 | 71.13 | 176,126 | 48.55 | 67478
Baek et al [3] | Real | 64.97 | 75.98 | 138,479 | 53.96 | 54,346
SCATTER [22] | Real | 66.68 | 77.98 | 128,219 | 55.38 | 52,535
OCRFLY_v2 | Syn + Real | 70.31 | 81.02 | 123,947 | 59.61 | 46,048
OOV3decode | Syn + Real | 70.22 | 81.58 | 94,259 | 58.86 | 40,175
VTBM | Syn + Real | 70.00 | 81.36 | 94,701 | 58.64 | 40,187
DAT | - | 69.90 | 80.78 | 96,513 | 59.03 | 40,082
OCRFLY | Syn + Real | 69.83 | 80.63 | 131,232 | 59.03 | 53,243
GGUI | - | 69.80 | 80.74 | 96,597 | 58.86 | 40,171
vitE3DCV | Syn + Real | 69.74 | 80.74 | 96,477 | 58.74 | 40,115
DataMatters | Syn + Real | 69.68 | 80.71 | 96,544 | 58.65 | 40,177
MaskOCR | Real | 69.63 | 80.60 | 108,894 | 58.65 | 44,971
SCATTER | Syn + Real | 69.58 | 79.72 | 113,482 | 59.45 | 43,89
Summer | Syn + Real | 68.77 | 79.48 | 103,211 | 58.06 | 42,118
LMSS | Syn + Real | 68.46 | 80.81 | 116,503 | 56.11 | 51,165
UORD | Real | 68.28 | 79.28 | 118,185 | 57.27 | 48,517
PTVIT | Syn + Real | 66.29 | 77.52 | 120,449 | 55.06 | 49,41
GORDON | Syn + Real | 65.86 | 77.25 | 124,347 | 54.47 | 48,907
TRBA_CocoValid | Syn + Real | 63.98 | 77.76 | 132,781 | 50.20 | 60,693
HuiGuan | Real | 63.73 | 74.77 | 162,87 | 52.69 | 68,926
EOCR | - | 46.66 | 55.30 | 350,166 | 38.02 | 113,317
NNRC | - | 38.54 | 45.36 | 405,603 | 31.73 | 136,384
NN | - | 37.17 | 43.38 | 426,074 | 30.97 | 144,032
CCL | Real | 31.06 | 47.40 | 552,57 | 14.73 | 202,087
Table 3: Word accuracy and Edit Distance for state of the art recognition
models trained on different datasets.

Regarding the architecture choices of the submitted methods, almost all of
them, especially the top performing ones make use of the ViT [7] architecture
either as backbone for the recognition pipeline or directly for extracting
features. Another commonly preferred building block is CTC [10] based encoder
or decoder mechanism. Furthermore, the top 2 performing methods, CLOVA OCR
DEER and Detector Free E2E, utilize a Deformable DETR architecture [44],
showing its effectiveness in the end-to-end text recognition task. Lastly, we
observe that almost all the methods make use of synthetic data to either pre-
train or finetune together with the real data.

###  6.2 Task 2

The results can be found in Table 3 where the top part represents the
baselines while the bottom part is for the submitted methods. We present
results exactly the same way as in Task 1 in two different subsets, namely in-
vocabulary (IV) and out-of-vocabulary (OOV) words. Moreover, we rank the
methods according to the balanced Total Word Accuracy which is calculated as
the average of the Word Accuracy of IV and OOV, giving the same emphasis on
both sets. As can be seen from Table 3, our baselines trained with real data
clearly outperform the ones trained with synthetic data. We also observe in
our baselines that having a boost in IV words also translates to an
improvement in OOV performance.

Regarding the submitted methods, the winner in total word accuracy is
OCRFLY_v2 even though by a slight margin. We also notice that OCRFLY_v2 is the
best method in OOV performance in terms of accuracy; however, DAT is the best
method in terms of edit distance. On the other hand, OOV3decode achieves state
of the art performance in IV. We note that all top 3 methods are trained with
both synthetic and real data. As a matter of fact, most of the methods are
trained with combined data, confirming the effectiveness of the usage of the
combined data. In terms of the favored architectures, we see a similar trend
in Task 2 as in Task 1. We observe that the ViT [7] architecture being used in
most of the methods combined with CTC and Attention. This demonstrates the
clear advantage of the Transformer architecture over LSTMs and RNNs which was
state-of-the-art in text recognition literature previously.

##  7 Analysis

$0$$5$$10$$15$$20$$25$$0.1$$0.3$$0.5$$0.7$Word lengthAverage recallDetector
Free E2E

$0$$5$$10$$15$$20$$25$$0.1$$0.3$$0.5$$0.7$Word lengthAverage recallE2E
TextSpotter

$0$$5$$10$$15$$20$$25$$0.1$$0.3$$0.5$$0.7$Word lengthAverage recalloCLIP_v2

$0$$5$$10$$15$$20$$25$$0.1$$0.3$$0.5$$0.7$Word lengthAverage recallCLOVA OCR
DEER

Figure 4: Average recall for different character lengths in the End-to-End task. The red and blue lines represent the average recall for IV and OOV words, respectively. |
---|---
Figure 5: Model accuracy according to the type of scene text categorized by
regular expressions in Task 1. Best viewed in color.

In this section, we analyze the performance of the top submitted methods in
terms of word length and different word categories. Specifically, we study
whether the word length has an effect on the performance of a model and how
well they perform in terms of the category of the word.

###  7.1 Task 1

Figure 4 shows the performance of the top 4 methods for words of different
character lengths. The metric reported is the average recall of each method
for both OOV and IV words. For all methods, the results on IV words shorter
than 15 characters are higher than for OOV words. The models seem to have less
difficulty dealing with short in-vocabulary words, most likely as a product of
vocabulary reliance. For IV words longer than 15 characters, the performance
is comparable or sometimes even worse than for OOV words of the same lengths.
Interestingly, results on OOV words seem to be consistent regardless of the
word length, although we seem to observe fluctuations on the score for words
longer than 20 characters. This could be attributed to statistical anomalies
due to the low number of OOV words of this length (as seen in Figure 3, there
are fewer words with of than 20 characters).

Additionally, Figure 5 shows the recognition accuracy by employing an
automatic categorization of words via the usage of regular expressions. In
both scenarios (IV and OOV), whenever a scene text instance is mostly formed
by numbers (units, prices, phone numbers) the accuracy remains uniform. We
hypothesize that this outcome is a direct effect of the distribution of the
training data that contains numbers. Since numbers do not follow a specific
distribution as characters in a given language, scene text models are more
flexible at correctly predicting numbers. Subsequently, the direct effect is
found in categories where numbers are not common or absent at all, such as in
emails, urls and others.

###  7.2 Task 2

$0$$5$$10$$15$$20$$25$$0.3$$0.5$$0.7$$0.9$Word lengthAverage accuracyDAT

$0$$5$$10$$15$$20$$25$$0.3$$0.5$$0.7$$0.9$Word lengthAverage accuracyOCRFLY

$0$$5$$10$$15$$20$$25$$0.3$$0.5$$0.7$$0.9$Word lengthAverage accuracyWow

$0$$5$$10$$15$$20$$25$$0.3$$0.5$$0.7$$0.9$Word lengthAverage accuracyViT

Figure 6: Average precision for different character lengths in the Cropped Word Recognition task. The red and blue lines represent the average recall for IV and OOV words, respectively. |
---|---
Figure 7: Model accuracy according to the type of scene text categorized by
regular expressions in Task 2. Best viewed in color.

Figure 4 shows the performance of the top 4 methods for words of different
lengths of characters. In this case we feature the average precision
(correctly recognized words) of the top 4 submitted methods, for both OOV and
IV. We observe a similar pattern as in the End-to-End task, the models seem to
perform better on IV words of character length of 15 or less. The results on
OOV words also seem to remain consistent for different character lengths. Like
we have observed in Task 1, performance on OOV words of more than 15
characters is similar or superior to the IV of the same length, which suggest
that OCR systems have trouble with longer sequences, regardless of whether
they are in vocabulary or not. Similarly to the previous subsection, we show
in Figure 7 the performance of the top 4 models on different word categories.
Since in Task 2, no detection is involved, we observe a slightly different
behaviour compared to the previous task. Whenever solely numbers are contained
in a cropped image, the accuracy in IV and OOV remains similar, as in the case
of phone numbers. However, if numbers and characters are expected to be found
in a cropped word, the gap in performance is very large in the rest of the
categories, except for emails. Even though the performance of all models is
very close in task 2 ( Table 3), we observe that the winning entry, OCRFLY,
gets an edge on prices, emails and other categories in IV and on phone numbers
and emails in OOV words.

##  8 Conclusion and Future Work

In this work, we introduce a new task called Out-Of-Vocabulary Challenge, in
which end-to-end text recognition and cropped scene text recognition were the
two challenges that made up the competition. In order to cover a wide range of
data distributions, the competition creates a collection of open scene text
datasets that include 326K images and 4.8M scene text instances. Surprisingly,
state-of-the-art models exhibit a considerable performance discrepancy on the
OOV task. This is especially apparent in the performance gap between in-
vocabulary and out-of-vocabulary words. We come to the conclusion that in
order to create scene text models that produce more reliable and generalized
predictions, the OOV dataset suggested in this challenge would be a crucial
area to investigate forward.

## References

  * [1] Aberdam, A., Ganz, R., Mazor, S., Litman, R.: Multimodal semi-supervised learning for text recognition. arXiv preprint arXiv:2205.03873 (2022)
  * [2] Aberdam, A., Litman, R., Tsiper, S., Anschel, O., Slossberg, R., Mazor, S., Manmatha, R., Perona, P.: Sequence-to-sequence contrastive learning for text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15302–15312 (2021)
  * [3] Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S.J., Lee, H.: What is wrong with scene text recognition model comparisons? dataset and model analysis. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4715–4723 (2019)
  * [4] Baek, Y., Shin, S., Baek, J., Park, S., Lee, J., Nam, D., Lee, H.: Character region attention for text spotting. In: European Conference on Computer Vision. pp. 504–521. Springer (2020)
  * [5] Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., Karatzas, D.: Scene text visual question answering. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4291–4301 (2019)
  * [6] Ch’ng, C.K., Chan, C.S.: Total-text: A comprehensive dataset for scene text detection and recognition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 935–942. IEEE (2017)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [8] Fang, S., Xie, H., Wang, Y., Mao, Z., Zhang, Y.: Read like humans: autonomous, bidirectional and iterative language modeling for scene text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7098–7107 (2021)
  * [9] Feng, W., He, W., Yin, F., Zhang, X.Y., Liu, C.L.: Textdragon: An end-to-end framework for arbitrary shaped text spotting. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9076–9085 (2019)
  * [10] Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning. pp. 369–376 (2006)
  * [11] Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227 (2014)
  * [12] Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., Lu, S., et al.: Icdar 2015 competition on robust reading. In: 2015 13th international conference on document analysis and recognition (ICDAR). pp. 1156–1160. IEEE (2015)
  * [13] Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras, L.P.: Icdar 2013 robust reading competition. In: 2013 12th international conference on document analysis and recognition. pp. 1484–1493. IEEE (2013)
  * [14] Khosla, A., Zhou, T., Malisiewicz, T., Efros, A.A., Torralba, A.: Undoing the damage of dataset bias. In: European Conference on Computer Vision. pp. 158–171. Springer (2012)
  * [15] Kittenplon, Y., Lavi, I., Fogel, S., Bar, Y., Manmatha, R., Perona, P.: Towards weakly-supervised text spotting using a multi-task transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4604–4613 (2022)
  * [16] Krylov, I., Nosov, S., Sovrasov, V.: Open images v5 text annotation and yet another mask text spotter. In: Asian Conference on Machine Learning. pp. 379–389. PMLR (2021)
  * [17] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4. International Journal of Computer Vision 128(7), 1956–1981 (2020)
  * [18] Li, H., Wang, P., Shen, C.: Towards end-to-end text spotting with convolutional recurrent neural networks. In: Proceedings of the IEEE international conference on computer vision. pp. 5238–5246 (2017)
  * [19] Liao, M., Wan, Z., Yao, C., Chen, K., Bai, X.: Real-time scene text detection with differentiable binarization. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 11474–11481 (2020)
  * [20] Liao, M., Zhang, J., Wan, Z., Xie, F., Liang, J., Lyu, P., Yao, C., Bai, X.: Scene text recognition from two-dimensional perspective. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 8714–8721 (2019)
  * [21] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [22] Litman, R., Anschel, O., Tsiper, S., Litman, R., Mazor, S., Manmatha, R.: Scatter: selective context attentional scene text recognizer. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11962–11972 (2020)
  * [23] Liu, Y., Chen, H., Shen, C., He, T., Jin, L., Wang, L.: Abcnet: Real-time scene text spotting with adaptive bezier-curve network. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9809–9818 (2020)
  * [24] Long, S., Qin, S., Panteleev, D., Bissacco, A., Fujii, Y., Raptis, M.: Towards end-to-end unified scene text detection and layout analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1049–1059 (2022)
  * [25] Lyu, P., Liao, M., Yao, C., Wu, W., Bai, X.: Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 67–83 (2018)
  * [26] Mafla, A., Dey, S., Biten, A.F., Gomez, L., Karatzas, D.: Multi-modal reasoning graph for scene-text based fine-grained image classification and retrieval. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 4023–4033 (2021)
  * [27] Mishra, A., Alahari, K., Jawahar, C.: Scene text recognition using higher order language priors. In: BMVC-British machine vision conference. BMVA (2012)
  * [28] Nayef, N., Patel, Y., Busta, M., Chowdhury, P.N., Karatzas, D., Khlif, W., Matas, J., Pal, U., Burie, J.C., Liu, C.l., et al.: Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019. In: 2019 International conference on document analysis and recognition (ICDAR). pp. 1582–1587. IEEE (2019)
  * [29] Nuriel, O., Fogel, S., Litman, R.: Textadain: Fine-grained adain for robust text recognition. arXiv preprint arXiv:2105.03906 (2021)
  * [30] Qiao, Z., Zhou, Y., Yang, D., Zhou, Y., Wang, W.: Seed: Semantics enhanced encoder-decoder framework for scene text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13528–13537 (2020)
  * [31] Qin, S., Bissacco, A., Raptis, M., Fujii, Y., Xiao, Y.: Towards unconstrained end-to-end text spotting. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4704–4714 (2019)
  * [32] Ronen, R., Tsiper, S., Anschel, O., Lavi, I., Markovitz, A., Manmatha, R.: Glass: Global to local attention for scene-text spotting. arXiv preprint arXiv:2208.03364 (2022)
  * [33] Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence 39(11), 2298–2304 (2016)
  * [34] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317–8326 (2019)
  * [35] Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., Hassner, T.: Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8802–8812 (2021)
  * [36] Slossberg, R., Anschel, O., Markovitz, A., Litman, R., Aberdam, A., Tsiper, S., Mazor, S., Wu, J., Manmatha, R.: On calibration of scene-text recognition models. arXiv preprint arXiv:2012.12643 (2020)
  * [37] Veit, A., Matera, T., Neumann, L., Matas, J., Belongie, S.: Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140 (2016)
  * [38] Wan, Z., Zhang, J., Zhang, L., Luo, J., Yao, C.: On vocabulary reliance in scene text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11425–11434 (2020)
  * [39] Wang, K., Babenko, B., Belongie, S.: End-to-end scene text recognition. In: 2011 International conference on computer vision. pp. 1457–1464. IEEE (2011)
  * [40] Wang, Y., Xie, H., Fang, S., Wang, J., Zhu, S., Zhang, Y.: From two to one: A new scene text recognizer with visual language modeling network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14194–14203 (2021)
  * [41] Yao, C., Bai, X., Liu, W., Ma, Y., Tu, Z.: Detecting texts of arbitrary orientations in natural images. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 1083–1090. IEEE (2012)
  * [42] Zhang, X., Su, Y., Tripathi, S., Tu, Z.: Text spotting transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9519–9528 (2022)
  * [43] Zhang, X., Zhu, B., Yao, X., Sun, Q., Li, R., Yu, B.: Context-based contrastive learning for scene text recognition. AAAI (2022)
  * [44] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)