# From Competition to Collaboration: Making Toy Datasets on Kaggle Clinically
Useful for Chest X-Ray Diagnosis Using Federated Learning

Pranav Kulkarni1 Adway Kanhere1,2 Paul H. Yi1 Vishwa S. Parekh1
1University of Maryland Medical Intelligent Imaging (UM2ii) Center
Department of Diagnostic Radiology and Nuclear Medicine
University of Maryland School of Medicine
Baltimore, MD 21201
{pkulkarni, pyi<EMAIL_ADDRESS>
2Department of Biomedical Engineering
Johns Hopkins University
Baltimore, MD 21218
<EMAIL_ADDRESS>
###### Abstract

Chest X-ray (CXR) datasets hosted on Kaggle, though useful from a data science
competition standpoint, have limited utility in clinical use because of their
narrow focus on diagnosing one specific disease. In real-world clinical use,
multiple diseases need to be considered since they can co-exist in the same
patient. In this work, we demonstrate how federated learning (FL) can be used
to make these toy CXR datasets from Kaggle clinically useful. Specifically, we
train a single FL classification model (‘global‘) using two separate CXR
datasets – one annotated for presence of pneumonia and the other for presence
of pneumothorax (two common and life-threatening conditions) – capable of
diagnosing both. We compare the performance of the global FL model with models
trained separately on both datasets (‘baseline‘) for two different model
architectures. On a standard, naive 3-layer CNN architecture, the global FL
model achieved AUROC of 0.84 and 0.81 for pneumonia and pneumothorax,
respectively, compared to 0.85 and 0.82, respectively, for both baseline
models (p>0.05). Similarly, on a pretrained DenseNet121 architecture, the
global FL model achieved AUROC of 0.88 and 0.91 for pneumonia and
pneumothorax, respectively, compared to 0.89 and 0.91, respectively, for both
baseline models (p>0.05). Our results suggest that FL can be used to create
global ‘meta‘ models to make toy datasets from Kaggle clinically useful, a
step forward towards bridging the gap from bench to bedside.

##  1 Introduction

Chest X-Ray (CXR) is the most commonly ordered medical imaging study globally
and is critical for screening many life threatening conditions (e.g.,
pneumonia). Accordingly, many large-scale public CXR datasets have been
released through curation of [8, 2, 3, 5]. These, in turn, have resulted in
numerous data science competitions hosted on platforms like Kaggle (e.g., RSNA
pneumonia detection challenge [7]), resulting in expert-level performance for
disease diagnoses.

Although useful from a data science competition standpoint, these Kaggle-
hosted CXR datasets have limited clinical utility because of their narrow
focus on one single diagnostic task. For example, the two Kaggle CXR
competitions hosted by RSNA and SIIM have focused on diagnosis of a single
disease, like pneumonia and pneumothorax [9]. Although impressive results have
resulted from these competitions, their utility is limited given the dozens of
diagnoses that could present in real-world clinical practice. Therefore, a
method to harmonize these toy datasets to train a clinically-useful model
could revolutionize how small, narrowly-focused datasets can be leveraged in
aggregate for development of clinically-relevant deep learning models.

We propose CheXViz, a federated learning (FL) framework for training a single
model on spatially distributed datasets with different disease annotations
into a ‘global‘ meta-deep learning model. Briefly, FL is a machine learning
technique that approaches the problem from a multi-domain and multi-task
perspective. By using a decentralized and distributed approach, consisting of
a central server and nodes, a global ‘meta‘ model can be trained to generalize
distributed tasks with non-iid labels. During each training step (‘FL round‘),
every node trains a local model. Then, the weights across all nodes are
aggregated by the central server and redistributed back to the nodes. In
medical imaging, FL has enabled training of large-scale ‘global‘ deep learning
models using datasets spread across multiple institutions without sharing
sensitive patient data. In this preliminary work, we demonstrate the utility
of CheXViz for training a single model to diagnose pneumonia and pneumothorax
using two toy datasets from Kaggle for these two respective diseases. Put
another way, we demonstrate how toy datasets from Kaggle can be made
clinically useful using FL.

Figure 1: Illustration of the CheXViz framework

##  2 Methods

###  2.1 CheXViz

We developed CheXViz using a multi-task FL setup. The CheXViz model is
initialized as a deep neural network consisting of two distinct blocks - a
representation block and a task block. The CheXViz model is distributed across
all the participating nodes to train their tasks. During training, only the
weights corresponding to the representation block are aggregated and
redistributed by the central server back to the nodes, thereby preserving
task-related information for each node in their task block. Figure 1
illustrates the CheXViz framework.

###  2.2 Experimental Setup

We evaluated the CheXViz framework for the task of training a generalized
global model that can classify cases of pneumonia and pneumothorax using
distributed and non-iid CXR datasets from the RSNA Pneumonia Detection and the
SIIM-ACR Pneumothorax Segmentation competitions on Kaggle. Our experiments
involved two different deep network architectures: A naive convolutional
neural network (CNN) consisting of 3 convolutional layers (‘standard‘ model)
and a pre-trained DenseNet121 architecture using transfer learning (TL) [1].
We implemented Federated Averaging (FedAvg) for model weight aggregation as
described in [4].

The standard baseline models for pneumonia and pneumothorax classification
were trained for 300 epochs using a learning rate scheduler with an initial
learning rate of 1e-3. During FL, we trained the standard global model for 300
FL rounds where for each round, models were locally trained for 1 epoch before
aggregation. The DenseNet121 model was initialized with pre-trained ImageNet
weights (‘base model‘) and a new classification head was trained for 30 epochs
with a learning rate of 1e-3 while the rest of the base model weights were
frozen and then fine tuned with a slower learning rate of 1e-5 to prevent
overfitting. For the baseline models, the models were fine tuned for 150
epochs. For FL, we utilized FedAvg during the fine tuning step of TL. The
global model was fine tuned for 150 FL rounds with 1 epoch before aggregation
with a learning rate of 1e-5. All the models were trained and evaluated on a
Google Cloud VM with four NVIDIA T4 GPUs.

We computed sensitivity, specificity, area under precision-recall (AUPR)
curve, and the area under receiver operating characteristic (AUROC) curve for
each model. The AUROC values between the FL global model and the task’s
baseline model were compared using bootstrapping and a paired t-test.
Statistical significance was defined as $p<0.05$.

##  3 Results

(a)

(b)

(c)

(d)

Figure 2: ROC curves obtained from baseline and CheXViz models evaluated
across both the datasets.

The CheXViz framework trained ‘meta‘ models demonstrated excellent performance
compared to the baseline models for the diagnostic classification of pneumonia
and pneumothorax abnormalities, as shown in Figure 2. For pneumonia
classification with the standard model architecture, the baseline and FL
models achieved a validation AUROC of 0.85 and 0.84 respectively. We observed
a similar trend with the DenseNet121 architecture, where the baseline and FL
models achieved a validation AUROC of 0.89 and 0.88 respectively. For
pneumothorax classification with the standard model architecture, the baseline
and FL models achieved a validation AUROC of 0.82 and 0.81 respectively.
Again, a similar result was achieved with the DenseNet121 architecture, where
the baseline and FL models both achieved a validation AUROC of 0.91. The
results are detailed in Appendix Table 1.

We further visualized the Grad-CAM outputs for evaluating the explainability
and generalizability of the models [6]. Our preliminary analysis suggests that
the heatmaps from CheXViz models demonstrate higher and focused activations
within the lungs, compared to the baseline models as shown in Appendix Figure
3. For future work, we intend to outline a methodology to quantify the
generalizability of models for CXR classification using Grad-CAM heatmaps.

##  4 Discussion

Although Kaggle CXR datasets and data science competitions have made an
indelible impact on data science and AI for healthcare, they are still a far
cry from being clinically useful datasets. This is understandable, given the
challenges in curating expert-level annotations for diseases, and ostensibly
why these Kaggle-hosted competitions have focused largely on single
diseases[7, 9]. Nevertheless, there is a gap between these toy datasets and
clinical utility. Put another way, it is unclear how to use these datasets to
train clinically-useful models capable of detecting multiple diseases. Our
findings demonstrate that our FL framework (CheXViz) can be used to create
global ‘meta‘ models to make toy datasets from Kaggle clinically useful, a
large step forward towards bridging the gap from bench to bedside. Although
preliminary in nature and focusing on only two datasets, our framework and
results are extensible to any number of datasets and disease labels, as well
as tasks beyond classification (e.g., segmentation and object detection). It
is our hope that our work can be a first step towards moving Kaggle CXR
datasets from competition to collaboration and transform these toy datasets
into clinically useful models.

## References

  * Huang et al. [2017] G. Huang et al.  Densely connected convolutional networks.  _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.  doi: https://doi.org/10.48550/arXiv.1608.06993.
  * Irvin et al. [2019] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 590–597, 2019.
  * Johnson et al. [2019] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  _arXiv preprint arXiv:1901.07042_ , 2019.
  * McMahan and more [2017] B. McMahan and more.  Communication-Efficient Learning of Deep Networks from Decentralized Data.  In _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , volume 54 of _Proceedings of Machine Learning Research_ , pages 1273–1282. PMLR, 20–22 Apr 2017.
  * Nguyen et al. [2022] H. Q. Nguyen, K. Lam, L. T. Le, H. H. Pham, D. Q. Tran, D. B. Nguyen, D. D. Le, C. M. Pham, H. T. Tong, D. H. Dinh, et al.  Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations.  _Scientific Data_ , 9(1):1–7, 2022.
  * Selvaraju et al. [2017] R. R. Selvaraju et al.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  _Proceedings of the IEEE international conference on computer vision_ , 2017.  doi: https://doi.org/10.1007/s11263-019-01228-7.
  * Shih et al. [2019] G. Shih, C. C. Wu, S. S. Halabi, M. D. Kohli, L. M. Prevedello, T. S. Cook, A. Sharma, J. K. Amorosa, V. Arteaga, M. Galperin-Aizenberg, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  _Radiology. Artificial intelligence_ , 1(1), 2019.
  * Wang et al. [2017] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2097–2106, 2017.
  * Yi et al. [2021] P. H. Yi, T. K. Kim, E. Siegel, and Y.-F.-A. N.  Demographic reporting in publicly available chest radiograph data sets: Opportunities for mitigating sex and racial disparities in deep learning models.  _Journal of the American College of Radiology_ , 19(1 Pt B):192–200, 2021.

## Societal Impact

Our work has a potential positive societal impact by taking an important step
towards translational research. Our work can be a first step towards moving
Kaggle CXR datasets from competition to collaboration and transform these toy
datasets into clinically useful models. However, the use of FL can potentially
lead to privacy and security risks, making the system vulnerable to client and
server attacks, thereby leading to potentially negative societal impacts. We
are actively working to address these vulnerabilities in FL systems.

##  Appendix A Appendix

Table 1: Model Metrics Task | Model | Loss | Sensitivity | Specificity | AUPR | AUROC | p-value
---|---|---|---|---|---|---|---
Pneumonia | Standard | 0.38 | 82.57 | 71.97 | 0.63 | 0.85 | -
| Standard w/ FL | 0.39 | 78.01 | 74.41 | 0.61 | 0.84 | 0.10
| DenseNet121 | 0.34 | 84.90 | 76.76 | 0.71 | 0.89 | -
| DenseNet121 w/ FL | 0.35 | 80.08 | 79.79 | 0.70 | 0.88 | 0.19
Pneumothorax | Standard | 0.41 | 74.13 | 75.89 | 0.54 | 0.82 | -
| Standard w/ FL | 0.42 | 80.22 | 69.87 | 0.52 | 0.81 | 0.71
| DenseNet121 | 0.31 | 91.30 | 76.31 | 0.73 | 0.91 | -
| DenseNet121 w/ FL | 0.31 | 84.57 | 83.10 | 0.73 | 0.91 | 0.76

(a) Pneumonia

(b) Pneumothorax

Figure 3: Grad-CAM Visualization of True Positives


# On the Hidden Mystery of OCR in
Large Multimodal Models

Yuliang Liu1, Zhang Li1, Biao Yang1, Chunyuan Li2, Xu-Cheng Yin3, Cheng-Lin
Liu4, Lianwen Jin5, Xiang Bai1
1Huazhong University of Science and Technology 2Microsoft Research, Redmond
3University of Science and Technology Beijing 4Chinese Academy of Sciences
5South China University of Technology
<EMAIL_ADDRESS>
Corresponding Author.

###### Abstract

Large models have recently played a dominant role in natural language
processing and multimodal vision-language learning. However, their
effectiveness in text-related visual tasks remains relatively unexplored. In
this paper, we conducted a comprehensive evaluation of Large Multimodal
Models, such as GPT4V and Gemini, in various text-related visual tasks
including Text Recognition, Scene Text-Centric Visual Question Answering
(VQA), Document-Oriented VQA, Key Information Extraction (KIE), and
Handwritten Mathematical Expression Recognition (HMER). To facilitate the
assessment of Optical Character Recognition (OCR) capabilities in Large
Multimodal Models, we propose OCRBench, a comprehensive evaluation benchmark.
Our study encompasses 29 datasets, making it the most comprehensive OCR
evaluation benchmark available. Furthermore, our study reveals both the
strengths and weaknesses of these models, particularly in handling
multilingual text, handwritten text, non-semantic text, and mathematical
expression recognition. Most importantly, the baseline results showcased in
this study could provide a foundational framework for the conception and
assessment of innovative strategies targeted at enhancing zero-shot multimodal
techniques. The evaluation pipeline and benchmark are available at
https://github.com/Yuliang-Liu/MultimodalOCR.

_K_ eywords Large Multimodal Model, OCR, Text Recognition, Scene Text-Centric
VQA, Document-Oriented VQA, Key Information Extraction, Handwritten
Mathematical Expression Recognition

##  1 Introduction

The advent of large models has unlocked a wealth of potentials in the realm of
advanced computing. In recent years, there has been an explosion in the
development of large language models (LLM) such as ChatGPT [1] and GPT-4 [2],
giving rise to extraordinary applications in zero-shot task transfer to many
new real-world scenarios. The success of proprietary LLMs has stimulated
tremendous interest in developing open-source LLMs. Among them, LLaMA [3] is
an open-source LLM that matches the performance of GPT-3, followed by Alpaca
[4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to
follow human instruction, reporting impressive performance compared with
proprietary LLMs.

The success of large models has also been extended to the multimodal vision-
language space [7], leading to a line of research on large multimodal models
(LMM), including contrastive learning [8, 9, 10, 11] and generative modeling
[2, 12, 13, 14, 15]. Surprisingly, Liu et al. [15] show that LMM exhibits
excellent zero-shot OCR performance in the wild, without explicitly training
on the OCR domain-specific data. Understanding the efficacy of LMM in handling
text-related visual tasks is pivotal, given their potential to infer context
from multiple data sources, such as text and images. Despite this advantage,
these models may face challenges when dealing with complex relationships
between different data types due to their general training on web-scale data.
Recognizing these limitations could guide improvements in multimodal
methodologies and inspire the creation of more robust models that can handle
text-related tasks more efficiently. Additionally, this knowledge can open up
novel applications in areas such as digital marketing or social media
analysis, where understanding the interplay between textual and visual content
in the images is crucial.

To this end, we undertake a comprehensive study on 14 LMMs by evaluating their
OCR ability on five representative tasks: text recognition, Scene Text-Centric
VQA, Document-Oriented VQA, key information extraction and handwritten
mathematical expression recognition. Our findings indicate that even state-of-
the-art large multimodal models, such as [16] and GPT4V [17], encounter
difficulties in recognizing handwritten mathematical expressions. LMMs
continue to struggle with blurry text images, handwritten text, multilingual
text. Moreover, they heavily rely on semantic understanding to recognize
words, often favoring common words over random letter sequences. Consequently,
these findings highlight even the most powerful LMM leaves a gap to match the
domain-specific methods in various traditional text tasks, and even worse for
more complicated tasks. However, it highlights the vast opportunity to improve
the OCR ability of LMM by adapting them to the OCR domains.

##  2 Related Work

###  2.1 Large Multimodal Models

The remarkable success of Large Language Models (LLMs) has paved the way for
the development of large multimodal models, which combine pretrained visual
models with LLMs to enable their visual capabilities. BLIP2 [18] introduces
the Querying Transformer (Q-Former) as a means to bridge the gap between
vision and language models. Flamingo [13] and OpenFlamingo [19] enhance a
frozen pretrained LLM by incorporating novel gated cross-attention-dense
layers, enabling conditioning on visual inputs. LLaVA [15] pioneers the use of
GPT-4 to generate multimodal instruction-following data. Other works, such as
[20, 21], also focus on aligning the vision module and LLM for improved
multimodal understanding. Additionally, [22, 23] emphasize modality
collaboration for image and text. LLaVAR [24] collects training data with rich
text and uses a higher-resolution CLIP as the visual encoder to enhance
LLaVA’s OCR ability. BLIVA [25] combines instruction-aware and global visual
features to capture richer image information. MiniGPT4V2 [26] uses unique
identifiers for different tasks when training the model to better distinguish
each task instruction effortlessly. UniDoc [27] performs unified multimodal
instruction tuning on large-scale instruction following datasets and leverages
the beneficial interactions among tasks to enhance the performance of each
individual task. Docpedia [28] directly processes visual input in the
frequency domain rather than the pixel space. Monkey [29] enhanced the LMM’s
ability to perceive details at a low cost by the generated detailed caption
and a high-resolution model architecture.

###  2.2 Benchmarks for Large Multimodal Models

With the ongoing advancements in Large Multimodal Models, the question of how
to effectively evaluate their performance has emerged. [15, 30, 22] introduces
GPT-4 or human evaluation to assess the output of LMMs. MMBench [31] evalute
LMM using multiple-choice questions across various dimensions of ability. MME
[32] measures both perception and cognition abilities on True or False
question. However, their testing on OCR data is limited. Additionally, the
True/False question or multiple-choice question cannot accurately assess the
OCR ability to recognize words in an image. In this paper, we conducted an
extensive study on various LMMs for five prominent text-related tasks. To
facilitate the evaluation of LMMs’ OCR ability, we also present OCRBench, a
collection of 1000 manually filtered and corrected question-answer pairs on
five representative text-related tasks.

Figure 1: Visualization results of the five tasks. ‘[Q]’ represents question,
‘[G]’ represents ground truth, ‘[P]’ represents prediction generated by LMM.

##  3 Experiments

###  3.1 Evaluation Metric and Evaluation Dataset

The responses generated by LMM often include many explanatory terms, so the
exact matching approach or Average Normalized Levenshtein Similarity(ANLS)
[33] used in the original dataset are not suitable for evaluating LMM in zero-
shot scenarios. We have defined a unified and simple evaluation criterion for
all datasets, which is to determine whether the ground truth (GT) is present
in the output of the LMM. To reduce false positives, we filter out questions
that have answers containing fewer than 4 symbols from all datasets.
Additionally, we choose 3000 question instances from some large datasets for
testing purposes.

Text Recognition: We evaluate LMM using widely-adopted OCR text recognition
datasets, including (1) Regular Text Recognition: IIIT5K [34], SVT [35], IC13
[36]; (2)Irregular Text Recognition: IC15 [37], SVTP [38], CT80 [39],
COCOText(COCO) [40], SCUT-CTW1500 (CTW) [41], Total-Text (TT) [42]; (3)
Occlusion Scene Text [43], encompassing weakly occluded scene text (WOST) and
heavily occluded scene text (HOST); (4) Artistic Text Recognition: WordArt
[44]; (5) Handwirtten Text Recognition: IAM [45]; (6) Chinese Text
Recognition: ReCTS [46]; (7) Handwritten Digit String Recognition: ORAND-
CAR-2014(CAR-A) [47]; (8) Non-Semantic Text(NST) and Semantic Text(ST): LMMs
primarily rely on semantic understanding to recognize words. In the
experiments, we find that the LMMs have poor recognition performance on
character combinations that lacked semantics. To confirm this, we create two
datasets: Semantic Text (ST) and Non-Semantic Text (NST) using the IIIT5k
dictionary. The ST dataset consists of 3000 images with words from the IIIT5K
dictionary, while the NST dataset contains the same words but with shuffled
characters without semantics. For english words, we employ the consistent
prompt "what is written in the image?". For Chinese words in ReCTS dataset, we
adapt the prompt to "What are the Chinese characters in the image?". For
handwritten digit string, we utilize the prompt "what is the number in the
image?".

Scene Text-Centric VQA: We test LMMs on STVQA [48], TextVQA [49], OCRVQA [50]
and ESTVQA [51]. Scene Text Visual Question Answering (STVQA) consists of
31,000+ questions across 23,000+ images collected from various public
datasets. TextVQA dataset comprises 45,000+ questions on 28,000+ images
sampled from specific OpenImages dataset categories expected to contain text.
OCRVQA features over 1 million question-answer pairs spanning 207,000+ book
cover images. ESTVQA contains 20757 images along with 15056 English questions
and 13006 Chinese questions. We have divided the ESTVQA dataset into
ESTVQA(CN) and ESTVQA(EN), which specifically include questions and answers in
Chinese or English respectively.

Document-Oriented VQA: We assess these LMMs on DocVQA[52], InfographicVQA [53]
and ChartQA [54]. DocVQA is a large-scale dataset with 12,767 document images
of diverse types and content, and over 50,000 questions and answers.
InfographicVQA is a diverse collection of infographics that includes 5,485
images and a total of 30,035 questions. ChartQA includes a total of 9,608
human-written questions covering 4,804 charts, as well as 23,111 questions
generated from human-written chart summaries on 17,141 charts.

Key Information Extraction: We conduct experiments on SROIE [55], FUNSD [56]
and POIE [57]. SROIE contains 1000 complete scanned receipt images for OCR and
key information extraction competitions. In this competition, it is required
to extract the company, date, address, and total expenditure information based
on the receipts. FUNSD dataset consists of 199 real, fully annotated, scanned
forms that may contain noise. POIE consists of camera images from Nutrition
Facts label of products in English and 3,000 images with 111,155 text
instances are collected. KIE dataset requires to extract key-value pairs in
the image. To enable LMMs to accurately extract the correct value for a given
key in the KIE dataset, we employ manual prompt design. For SROIE dataset, we
utilize the following prompts to assist LMMs in generating the respective
values for "company", "date", "address", and "total": "what is the name of the
company that issued this receipt?", "when was this receipt issued?", "where
was this receipt issued?", and "what is the total amount of this receipt?".
Additionally, to retrieve the corresponding value for a given key in FUNSD and
POIE, we utilize the prompt "What is the value for ‘{key}’?"

Handwritten Mathematical Expression Recognition(HMER): We evaluate on HME100K
[58], which consists of 74,502 images for training and 24,607 images for
testing, with 245 symbol classes. During evaluation, we use the prompt "Please
write out the expression of the formula in the image using LaTeX format.".

Method | Regular | Irregular | Occluded | Artistic | Handwritten | Chinese | Digits | Semantic
---|---|---|---|---|---|---|---|---
IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | COCO | CTW | TT | HOST | WOST | WordArt | IAM | ReCTS | ORAND | NST | ST
BLIP2-OPT-6.7B | 79.1 | 86.7 | 83.4 | 71.2 | 78.8 | 76.5 | 51.4 | 62.4 | 68.7 | 59.8 | 69.4 | 68.4 | 32.9 | 0 | 1.2 | 13.0 | 82.6
mPLUG-Owl | 81.1 | 84.3 | 85.9 | 67.5 | 73.9 | 81.4 | 52.3 | 69.2 | 74.4 | 49.7 | 62.7 | 72.1 | 34.8 | 0 | 13.5 | 44.7 | 92.3
InstructBLIP | 86.3 | 92.0 | 86.8 | 80.9 | 85.6 | 86.3 | 62.6 | 70.8 | 77.9 | 67.8 | 78.8 | 73.7 | 42.6 | 0 | 18.4 | 25.3 | 89.7
LLaVAR | 84.0 | 87.6 | 87.7 | 79.4 | 84.0 | 84.5 | 61.9 | 69.5 | 75.6 | 61.1 | 71.9 | 67.1 | 49.4 | 0 | 9.8 | 36.2 | 86.5
BLIVA | 86.5 | 90.6 | 87.3 | 80.9 | 87.7 | 86.7 | 64.8 | 71.2 | 78.1 | 67.7 | 77.6 | 73.7 | 45.1 | 0 | 13.8 | 20.4 | 89.4
mPLUG-Owl2 | 80.9 | 69.6 | 79.8 | 53.9 | 53.5 | 74.8 | 52 | 59.1 | 60.9 | 32.5 | 50.6 | 60.6 | 23.8 | 0 | 9.9 | 48.2 | 93.9
LLaVA1.5-7B | 84.2 | 85.7 | 86.4 | 71.9 | 79.8 | 82.7 | 55.6 | 66.8 | 73.2 | 61.4 | 70.6 | 68.7 | 55.4 | 0 | 10.4 | 15.2 | 85.3
UniDoc | 91.9 | 89.2 | 90.9 | 78 | 80.3 | 88.2 | 64.1 | 75.3 | 78.2 | 52.4 | 68.5 | - | - | - | - | - | -
Monkey | 83.7 | 75.1 | 85.4 | 53.4 | 58.4 | 73.9 | 43.5 | 64.5 | 64.6 | 43.3 | 54.9 | 67.7 | 30.3 | 13.1 | 29.1 | 56.5 | 95.5
Supervised-SOTA | 96.6 | 93.0 | 96.7 | 85.7 | 89.3 | 89.9 | 64.4 | 78.6 | 80.1 | 73.1 | 81.6 | 72.5 | 91.2 | 94.8 | 95.5 | 95.4 | 100.0

Table 1: Text recognition results. Bold black digits indicate the best result,
while blue signifies the second best.

Method | Scene Text-Centric VQA | Document-Oriented VQA. | KIE | HMER
---|---|---|---|---
STVQA | TextVQA | OCRVQA | ESTVQA(EN) | ESTVQA(CN) | DocVQA | InfoVQA | ChartQA | FUNSD | SROIE | POIE | HME100K
BLIP2-OPT-6.7B | 20.9 | 23.5 | 9.7 | 40.7 | 0 | 3.2 | 11.3 | 3.4 | 0.2 | 0.1 | 0.3 | 0
mPLUG-Owl | 30.5 | 34 | 21.1 | 52.7 | 0 | 7.4 | 20 | 7.9 | 0.5 | 1.7 | 2.5 | 0.1
InstructBLIP | 27.4 | 29.1 | 41.3 | 48.6 | 0.1 | 4.5 | 16.4 | 5.3 | 0.2 | 0.6 | 1 | 0
LLaVAR | 39.2 | 41.8 | 24 | 58.2 | 0 | 12.3 | 16.5 | 12.2 | 0.5 | 5.2 | 5.9 | 0
BLIVA | 32.1 | 33.3 | 50.7 | 51.2 | 0.2 | 5.8 | 23.6 | 8.7 | 0.2 | 0.7 | 2.1 | 0.1
mPLUG-Owl2 | 49.8 | 53.9 | 58.7 | 68.6 | 4.9 | 17.9 | 18.9 | 19.4 | 1.4 | 3.2 | 9.9 | 0
LLaVA1.5-7B | 38.1 | 38.7 | 58.1 | 52.3 | 0 | 8.5 | 14.7 | 9.3 | 0.2 | 1.7 | 2.5 | 0
UniDoc | 35.2 | 46.2 | 36.8 | - | - | 7.7 | 14.7 | 10.9 | 1 | 2.9 | 5.1 | 0.4
DocPedia | 45.5 | 60.2 | 57.2 | - | - | 47.1 | 15.2 | 46.9 | 29.9 | 21.4 | 39.9 | -
Monkey | 54.7 | 64.3 | 64.4 | 71 | 42.6 | 50.1 | 25.8 | 54.0 | 24.1 | 41.9 | 19.9 | 0.2
Supervised-SOTA | 69.6 | 73.7 | 68.1 | 43.3 | 43.3 | 90.16 | 36.8 | 70.5 | 93.1 | 98.7 | 79.5 | 64.3

Table 2: Results of Scene Text-Centric VQA, Document-Oriented VQA KIE and
HMER. Bold black digits indicate the best result, while blue signifies the
second best. Since the Supervised-SOTA on ESTVQA dataset does not provide
separate results on Chinese and English data, the average performance 42.3 is
used as a reference.

###  3.2 Results

The results of text recognition are shown in Tab. 1. LMMs achieve comparable
performance to state-of-the-art supervised models in recognizing regular text,
irregular text, occluded text, and artistic text. Even in the WordArt dataset,
which primarily features challenging artistic text, InstructBLIP2 and BLIVA
surpass the performance of supervised state-of-the-art model. However, LMMs
perform poorly in recognizing handwritten text, Chinese text, handwritten
strings, and non-semantic text. In tasks such as Scene Text-Centric VQA,
Document-Oriented VQA, and KIE, LMMs with smaller input resolutions
consistently yield poorer results compared to those with larger input sizes.
This is because texts often contain intricate structures and can vary in size,
requiring the LMMs to capture fine details. The results are shown in Tab. 2
LMMs struggle to complete the complex task of recognizing handwritten
mathematical expressions. Through extensively analyzing the results, we
qualitatively summarize the limitations of LMMs in text tasks:

  * •

Semantic-Reliance. LMMs primarily rely on semantic understanding to recognize
words. In our experiments, we observed that LMMs exhibit poor recognition
performance when it comes to character combinations that lack semantic
meaning. Specifically, when we altered the order of letters in each word, the
accuracy of LMMs on the NST dataset decreased by an average of 57.0% compared
to the ST dataset, while the SOTA method for scene text recognition only drops
by around 4.6%. We believe this is because the SOTA method for scene text
recognition directly recognizes each character, and semantic information is
just used to assist the recognition process, while LMMs primarily rely on
semantic understanding to recognize words. This finding is further supported
by the low accuracy observed on the ORAND dataset. As shown in Fig. 1, LMM
successfully identified the word "Message," but incorrectly recognized
"egaesMs," which is a rearranged version of the word "Message."

  * •

Handwritten Text. LMMs may encounter difficulties in accurately recognizing
handwritten text due to various reasons, including the resemblances in shapes
between handwritten letters and numbers. Handwritten text often appears
incomplete or blurry due to factors like fast writing speed, irregular
handwriting, or low-quality paper. On average, LMMs perform 51.9% worse than
supervised state-of-the-art model in this task.

  * •

Multilingual Text. The notable performance gap observed in LMMs on ReCTS,
ESTVQA(En), and ESTVQA(Ch) highlights their limited proficiency in the Chinese
language, as indicated in Tab. 1 and Tab. 2. Among the LMMs, accurately
recognizing Chinese words or responding to Chinese questions proves to be a
challenge. Monkey outperforms other LMMs in Chinese scenarios due to its LLM
and visual encoder being trained on a substantial amount of Chinese data.

  * •

Fine-grain Perception. Currently, most LMMs have their input images limited to
a resolution of 224 x 224, consistent with the input size of the visual
encoder used in their architecture. However, high-resolution input images can
capture more details from the image. Due to the limited input resolution of
LMMs such as BLIP2, their capacity to extract fine-grained information in
tasks like Scene Text-Centric VQA, Document-Oriented VQA, and KIE is
constrained. However, LMMs such as Monkey, which can support a resolution of
1344×896, exhibit improved performance in these specific tasks.

  * •

HMER. LMMs struggle to recognize handwritten mathematical expressions due to
the presence of messy handwritten characters, complex spatial structures, and
the indirect LaTeX representation. The lack of training data for the
handwritten mathematical expression recognition task is also a crucial factor.

Method | Recog. | $VQA^{S}$ | $VQA^{D}$ | KIE | HMER | Final Score
---|---|---|---|---|---|---
Gemini | 215 | 174 | 128 | 134 | 8 | 659
GPT4V | 167 | 163 | 146 | 160 | 9 | 645
Monkey | 174 | 161 | 91 | 88 | 0 | 514
mPLUG-Owl2 | 153 | 153 | 41 | 19 | 0 | 366
LLaVAR | 186 | 122 | 25 | 13 | 0 | 346
LLaVA1.5-13B | 176 | 129 | 19 | 7 | 0 | 331
LLaVA1.5-7B | 160 | 117 | 15 | 5 | 0 | 297
mPLUG-Owl | 172 | 104 | 18 | 3 | 0 | 297
BLIVA | 165 | 103 | 22 | 1 | 0 | 291
InstructBLIP | 168 | 93 | 14 | 1 | 0 | 276
BLIP2-OPT-6.7B | 154 | 71 | 10 | 0 | 0 | 235
MiniGPT4V2 | 124 | 29 | 4 | 0 | 0 | 157

Table 3: Results of LMMs on OCRBench. Recog. represents text recognition,
$VQA^{S}$ represents Scene Text-Centric VQA, $VQA^{D}$ represents Document-
Oriented VQA. Bold black digits indicate the best result.

###  3.3 OCRBench

Evaluating multiple datasets can be time-consuming, and the inaccurate
annotations in some datasets make accuracy-based evaluation less precise.
Realizing these limitations, we have developed OCRBench to facilitate the
accurate and convenient evaluation of LMMs’ OCR capabilities. OCRBench
consists of five components: text recognition, Scene Text-Centric VQA,
Document-Oriented VQA, KIE, and HMER. It includes 1000 question-answer pairs,
and for the KIE task, we added the prompt "Answer this question using the text
in the image directly." to restrict the model’s response format. The specific
composition of OCRBench is shown in appendix. To ensure a more accurate
evaluation, we manually verified and corrected incorrect answers for the 1000
question-answer pairs, providing alternative correct answer candidates. The
final results are presented in the Tab. 3, with Gemini achieving the highest
score and GPT4V securing the second position. It is important to note that due
to rigorous safety reviews by OPEN AI, GPT4V refused to provide results for 84
images in OCRBench. Monkey demonstrated OCR capabilities that were only
inferior to GPT4V and Gemini. From the test results, we can observe that even
state-of-the-art models like GPT4V and Gemini struggle with the HMER task.
Furthermore, they also face challenges in handling unclear images, handwritten
text, Non-Semantic text and adhering to task instructions. As shown in Fig. 2
(g), even when explicitly requested to answer using the text found in the
image, Gemini consistently interprets "02/02/2018" as "2 February 2018".

Figure 2: Some erroneous results from Gemini and GPT-4V. ‘[Q]’ represents
question, ‘[G]’ represents ground truth. (a) showcases results on blurry
image, (b) presents results on handwritten text, (c) and (d) showcase results
on Non-Semantic text, (e) and (f) showcase results on HMER, (g) displays
instances where the task instructions are not adhered to.

##  4 Discussions

Why LMMs work for OCR? While we offer some analysis based on the results, the
question as to why these multimodal models can deliver acceptable performance
on OCR tasks remains challenging to conclusively explain. One plausible
explanation for the success of multimodal models in Optical Character
Recognition (OCR) tasks lies in the training of multimodal models (similar to
CLIP), which we believe includes some OCR data. However, unsupervised text-
image pairs in the training data cannot compete with fully supervised data.

From a network perspective, the pre-trained models of the visual encoder and
Large Language Model (LLM) already demonstrate a strong grasp of their
respective domain data, each operating within their structured feature space.
These LMMs bridge visual and language information using components such as a
linear projection layer, that functions as a step of visual tokenization. This
process aligns visual tokens within the pre-trained language model’s word
embedding space, resulting in visual embeddings that closely mirror their
corresponding word embeddings. This alignment facilitates text recognition,
and the LLM subsequently presents this OCR data to users in a generative
manner.

Future research could benefit from an ablation analysis to understand the
impact of the volume of multimodal training data on OCR performance.

Future Directions: Despite demonstrating promise, current LMMs struggle with
complex tasks and cannot rival domain-specific methods in traditional text
tasks. To improve, these models should focus on refining their capabilities in
detecting fine-grained features in images, enhancing their understanding of
individual character shapes, and building multilingual data for training.

Recent work such as the ICL-D3IE [59], introduces an in-context learning
framework leveraging large language models like GPT-3 and ChatGPT for document
information extraction. This approach effectively navigates the modality and
task gaps in this field, extracting challenging and distinct segments from
difficult training documents, designing relationship demonstrations for
enhanced positional understanding, and incorporating formatting demonstrations
for answer extraction. Despite its notable success across various datasets and
settings, ICL-D3IE encounters challenges in the in-domain setting on CORD
[60]. This result points to the potential of large language models in managing
tasks involving visually intricate documents and encourages the development of
novel, minimally supervised OCR techniques.

Moreover, a study by Li et al. [61], evaluates ChatGPT across seven detailed
information extraction tasks. This evaluation reveals that ChatGPT performs
well in OpenIE settings, generates high-quality responses, shows a tendency
towards overconfidence, and maintains strong fidelity to original texts.
However, this study focused exclusively on pure text, without venturing into
visually rich document texts within the OCR domain. Future research should
therefore explore ChatGPT’s performance in these visually rich contexts.

The application of LMMs in specialized domains is also a crucial field, and
recent developments have shown their vast potential. One such model, developed
by Google, is Med-PaLM2 [62], which is fine-tuned on medical knowledge from
PaLM2. It is the first language model to perform an expert level on US Medical
Licensing Examination (USMLE) style questions and can analyze patients’
conditions through medical images, including plain films and mammograms.
Google claims that it has approached the performance of clinician experts.
Additionally, LLaVA-Med [63] attempts to extend multimodal instruction-tuning
to the biomedical domain and demonstrates excellent chat abilities with domain
knowledge. These studies highlight the potential of LMMs in vertical domain
tasks. Further exploration of their applications in other domains, such as
gaming and education, is also warranted.

Ultimately, these developments and future research directions could
potentially pave the way for multimodal models that can more efficiently
handle complex tasks like OCR, expanding the application range of LMM.

##  5 Conclusion

This paper has presented an extensive study on performance of LMM on OCR
tasks, including text recognition, Scene Text-Centric, Document-Oriented VQA,
KIE and HMER. Our quantitative assessment reveals that LMM can achieve
promising results, especially in text recognition, even attaining SOTA
performance in some datasets. However, significant gaps persist compared to
domain-specific supervised methods, suggesting that specialized techniques
tailored to each task are still essential, as the latter uses much less
computational resources and data. However, the emergence of LMM does present
promising directions for advancements in OCR techniques. It stimulates an
intuitive query for potential improvements: after training on a larger amount
of text-related data, can LMM surpass supervised methods? We hope this work
provides readers with valuable insights into the application of LMM for future
exploration in OCR technology development. Importantly, the baseline results
presented in this paper can serve as a foundation for the development and
evaluation of novel ideas aimed at improving text-related multimodal
techniques.

##  6 Acknowledgements

Thanks for the help from Hongliang Li, Wenwen Yu, Yang Liu, Mingxin Huang,
Dezhi Peng, Mingyu Liu and Mingrui Chen.

## References

  * [1] OpenAI.  ChatGPT.  https://openai.com/blog/chatgpt/, 2023.
  * [2] OpenAI.  Gpt-4 technical report, 2023.
  * [3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [4] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [5] Vicuna.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  https://vicuna.lmsys.org/, 2023.
  * [6] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with GPT-4.  arXiv preprint arXiv:2304.03277, 2023.
  * [7] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al.  Vision-language pre-training: Basics, recent advances, and future trends.  Foundations and Trends® in Computer Graphics and Vision, 2022.
  * [8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [9] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [10] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  arXiv preprint arXiv:2102.05918, 2021.
  * [11] Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao.  ELEVATER: A benchmark and toolkit for evaluating language-augmented visual models.  In NeurIPS Track on Datasets and Benchmarks, 2022.
  * [12] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  PaLM-E: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * [13] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In Advances in Neural Information Processing Systems, volume 35, pages 23716–23736, 2022.
  * [14] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [15] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [16] Google Gemini Team.  Gemini: A family of highly capable multimodal models.  2023\.
  * [17] OpenAI.  Gpt-4v(ision) system card.  2023\.
  * [18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
  * [19] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo, March 2023.
  * [20] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023.
  * [21] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  MiniGPT-4: Enhancing vision-language understanding with advanced large language models, 2023.
  * [22] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mPLUG-Owl: Modularization empowers large language models with multimodality, 2023.
  * [23] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.  mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023.
  * [24] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.  Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023.
  * [25] Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu.  Bliva: A simple multimodal llm for better handling of text-rich visual questions, 2023.
  * [26] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023.
  * [27] Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang.  Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding, 2023.
  * [28] Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang.  Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding, 2023.
  * [29] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai.  Monkey: Image resolution and text label are important things for large multi-modal models, 2023.
  * [30] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning, 2023.
  * [31] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin.  Mmbench: Is your multi-modal model an all-around player?, 2023.
  * [32] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji.  Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023.
  * [33] Ali Furkan Biten, Rubèn Pérez Tito, Andrés Mafla, Lluís Gómez, Marçal Rusiñol, Ernest Valveny, C. V. Jawahar, and Dimosthenis Karatzas.  Scene text visual question answering.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4290–4300, 2019.
  * [34] Anand Mishra, Karteek Alahari, and CV Jawahar.  Top-down and bottom-up cues for scene text recognition.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2687–2694. IEEE, 2012.
  * [35] Cunzhao Shi, Chunheng Wang, Baihua Xiao, Song Gao, and Jinlong Hu.  End-to-end scene text recognition using tree-structured models.  Pattern Recognition, 47:2853–2866, 2014.
  * [36] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, M. Iwamura, Lluís Gómez i Bigorda, Sergi Robles Mestre, Joan Mas Romeu, David Fernández Mota, Jon Almazán, and Lluís-Pere de las Heras.  ICDAR 2013 robust reading competition.  In 2013 12th International Conference on Document Analysis and Recognition, pages 1484–1493, 2013.
  * [37] Dimosthenis Karatzas, Lluís Gómez i Bigorda, Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, M. Iwamura, Jiri Matas, Lukás Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny.  ICDAR 2015 competition on robust reading.  In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 1156–1160, 2015.
  * [38] Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan.  Recognizing text with perspective distortion in natural scenes.  In 2013 IEEE International Conference on Computer Vision, pages 569–576, 2013.
  * [39] Anhar Risnumawan, Palaiahnakote Shivakumara, Chee Seng Chan, and Chew Lim Tan.  A robust arbitrary text detection system for natural scene images.  Expert Syst. Appl., 41:8027–8048, 2014.
  * [40] Andreas Veit, Tomas Matera, Lukás Neumann, Jiri Matas, and Serge J. Belongie.  COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images.  ArXiv, abs/1601.07140, 2016.
  * [41] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  Pattern Recognit., 90:337–345, 2019.
  * [42] Chee-Kheng Chng and Chee Seng Chan.  Total-Text: A comprehensive dataset for scene text detection and recognition.  In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 935–942, 2017.
  * [43] Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang.  From two to one: A new scene text recognizer with visual language modeling network.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14174–14183, 2021.
  * [44] Xudong Xie, Ling Fu, Zhifei Zhang, Zhaowen Wang, and Xiang Bai.  Toward understanding WordArt: Corner-guided transformer for scene text recognition.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pages 303–321. Springer, 2022.
  * [45] U-V Marti and Horst Bunke.  The iam-database: an english sentence database for offline handwriting recognition.  International Journal on Document Analysis and Recognition, 5:39–46, 2002.
  * [46] Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al.  Icdar 2019 robust reading challenge on reading chinese text on signboard.  In 2019 international conference on document analysis and recognition (ICDAR), pages 1577–1581. IEEE, 2019.
  * [47] Markus Diem, Stefan Fiel, Florian Kleber, Robert Sablatnig, Jose M. Saavedra, David Contreras, Juan Manuel Barrios, and Luiz S. Oliveira.  Icfhr 2014 competition on handwritten digit string recognition in challenging datasets (hdsrc 2014).  In 2014 14th International Conference on Frontiers in Handwriting Recognition, pages 779–784, 2014.
  * [48] Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluis Gomez, Marçal Rusiñol, Minesh Mathew, C.V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas.  Icdar 2019 competition on scene text visual question answering.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1563–1570, 2019.
  * [49] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards VQA models that can read, 2019.
  * [50] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  OCR-VQA: visual question answering by reading text in images.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947–952, 2019.
  * [51] Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang.  On the general value of evidence, and bilingual scene-text visual question answering.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10123–10132, 2020.
  * [52] Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar.  DocVQA: A dataset for vqa on document images, 2021.
  * [53] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar.  Infographicvqa.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022.
  * [54] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  arXiv preprint arXiv:2203.10244, 2022.
  * [55] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar.  ICDAR2019 competition on scanned receipt OCR and information extraction.  In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, sep 2019.
  * [56] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran.  FUNSD: A dataset for form understanding in noisy scanned documents, 2019.
  * [57] Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, Yu Zhou, and Xiang Bai.  Visual information extraction in the wild: Practical dataset and end-to-end solution.  arXiv preprint arXiv:2305.07498, 2023.
  * [58] Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai.  Syntax-aware network for handwritten mathematical expression recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4553–4562, 2022.
  * [59] Jiabang He, Lei Wang, Yingpeng Hu, Ning Liu, Hui juan Liu, Xingdong Xu, and Hengtao Shen.  ICL-D3IE: In-context learning with diverse demonstrations updating for document information extraction.  ArXiv, abs/2303.05063, 2023.
  * [60] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee.  Cord: A consolidated receipt dataset for post-ocr parsing.  In Document Intelligence Workshop at Neural Information Processing Systems, 2019.
  * [61] Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang.  Evaluating ChatGPT’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  ArXiv, abs/2304.11633, 2023.
  * [62] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan.  Towards expert-level medical question answering with large language models, 2023.
  * [63] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day, 2023.
  * [64] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  OPT: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * [65] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [66] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [67] Darwin Bautista and Rowel Atienza.  Scene text recognition with permuted autoregressive sequence models.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pages 178–196. Springer, 2022.
  * [68] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2315–2324, 2016.
  * [69] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data and artificial neural networks for natural scene text recognition.  arXiv preprint arXiv:1406.2227, 2014.
  * [70] Dmitrijs Kass and Ekta Vats.  Attentionhtr: Handwritten text recognition based on attention encoder-decoder networks, 2022.
  * [71] Ming-Ming Yu, Heng Zhang, Fei Yin, and Cheng-Lin Liu.  An efficient prototype-based model for handwritten text recognition with multi-loss fusion.  In Frontiers in Handwriting Recognition: 18th International Conference, ICFHR 2022, Hyderabad, India, December 4–7, 2022, Proceedings, pages 404–418. Springer, 2022.
  * [72] Yixuan Qiao, Hao Chen, Jun Wang, Yihao Chen, Xianbin Ye, Ziliang Li, Xianbiao Qi, Peng Gao, and Guotong Xie.  Winner team mia at textvqa challenge 2021: Vision-and-language representation learning with pre-trained sequence-to-sequence model, 2021.
  * [73] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 2020.
  * [74] Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang, Shi Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-layout: Layout knowledge enhanced pre-training for visually-rich document understanding.  In Conference on Empirical Methods in Natural Language Processing, 2022.
  * [75] Zeqin Fang, Lin Li, Zhongwei Xie, and Jingling Yuan.  Cross-modal attention networks with modality disentanglement for scene-text vqa.  In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2022.
  * [76] Kriti Aggarwal, Aditi Khandelwal, Kumar Tanmay, Owais Mohammed Khan, Qiang Liu, Monojit Choudhury, Subhojit Som, Vishrav Chaudhary, and Saurabh Tiwary.  Dublin–document understanding by language-image network.  arXiv preprint arXiv:2305.14218, 2023.
  * [77] Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun.  Deplot: One-shot visual language reasoning by plot-to-table translation.  arXiv preprint arXiv:2212.10505, 2022.
  * [78] Yulin Li, Yuxi Qian, Yuchen Yu, Xiameng Qin, Chengquan Zhang, Yang Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding.  Structext: Structured text understanding with multi-modal transformers.  In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
  * [79] Bohan Li, Ye Yuan, Dingkang Liang, Xiao Liu, Zhilong Ji, Jinfeng Bai, Wenyu Liu, and Xiang Bai.  When counting meets hmer: Counting-aware network for handwritten mathematical expression recognition.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pages 197–214. Springer, 2022.

##  Appendix A Summary of the Evaluation Benchmarks.

Tab. 4 presents the detailed composition of OCRBench. The Text Recognition
task consists of a total of 300 images, including Regular Text Recognition,
Irregular Text Recognition, Artistic Text Recognition, Handwriting
Recognition, Digit String Recognition, and Non-Semantic Text Recognition, with
50 images each. The Scene Text-centric VQA task includes 200 questions, with
50 questions each from STVQA, TextVQA, ESTVQA(EN), and OCRVQA datasets. The
Doc-oriented VQA task comprises 200 questions, with 50 questions each from
DocVQA, ChartQA(Aug.), ChartQA(Hum.), and InfoVQA datasets. The Key
Information Extraction task consists of 200 questions, with 67 questions from
SROIE, 66 questions from FUNSD, and 67 questions from POIE. The Handwritten
mathematical expression recognition task includes 100 images from the HME100k
dataset. OCRBench encompasses a total of 1000 questions across these five
tasks, and all answers have been manually filtered and corrected.

Task | Dataset | Smaples
---|---|---
Text Recognition | Regular Text Recognition | 50
Irregular Text Recognition | 50
Artistic Text Recognition | 50
Handwriting Recognition | 50
Digit String Recognition | 50
Non-Semantic Text Recognition | 50
Scene Text-centric VQA | STVQA | 50
TextVQA | 50
ESTVQA(EN) | 50
OCRVQA | 50
Doc-oriented VQA | DocVQA | 50
ChartQA(Aug.) | 50
ChartQA(Hum.) | 50
InfoVQA | 50
Key Information Extraction | SROIE | 67
FUNSD | 66
POIE | 67
HMER | HME100k | 100
All | - | 1000
Table 4: Details of the OCRBench.

##  Appendix B Summary of the Models.

Tab. 5 displays the details of the testing models, with most LMMs having their
resolution limited to an input size of 224 or 336 due to the visual module. As
a result, they perform poorly on Doc-oriented VQA and KIE tasks. Conversely,
models with higher resolutions exhibit better performance on these tasks.

Models | Language Model | Input Resolution
---|---|---
BLIP2-OPT-6.7B | OPT-6.7B [64] | 224
mPLUG-Owl | LLaMA-2 7B [65] | 224
InstructBLIP | Vicuna-7B [5] | 224
LLaVAR | Vicuna-7B [5] | 336
BLIVA | Vicuna-7B [5] | 224
mPLUG-Owl2 | LLaMA-2 7B [65] | 448
LLaVA1.5-7B | Vicuna-7B [5] | 336
LLaVA1.5-13B | Vicuna-13B [5] | 336
MiniGPT4V2 | Llama2-Chat-7B [65] | 224
Monkey | Qwen-7B [66] | 896
Unidoc | Vicuna [5] | 336
DocPedia | Vicuna [5] | 2560
GPT4V | gpt-4-vision-preview
Gemini | gemini-pro-vision
Table 5: Details of the models.

##  Appendix C Supervised SOTA

State-of-the-art (SOTA) in widely-adopted OCR text recognition datasets is
achieved by PARSeq [67], which utilizes Permutation Language modeling for
enhanced contextual information and unifies context-free non-AR and context-
aware AR inference. We train Parseq on the ST [68] and MJ [69] synthetic
datasets, and test it directly on real datasets. The ReCTS dataset is not
commonly used, therefore, we chose TPS-ResNet, which has the highest ranking
on the ICDAR ranking table along with a published article, as the SOTA. The
model employs Thin-plate-spline (TPS) based Spatial Transformer Network (STN)
to normalize the input text images, followed by a ResNet-based feature
extractor and BiLSTM for text recognition. Although the evaluation metric for
ReCTS is Normalized Edit Distance, we still use it as a reference.
AttentionHTR [70] proposes an attention-based sequence-to-sequence model to
achieve SOTA on IAM. The word error rate of this method on IAM is 8.76, so we
use 91.24 as the corresponding word accuracy. Yu [71] achieves SOTA on ORAND-
CAR-2014 by proposing an efficient text line recognition method based on
prototype learning with feature-level sliding windows for classification.

For Scene Text Visual Question Answering (STVQA) and Optical Character
Recognition Visual Question Answering (OCR-VQA), SOTA is reached by GIT [14].
GIT trains a Generative Image-to-text Transformer which contains one image
encoder and one text decoder under a single language modeling task. For
TextVQA, Mia [72] achieves SOTA by using T5 [73] for TextVQA task. To align
object feature and scene text, Mia is pretrained by masked language
modeling(MLM) and relative position prediction(RPP) task. For DocVQA Dataset,
BAIDU-DI assembles ERNIE-Layout [74] and DocPrompt (a few-shot model using
multi-stage training based on ERNIE-Layout) to achieve SOTA. ESTVQA dataset
has not been explored well, Fang [75] achieves SOTA by using a cross-modal
attention network to guide the expression of visual and textual features
simultaneously while using a transformer decoder to output the results.
However, since Fang [75] does not provide separate results on Chinese and
English data, the average performance is used as a reference. For
InfographicVQA, SOTA is reached by DUBLIN[76]. For ChartQA, Liu[77] achieves
SOTA by designing a modality conversion module DEPLOT.

The SOTA on the FUNSD dataset is achieved by ERNIE-Layout [74], which enhances
layout knowledge by correcting the reading order in pretraining phase. For
SROIE, StrucTexT [78] achieves the SOTA. StrucTexT introduces a segment-token
aligned encoder for transformer and is pretrained by Masked Visual Language
Modeling task and the new Sentence Length Prediction and Paired Boxes
Direction tasks. For POIE, Kuang [57] achieves SOTA by adopting contrastive
learning to effectively establish the connections between the tasks of OCR and
information extraction. For HME100K, Li [79] proposes a Counting-Aware Network
which jointly optimizes HMER and symbol counting tasks to achieve SOTA.