mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
Anwen Hu Haiyang Xu Liang Zhang Jiabo Ye Ming Yan Ji Zhang Qin Jin Fei Huang Jingren Zhou



	Automatic occlusion removal from 3D maps for maritime situational awareness
    
	Felix Sattler1, Borja Carrillo Perez1, Maurice Stephan1, Sarah Barnes1
	1German Aerospace Center (DLR), Institute for the Protection of Maritime Infrastructures,
		Bremerhaven, Germany

    Received Month dd, yyyy; accepted Month dd, yyyy
===============================================================================================================================================================================================





[1]Corresponding author



   

   Anwen Hu1    Haiyang Xu1[1]     Liang Zhang2    Jiabo Ye1     Ming Yan1[1] 
 
    Ji Zhang1    Qin Jin2    Fei Huang1     Jingren Zhou1 

     1Alibaba Group     2Renmin University of China 

    {huanwen.haw, shuofeng.xhy, ym119608}@alibaba-inc.com 
   <https://github.com/X-PLUG/mPLUG-DocOwl>







§ ABSTRACT 
Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, 

we propose a  module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the  under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning.  sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our  achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at <https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2>.






§ INTRODUCTION

Understanding a multi-page document or news video is common in human daily life. To tackle such scenarios, Multimodal Large Language Models (MLLMs) <cit.> should be equipped with the ability to understand multiple images with rich visually-situated text information.
Different from natural images mainly comprising of objects, comprehending document images asks for a more fine-grained perception to recognize all texts. To tackle high-resolution document images, some works <cit.> propose to add an additional high-resolution encoder while more works <cit.> choose to crop a high-resolution image to low-resolution sub-images and let the Large Language Model to understand their relationship. By increasing the cropping number, the latter achieves better performance of OCR-free document understanding but also results in too many visual tokens for only 1 document image, e.g., InternVL 2 <cit.> costs a average of 3k visual tokens on single-page document understanding benchmark DocVQA <cit.>. 
As shown in <ref>(a), such long visual tokens not only result in long inference time but also occupy too much GPU memory, making it difficult to understand a complete document or video and greatly limiting their application scenarios. Inspired by Natual Language Processing work <cit.> which summarizes a textual paragraph/document into fewer tokens and maintains most semantics, we argue that visual tokens of document images can also be further compressed while maintaining both layout and most textual information. 

Existing compressing architecture in MLLMs are hard to balance information retention and token efficiency during document image encoding. As shown in <ref>(a), independently compressing each crop of a document image <cit.> could reduce visual tokens of each sub-image but still results in a long sequence of visual tokens after concatenating all sub-images. Leveraging learnable queries <cit.> or selected tokens <cit.> as compressing guidance could produce an identical length of tokens for any resolution but overlook the overall layout information, as shown in <ref>(b). Layout-aware guidance is important for compressing visual features of document images because texts within a layout region are semantic-coherent and easier to summarize. For example, in a two-column paper, texts belonging to the `Related Work' section are difficult to summarize with texts on the same line but belonging to the `Method' section.

 In this work, as shown in <ref>(c), we propose a layout-aware compressing architecture  based on cross-attention to compress document images into fewer tokens and achieve better performance than existing compressing methods. Considering that a global low-resolution image can well capture the overall layout information, we utilize visual features of a global low-resolution image as the compressing guidance (query). Each visual feature in the global feature map just captures the layout information of partial regions. Therefore, each query attending to all high-resolution features will not only make information compression more difficult but also increase computation complexity. To summarize text information within a layout region, for each query from the global feature map, a group of high-resolution features with identical relative positions in the raw image is collected as compressing objects, sometimes spanning multiple sub-images. Besides, since the vision-to-text (V2T) module of MLLMs could convert visual features into textual feature space, we argue that compressing visual features after the vision-to-text module could better maintain textual semantics in document images. Therefore, based on the architecture of DocOwl 1.5 <cit.>, we propose mPLUG- by placing the  afther its V2T module: H-Reducer. To take full advantage of the compressing method, our model  is trained with a three-stage framework: Single-image Pretraining, Multi-image Continue-Pretraining, and Multi-task Finetuning to support both single-image and multi-image/frame understanding. Our experiments on single-page and multi-page document benchmarks demonstrate the good balance of OCR-free document understanding performance and token efficiency of . We perform sufficient ablation studies to validate the superiority of our  and the benefits of the three-stage training framework for both single-page and multi-page understanding performance.

 Our contributions in this work are three-fold:
 
    
  * We propose a novel compressing architecture, namely , to greatly reduce visual tokens of high-resolution document images. Compared with existing compressing methods, our method achieves better OCR-free single-image document Understanding performance with fewer visual tokens.
     
  *  achieves state-of-the-art performance on Multi-page Document understanding benchmarks with 50% First Token Latency.
    
  * Compared with state-of-the-art MLLMs with similar model size and training data,   achieves comparable performance with 20% visual tokens on 10 single-image document benchmarks.
   

 






§ RELATED WORK




 §.§ OCR-free Visual Document Understanding

Visual Document Understanding aims to comprehend images with rich text information, including scans of document pages <cit.>, infographics <cit.>, charts <cit.>, tables images <cit.>, webpage screenshots <cit.> and natural images with scene texts <cit.>. Recently, many Multimodal Large Language Models have been proposed to perform visual document understanding in an OCR-free manner. mPLUG-DocOwl <cit.> and UReader <cit.> first propose to unify different tasks across 5 types of document images in the seq-to-seq format. To encode rich text information in high-resolution images, UReader <cit.> proposes a Shape-adaptive Cropping Module to cut the raw image into multiple low-resolution sub-images and utilizes an identical low-resolution encoder to encode both sub-images and a global image. Monkey <cit.> proposes to employ a sliding window to partition high-resolution images and a resampler to reduce redundant information of each sub-image. mPLUG-DocOwl1.5 <cit.> increases the basic resolution of the low-resolution encoder and replaces the Visual Abstractor <cit.> with 1 simple convolution layer to better maintain the structure information. DocPedia <cit.> directly processes high-resolution images in the frequency domain. CogAgent <cit.> proposes to utilize a high-resolution encoder to encode high-resolution visual features and a low-resolution encoder to encode low-resolution global features. Series work of InternLM-XComposer <cit.> and InternVL <cit.> further optimize the cropping method or increase the cropping number and greatly improves the OCR-free Document Understanding performance. These works achieve promising performance but suffer from too many visual tokens for a high-resolution image (always 1k tokens for a common A4-sized document page), which hinders the development of OCR-free multi-page document understanding.



 §.§ Visual Feature Compressing

Reducing visual tokens of a single image enables a Multimodal Large Language Model with limited maximum sequence length to leverage more images as contexts to perform complex multimodal tasks, such as video understanding, embodied interaction, or multi-page document understanding. There have been some architectures proposed for compressing visual features of general images with fewer learnable queries, such as the Resampler <cit.>, Abstractor <cit.> and Q-former <cit.>. Randomly initialized Learnable queries can ensemble object information in general images but is hard to summarize rich text information in high-resolution document images. As a compromise solution, TokenPacker <cit.> proposes to compress each sub-image with its downsampled visual features as the query to perform cross-attention. TokenPacker just reduces each sub-image's visual tokens, thus still creates more than 1k visual tokens when processing high-resolution document images. TextMonkey <cit.> first filters valuable visual tokens and then uses them as guidance to aggregate all visual tokens. Due to that valuable visual tokens are selected by measuring the token similarity, visual information of partial regions may not be covered and thus not well compressed during following cross-attention. In this work, our  leverages visual features from the row-resolution global images as the query, the ensembled feature map of sub-images as key and value. This not only produces a fixed number of visual tokens for images of any resolution but also covers all areas during compression. Compared to Mini-Gemini <cit.> which compresses general visual features, there are major two differences with our . Firstly, we make full use of global visual features and sub-image features produced by an identical low-resolution vision encoder and don't need to add an extra high-resolution encoder. Secondly, for better summarizing textual information in document images, our cross-attention is applied based on visual features that have been aligned with textual features of LLM. We argue that directly compressing outputs of the vision encoder will lose more visually situated textual information while comprising features aligned with LLM is like summarizing texts <cit.> and can better maintain textual semantics in document images. Fair comparisons are performed in our experiments to support our hypothesis.




§ MPLUG-




As shown in <ref>,  leverages a Shape-adaptive Cropping Module and a low-resolution vision encoder to encode high-resolution document images. Then, it utilizes a vision-to-text module  to ensemble horizontal visual features and align the dimension of vision features with Large Language Models. Furthermore, a high-resolution compressor is designed to greatly reduce the number of visual features while maintain most visual information. Finally, compressed visual tokens of multiple images/pages are concatenated with text instructions and input to a Large Language Model for multimodal understanding.



 §.§ High Resolution Vision Encoding

Following UReader <cit.> and DocOwl 1.5 <cit.>,  utilizes a parameter-free Shape-adaptive Cropping Module to preprocess high-resolution images. Concretely, it cuts each high-resolution image I into R × C size-fixed sub-images I^s = {I^s_xy}, 1 ≤ x ≤ R, 1 ≤ y ≤ C, where cropping rows R and columns C are flexibly decided based on the raw resolution of I. Besides, to maintain the overall layout information, the raw image is also directly resized to a global image I^g. Both the global image and sub-images are sized H × W. 

After the cropping module, a low-resolution transformer-based vision encoder ViT <cit.> is utilized to independently extract vision features of each sub-image and the global image as follows:

 
    V^g =  ViT(I^g) 
    
        V^s_xy =  ViT(I^s_xy), 1 ≤ x ≤ R, 1 ≤ y ≤ C,

where both V^g and V^s_xy are visual features with the shape of h × w × d, d is the feature dimension and w, h are the width and height of the feature map.

Following DocOwl 1.5, after the ViT, for each sub-image or global image, we apply a vision-to-text module   to ensemble horizontal 4 features by a convolution layer and align the feature dimension with the Large Language Model with a fully connected layer. The calculation of  is represented as follows:
 
    V̂ =  FC( Conv(V)), V ∈{V^g, V^s_xy}, 1 ≤ x ≤ R, 1 ≤ y ≤ C,

where the shape of the visual feature map V̂ is h ×w/4×d̂, d̂ is the dimension of hidden states of the large language model.




 §.§ High Resolution Full-Compressing

Although the  has reduced the visual tokens of each sub-image or global image to 1/4 the length of original visual features, the token length of high-resolution images is still too long to perform multi-page/image joint understanding for Large Language Models. For example, the token 
 length of 1 high-resolution image in DocOwl 1.5 <cit.> is (R × C+1) × h ×w/4, which will be 2,560 when the raw resolution is 1,344 × 1,344. 

In Natural Language Processing, a sentence/paragraph/document of text tokens can be compressed into fewer summary vectors while maintaining most semantics <cit.>. Besides, since visual features have been aligned with the textual feature space of large language models, the visual tokens of document images after the vision-to-text module can also be treated as textual tokens encoding different parts of textual information in the image. Thus, taking into account these two points, in this work, we argue that visually situated textual information of document images can also be further compressed into fewer tokens, especially after the vision-to-text alignment.

Ideally, the compression of visual texts should be based on their layout. Texts from the same layout region (e.g., a title/paragraph region) are more appropriate to be fused into an identical token. After the vision-to-text module , the global visual feature V̂^g mainly encodes the overall text layout information while visual features of sub-images {V̂^s_xy} capture detailed textual information. Besides, due to both the global image and cropped sub-images come from an identical image, there is a clear mapping between the visual tokens of V̂^g and {V̂^s_xy}. As shown in <ref>, each visual token in V̂^g can be aligned with R × C visual tokens in {V̂^s_xy}. Therefore, in this work, with global visual features as query, and the visual features from sub-images as key and value, we propose to utilize cross-attention to ensemble textual semantics and greatly reduce the number of visual tokens of a high-resolution image to the one of a low-resolution global image. 

Concretely, we first re-organize feature maps of cropping images ({V̂^s_xy}, 1 ≤ x ≤ R, 1 ≤ y ≤ C) to a complete feature map V̂^s according to their positions in the raw high-resolution image. Then, for each visual token in the feature map V̂^g of the global image, we collect its corresponding R × C visual tokens from V̂^s as the key and value, the cross-attention layer in this compressor is calculated as follows:


    v̂^g_ij∈V̂^g, 1 ≤ i ≤ h, 1 ≤ j ≤ w/4
    v̂^s_ij = [v̂^s_i^'j^'] ⊂V̂^s, (i-1)R+1 ≤ i^'≤ iR, (j-1)C+1 ≤ j^'≤ jC  
    v̅_ij =  softmax(W^qv̂^g_ijW^kv̂^s_ij/√(d_k))W^vv̂^s_ij + v̂^g_ij


where v̂^g_ij is a visual token from the feature map of the global image, v̂^s_ij are visual tokens from the re-organized feature map of cropping images. v̂^g_ij and v̂^s_ij correspond to the same area in the raw image. W^q,W^k,W^v are learnable projection matrics. 

After high-resolution compressing, the compressed feature map of each image is organized into a sequence V̅=[v̅_1,v̅_2,...,v̅_h ×w/4]  for subsequent understanding of the large language model.



 §.§ Multi-image Modeling with LLM

Through the high-resolution compressing, the number of visual tokens for each high-resolution image is reduced from  (R × C+1) × h ×w/4 to h ×w/4. Such efficient vision encoding allows joint understanding of multiple document images with Large Language Models. To help the LLM better distinguish visual features from different images and understand the ordinal number of images, we add a textual ordinal token  before the visual features of each image, where x is the ordinal number. Overall, the decoding of the decoder for multiple images is as follows:

    Y =  LLM([P_0;V̅_0; P_1;V̅_1, ...,P_n; V̅_n;T])

where [;] means the concatenation operation, n is the number of images, P_x, 1 ≤ x ≤ n is the textual embedding of the ordinal token ,  V̅_x is the visual features for each image, T is the textual instruction and Y is the predicted answer. 



 §.§ Model Training

 is trained with three stages: Single-image Pre-training, Multi-image Continue Pretraining, and Multi-task Finetuning.

At the first stage, to ensure the compressed visual tokens can encode most visual information, especially visually situated texts, we first perform Unifed Structure Learning as DocOwl 1.5 with the dataset DocStruct4M <cit.>, which covers the learning of struct-aware document parsing, table parsing, chart parsing and natural image parsing of a single image. 

After Single-image Pretraining, to empower our model with the ability to correlate multiple images, we further perform Multi-image Continue Pretraing with a struct-aware multi-page document parsing dataset . With partial documents from two datasets of PixParse[<https://huggingface.co/datasets/pixparse/idl-wds>][<https://huggingface.co/datasets/pixparse/pdfa-eng-wds>], we design two symmetrical tasks of multi-image understanding: Multi-page Text Parsing and Multi-page Text Lookup. Given successive page images in a document, the Multi-page Text Parsing instructs the model to parse texts of specified one or two pages, such as . As for the Multi-page Text Lookup task, with texts from 1-2 pages as input, the model is required to predict the concrete ordinal number of images containing these texts, for example, . 
Besides , during this stage, we also randomly chose 0.5M samples from DocStruct4M to avoid the catastrophic forgetting of structure parsing across different types of images.

Finally, we ensemble single-image and multi-image instruction tuning datasets to perform multi-task tuning. We leverage DocDownstream-1.0 <cit.> and DocReason25K <cit.> as single-image datasets. DocDownstream-1.0 is an ensembled dataset comprising of DocVQA <cit.>, InfoVQA <cit.>, DeepForm <cit.>, KLC <cit.>, WTQ <cit.>, TabFact <cit.>, ChartQA <cit.>, TextVQA <cit.>, TextCaps <cit.> and VisualMRC <cit.>. DocReason25K is a question-answering dataset with detailed explanations.
As for multi-image understanding, we ensemble 2 document datasets, MP-DocVQA <cit.> and DUDE <cit.>, and 1 news video dataset NewsVideoQA <cit.> as concise question-answering datasets. MP-DocVQA contains 46k question-answering pairs on 60k page images scanned from 6k industry documents with rich tables, diagrams, pictures, and both handwritten and printed texts. DUDE covers more domains of documents, including medical, legal, technical, financial, etc. It contains 41k question-answering pairs on 5k documents. NewsVideoQA collects news videos with rich visually-situated texts from diverse English news channels around the world, such as BBC, CNN, etc. It contains 8k question-answering pairs framed on 3k videos. Besides, to trigger the ability of detailed explanations with evidence pages, we built MP-DocReason51K based on DocReason25K. Concretely, for each single-image sample from DocReason25K, we construct two multi-image samples with noisy images randomly chosen from the same or different categories. After randomly inserting the evidence image into noisy images, we add an extra evidence description (e.g., ) into the raw detailed explanation to get the target of multi-image samples. Most question-answering samples just focus on 1-2 pages of a document, to further strengthen the ability of a comprehensive understanding of a document, we leverage a small part of annotations from DocGenome <cit.> to construct text sequences in the JSON format, which represents the hierarchical structure of a scientific paper and partial detailed texts.

The detailed statistics of training datasets of  are shown in <ref>.



















§ EXPERIMENTS





 §.§ Implementation Details

The maximum number of crops is set to 12. The resolution of each sub-image or the global image is 504x504. The  comprises of 2 layers of cross attention. Initialized from mPLUG-Owl2 <cit.>, the vision encoder (ViT/L-14 <cit.>), H-Reducer and  are trained during the Sinlge-image Pretraining. Besides, the main parameters of the Large Language Model <cit.> are frozen while a Modality Adaptive Module (MAM) <cit.> used to distinguish visual and textual features in the LLM is tuned. The first stage is trained 12k steps with a batch size of 1,024 and the learning rate set as 1e-4. During the Multi-image Continue-pretraining, the vision encoder is further frozen and the H-Reducer,  and MAM is tuned. The second stage is trained 2.4k steps with a batch size of 1,024 and the learning rate set as 2e-5. At the final Multi-task Finetuning stage, all parameters except the vision encoder are optimized. The batch size, training step, and learning rate at this stage are set as 256, 9k, and 2e-5, respectively.




 §.§ Main Results




We compare  with state-of-the-art Multimodal Large Language Models on 10 single-image document understanding benchmarks, 2 Multi-page document Understanding benchmarks, and 1 text-rich video understanding benchmark. Both question-answering performance and the First Token Latency (seconds) are considered to show the effectiveness of our model. 



  §.§.§ Single-image Document Understanding

For Single-image Document Understanding, we divide baselines into three groups: (a) models without Large Language Models as decoders <cit.>, (b) Multimodal LLMs <cit.> with an average number of visual tokens over 1k for a single document image and (c) Multimodal LLMs <cit.> with an average number of visual tokens less than 1k. As shown in <ref>, although specifically fine-tuned on each downstream dataset, Donut <cit.> or PixsStruct <cit.> are not as good as Multimodal LLMs, showing the potential of MLLMs for generalized OCR-free document understanding. Compared with MLLMs with 1k visual tokens, our  achieves better or comparable performance on 10 benchmarks. Especially, with fewer visual tokens, our model outperforms both TextMonkey <cit.> and TokenPacker <cit.> which also aim to compress visual tokens, showing that our `Full Guidance & Full Candidate' architecture  is better at summarizing and maintaining textual information in high-resolution document images. Besides, compared with state-of-the-art MLLMs with 1k visual tokens,  achieves 80% performance on 7/10 benchmarks while with 20% visual tokens. <ref> visualizes the comparison with SOTA in terms of question-answering performance and the number of visual tokens. 





Furthermore, we compare the First Token Latency (seconds) on the 3 most frequently compared datasets, representing documents, charts, and natural images. As shown in <ref>, the far greater number of visual tokens enable InternVL 2 <cit.> and IXC 2.5 <cit.> to achieve better performance but also result in higher inference time. Considering the model architecture and training data, it's most fair to compare  with DocOwl 1.5. After adding the , with similar training data of OCR learning,  achieves 98% performance of DocOwl 1.5 while reducing 50% First Token Latency with just 20% visual tokens. This validates the effectiveness of our compressor for compressing visually-situated text information on the most common documents, charts, and natural images.



  §.§.§ Multi-page/Video Document Understanding

For Multi-page Document Understanding and Text-rich Video Understanding benchmarks, we choose recently proposed Multimodal LLMs <cit.> with multi-page OCR-free document understanding abilities and can be fed into more than 10 images under a single A100-80G as baselines. As shown in <ref>, with fewer visual tokens for a single image/frame, our model  achieve better question-answering performance and much less First Token Latency, validating the good balance of  between the OCR-free document understanding performance and token efficiency.








 §.§ Ablation Study

We perform sufficient ablation studies to show the effectiveness of the architecture of   and the three-stage training strategy of .


  §.§.§ Compressor Architecture

To validate the effectiveness of our , we compare different compressing architectures with an identical training pipeline of Single-image Pretraing and Single-image Document Understanding Finetuning, keeping both training data and training setting consistent. 

As shown in <ref>, compared with CAbstractor <cit.>, Resampler <cit.> achieves worse document understanding performance (r2 vs r1). This shows that due to no prior knowledge, such as spatial relationship, is leveraged as compressing guidance, utilizing queries learned from scratch to compress rich visually-situated text information is more challenging than simple adaptive mean pooling. Our  outperforms CAbstractor (r3 vs r2), validating that leveraging global visual features as layout-aware guidance can better distinguish the information density of each fine-grained visual feature and therefore maintain more visually-situated text information. 

Instead of placing the compressor after the vision-to-text module H-Reducer, we also try inserting it between the vision encoder and the vision-to-text module. Such a setting results in performance decreases across three datasets (r4 vs r3), validating our hypothesis that compressing features after the vision-to-text module is like summarizing textual features and can maintain more textual semantics while compressing visual features after the visual encoder loses more visually situated text information. Besides, without aligning each query token in the global feature map with R × C fine-grained visual tokens from the re-organized feature map to perform attention within a group as <ref>, we try utilizing each query token to attend all visual tokens of sub-images. Such complete attention not only brings higher computational complexity but also causes performance decreases (r5 vs r3), showing that the positional correspondence between the global visual map and the re-organized fine-grained visual map is a reliable prior knowledge for compressing visual features efficiently. Furthermore, directly performing mean pooling on each group of R × C fine-grained visual features underperforms utilizing global visual features as the query to perform cross-attention (r6 vs r3). This also proves the importance of reliable guidance during compressing.

Compared with 2 layers of cross-attention, decreasing cross-attention layers bring a slight performance increase on DocVQA <cit.> but more performance decrease on WikiTablesQA (WTQ) <cit.> (r7 vs r3). Further increasing to 4 layers doesn't significantly improve performance (r8 vs r3). This shows that compressing high-resolution visual features doesn't require a deep neural network. Finally, increasing the maximum number of crops and the base resolution of the global image or each sub-image are two main strategies to increase the supported input resolution. Our experiments show that increasing the cropping number (r9 vs r3) or basic resolution (r10 vs r9) benefits the document understanding performance. Increasing basic resolution brings more improvement because of more visual tokens after compressing.





  §.§.§ Training Strategy

 is trained with three stages: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. <ref> shows the influence of each stage for OCR-free single-page and multi-page document understanding. With the Single-image Pretraining and Single-image finetuning (r1), the model achieves promising performance on single-page benchmark DocVQA and documents from MP-DocVQA with only 1 page. Although only trained with 1 image as the input, the model can also achieve around 50% accuracy when fed into 2-10 page images. However, the model struggles to understand documents with more than 10 pages, which greatly exceeds the number of input images during training and brings great difficulty in correlating images and finding answers. Performing Multi-image Fintuing could greatly improve the model's ability to understand multiple images (r2 vs r1). Furthermore, adding the Multi-image Continue-pretraining could also improve the question-answering performance on downstream datasets, especially for documents with more than 10 pages (r3 vs r2). This demonstrates that parsing texts of the specified page or judging which pages contain specified texts among multi-page documents is a basic ability for multi-page document understanding. Finally, by ensembling both single-image and multi-image instruction tuning sets to perform the Multi-task Finetuning (r4),  achieves the best performance on both single-page and multi-page document benchmarks, showing the cross-improvement between single-image and multi-image comprehension.



 §.§ Qualitative Results








As shown in <ref>, after the Multi-image Continue Pretraining stage,  is able to locate the corresponding image of the given texts accurately. Besides, although representing each high-resolution image with just 324 tokens,  is still capable of parsing detailed texts of specified two images, validating the promising OCR-free multi-page document understanding performance of  . It also demonstrates our proposal that 324 tokens are enough to encode detailed text information in common A4-sized document pages and the effectiveness of our .

After the Multi-task Finetuning, given multiple images and a question,  can give a simple answer first and then provide a detailed explanation with the evidence, as shown in <ref>.  can comprehend not only page images rendered from PDF files (<ref>(c)) but also scan images of a document (<ref>(a-b)). When a question is unanswerable,  can also tell and give corresponding reasons (<ref>(c)). 

Besides multi-page documents,  is also capable of understanding text-rich videos. As shown in <ref>, among similar frames within a video,  can distinguish fine-grained textual differences, locate relevant frames, and give accurate answers.





§ CONCLUSION

In this work, we propose mPLUG-, a Multimodal Large Language Model with the ability of efficient OCR-free Multi-page Document Understanding. The novel architecture  in  compresses each high-resolution document image into 324 tokens through cross-attention with the global visual feature as guidance, and re-organized features of cropped images as keys and values. On single-image document understanding benchmarks, with fewer visual tokens,  outperforms existing compressing methods and achieves comparable performance with SOTA MLLMs with similar training data. Besides,  achieves OCR-free state-of-the-art performance on two multi-page document understanding benchmarks and 1 text-rich video understanding benchmark. Our experiments validate that thousands of visual tokens for 1 common A4-sized document page may be so redundant that too many computational resources are wasted. We hope  could bring more researchers' attention to the balance of efficient representation of high-resolution images and OCR-free Document Understanding performance.



iclr2024_conference





