LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via Hybrid Architecture
Xidong Wang Dingjie Song Shunian Chen Chen Zhang Benyou Wang


Single Pion Production off Free Nucleons: Analysis of Photon, Electron, Pion and Neutrino Induced Processes
    M. Kabirnezhad
    September 9, 2024
===========================================================================================================





§ ABSTRACT 








Expanding the long-context capabilities of Multi-modal Large Language Models (MLLMs) is crucial for video understanding, high-resolution image understanding, and multi-modal agents. This involves a series of systematic optimizations, including model architecture, data construction and training strategy, particularly addressing challenges such as degraded performance with more images and high computational costs. In this paper, we adapt the model architecture to a hybrid of Mamba and Transformer blocks, approach data construction with both temporal and spatial dependencies among multiple images and employ a progressive training strategy. The released model LongLLaVA (Long-Context Large Language and Vision Assistant) is the first hybrid MLLM, which achieved a better balance between efficiency and effectiveness. LongLLaVA  not only achieves competitive results across various benchmarks, but also maintains high throughput and low memory consumption. Especially, it could process nearly a thousand images on a single A100 80GB GPU, showing promising application prospects for a wide range of tasks.




§ INTRODUCTION


The rapid advancement of MLLMs <cit.> has demonstrated their remarkable capabilities across various applications <cit.>. However, multi-image scenario remain an important yet to-be-explored aspect. In particular, expanding the context of MLLMs to understand longer videos <cit.>, higher-resolution images <cit.>, and make decisions based on more historical messages <cit.> is crucial for enhancing user experience <cit.> and further broadening MLLMs' application scope <cit.>. 


However, extending the context length of MLLMs to improve their usability poses challenges related to degraded performance and high computational costs when processing more images. 

To maintain the performance in longer context, some studies <cit.> have concentrated on curating long-context training data involving multiple images to enhance performance. Additionally, other research efforts have explored innovative training strategies <cit.> to mitigate performance declines.  
Regarding the issue of high computational costs, <cit.> have made strides in improving multi-node efficiency by reducing communication costs. However, there remains a gap in solutions for accelerating the computation itself when managing longer contexts.



To address the challenges mentioned above, we propose a systematic solution called LongLLaVA, especially using a hybrid architecture for acceleration.
This solution comprehensively optimizes across three dimensions:Multi-modal architecture,  Data construction, and Training strategy.

    
  * For Multi-modal architecture, we adopt a hybrid architecture combining Transformer with Mamba and propose an efficient image representation method that applies 2D pooling to compress image tokens, significantly reducing computational costs while maintaining performance. 
    
  * For Data construction, we have designed unique formats for different tasks, enabling the model to distinguish between temporal and spatial dependencies among images.
    
  * In terms of Training strategy, we employ a three-phase method for multi-modal adaptation—Single-image Alignment, Single-image Instruction-tuning, and Multi-image Instruction-tuning—to incrementally enhance the model’s ability to handle multi-modal long contexts. 


 

Experiemntal results show that LongLLaVA excels in understanding multi-modal long contexts with high efficiency. It leads in retrieval, counting, and ordering tasks in VNBench <cit.> and achieves nearly 100% accuracy with 1,000 images on a single 80GB GPU for Needle-In-A-Haystack evaluation <cit.>.


Our summarized contributions are as follows:

    
  * We introduce LongLLaVA, a solution optimized through data construction, training strategies, and multi-modal architecture, effectively balancing performance and efficiency. To the best of our knowledge, this is the first hybrid architecture for MLLMs.
    
  * LongLLaVA demonstrates exceptional performance in multi-modal long-context understanding, excelling in retrieval, counting, and ordering tasks.
    In our commitment to transparency and community research, we will open source all models, codes, and datasets associated with LongLLaVA.





§ TOWARDS SCALING UP THE IMAGE NUMBER IN MLLMS










 §.§ The Curse of Image Numbers



The model's input length increases rapidly with the number of images, leading to issues such as degraded model performance and high inference costs. 

With the evolution of MLLM technology, many existing open-source MLLMs have demonstrated capabilities on par with closed-source models in single-image tasks <cit.>. 
However, when the task expands to multiple images, the capabilities of these models degrade significantly. Whether it is a time-related or semantically-related multi-image task, the models struggle considerably, lagging far behind the performance of closed-source models <cit.>. Such capability degradation severely limits the application scenarios of open-source MLLMs, and the open-source community urgently needs a systematic solution to address this issue.







  
Excessive Input Length 
Aside from the degraded performance, another challenge is managing the excessive input length when processing a larger number of images.
The visual encoder components of MLLMs typically transform each image into a large number of tokens. For instance, the CLIP encoder[openai/clip-vit-base-patch32] <cit.> generates 576 image tokens for an image with a resolution of 336 pixels. Representing a three-minute video at 1 FPS would require 3×60×576=103,680 tokens.
This results in a rapid increase in computational demand and memory usage, significantly degrading user experience and escalating service costs.
Although some works have proposed compressing image tokens in 2D or 3D formats <cit.>, this often comes at the expense of performance. Solely relying on compression methods to accommodate multiple images without substantial performance degradation is insufficient.



  
High Computational and Memory Complexity 
The excessive input length results in high computational and memory complexity.
The LLM part of existing MLLMs primarily relies on Transformer architectures, whose computational complexity grows quadratically with increases in sequence length. As the number of images (or sequence length) increases, memory usage becomes exceptionally high due to the need to store KV-Cache. 
As shown in Figure <ref>, current models on a single 80GB GPU can theoretically handle up to 384 images using  quantization. 
Current efforts mitigate this through techniques like ring attention <cit.> or sequence parallelism <cit.>, but these solutions introduce greater time overhead, or by switching to the Mamba architecture <cit.> to address these challenges. However, the capability for In-Context Learning (ICL) in multi-image scenarios is indispensable, posing a challenge for Mamba models <cit.>. Therefore, there is a need for a balanced approach that considers model architecture in multimodal contexts, addressing both the high number of image tokens and reducing computational complexity.



 §.§ Motivation for Hybrid Architecture




Effectiveness-efficiency Balance 
To balance effectiveness and efficiency in multimodal long-context scenarios, we propose to use a Mamba-Transformer hybrid architecture.



As shown in Table <ref>, the Mamba-Transformer hybrid architecture <cit.> is distinguished by its superior performance on ICL task and other benchmark tests, matching the performance of Mixtral-8x7B <cit.>.
In terms of efficiency,  the Mamba-Transformer hybrid  architecture is more efficient  due to Mamba's linear computational complexity. For example,  a popular hybrid architecture Jamba's throughput is typically three times that of Mixtral, which already have utilized sliding window techniques to reduce inference complexity. Jamba only requires  of KV-Cache memory to process a 256K token sequence under  activation parameters with  precision, substantially less than comparable models. Moreover, in scenarios using a single A100 80GB GPU with  precision, Jamba can handle up to 140K tokens in context length, significantly surpassing the 80K of LLaMA2-7B <cit.> and the 60K of Mixtral 8×7B.












 §.§ The Benefit of Scaling Up the Image Number


Adopting more  images significantly broadens the application scenarios for current MLLMs. We will explore this from two dimensions: Temporal Expansion and Spatial Expansion.

Temporal Expansion.
Understanding the temporal dependencies between images is crucial for a variety of applications. In multi-modal assistants, it enhances real-time recall capabilities, which is particularly beneficial for the elderly <cit.>. For mobile agents, it enables more personalized services and improves task planning <cit.>. In the healthcare sector, it assists in anomaly detection in 3D medical videos, thereby reducing diagnostic errors <cit.>.


Spatial Expansion. 
When dealing with high-resolution images <cit.> or when detailed understanding of images <cit.> is required, images are often decomposed into sub-images. This process highlights the importance of grasping spatial dependencies among these sub-images. In remote sensing, an increased number of images enhances both coverage and granularity <cit.>. In pathology, it minimizes information loss and improves diagnostic accuracy <cit.>. In the field of Molecular Learning, it facilitates the processing of complex reactions and the analysis of larger molecular graphs <cit.>.




§ LONGLLAVA: SCALING LLAVA TO LONGER CONTEXT


To address the aforementioned challenges and enhance the model's adaptability to long-context, multi-image scenarios, we introduce improvements from three perspectives: multi-modal model architecture (Sec. <ref>), data processing protocol (Sec. <ref>), and training strategy (Sec. <ref>).




 §.§ Multi-modal Architecture



Our multimodal architecture is constructed around three core components inspired by LLaVA <cit.>: the Vision Encoder, the Projector, and the LLM.




Vision Information Processing.
We employ CLIP[] as the vision encoder to encode visual information and a two-layer MLP as the projector to map vision features into the text embedding space suitable for the LLM. Prior to projection, bilinear pooling is applied, reducing the token representation of an image from 576 to 144 by aggregating 2×2 patch units into a single token. This approach effectively conserves training and inference time while maintaining essential spatial relationships between patches. Further details on the effectiveness of this strategy are provided in Appendix <ref>.

Hybrid LLM Architecture.
Our model employs a hybrid LLM architecture that integrates Transformer and Mamba layers in a 7:1 ratio, as depicted in Figure <ref>. It also features a Mixture of Experts (MoE) approach in every other layer, utilizing 16 experts and selecting the top-2 experts for each token. RMSNorm <cit.> is used between layers to enhance normalization, although positional embeddings are omitted. The model incorporates Grouped Query Attention (GQA) <cit.> and SwiGLU activation functions <cit.>, similar to other large language models. The total parameter count of the model is , with activation parameters during inference totaling .








 §.§ Data Processing Protocol
 





To ensure that the model effectively distinguishes between temporal and spatial dependencies among images in multi-image scenarios and performs well across various tasks, we meticulously differentiated special characters in different scenarios. As shown in Figure <ref>, these special characters comprehensively address the various relationships between images in different contexts, thereby enhancing the model's adaptability to diverse tasks.





Regular Single and Multiple Images: For regular single and multiple image inputs, we use  and  to enclose image tokens, helping the model differentiate between image and text tokens.

Video: For video inputs, to enable the model to understand the temporal relationship between frames, we first use  and  to enclose image tokens. Additionally, we add the special symbol  between different frames to represent the temporal dependency between them.

High Resolution Image: For complex single-image understanding that need to divide image into multiple sub-images, we use  to separate the main image from its sub-images. For the arrangement of sub-images, we traverse from the top-left to the bottom-right, adding  between split lines to preserve the relative spatial positions of the sub-images.




 §.§ Training Strategy



In our training strategy, we implement single-modal and multi-modal adaptations to transform a pre-trained language model into a multimodal long-context model.


















  
Pure-text Instruction-tuning.
We initially enhance the pre-trained language model's ability to follow instructions of varying lengths in pure-text contexts. This is achieved using a comprehensive dataset totaling 278k pure text entries from Evol-instruct-GPT4 <cit.>, WildChat <cit.>, and LongAlign <cit.>.

For multi-modal adaptation, following the Single-image Alignment and Single-image Instruction-tuning stages in LLaVA <cit.>, we introduce a Multi-image Instruction-tuning stage to progressively enhance the model's long-context capabilities.

We adopt progressive training not only for better control of variables but also to increase model reusability <cit.>. The specific dataset usage is detailed in Figure <ref>.



  
Stage I: Single-image Alignment.
This stage is to align visual modal features with textual modality. We utilize datasets such as ALLaVA-Caption <cit.> and ShareGPT4V <cit.>, which comprise approximately 600K high-quality image-caption pairs. During this phase, only the projector is trained while freezing the parameters of the Visual Encoder and LLM.



  
Stage II: Single-image Instruction-tuning.
This stage aims to endow the model with multimodal instruction-following capabilities. We use datasets like LLaVA-1.5 <cit.> and Mantis-Single <cit.>, totaling around 932K high-quality question-answer pairs. Here, only the Visual Encoder is frozen, and the projector and LLM parts are trained. This process ultimately results in the development of LongLLaVA (single image).




  
Stage III: Multi-image Instruction-tuning.
In this stage, the model is trained to follow instructions in multimodal long-context scenarios. We sample 200K, 200K and 50K data items from Mantis <cit.>, VideoChat2 <cit.> and ShareGPT4Video <cit.> respectively. To preserve the model’s single-image comprehension and pure-text dialogue capabilities, we include an additional 200K and 50K data items from the Single-image Instruction-tuning and Pure-text Instruction-tuning phases as the  component. Furthermore, to enhance the model's ability to interpret complex single images segmented into multiple sub-images, we extract 50K data items from the Single-image Instruction-tuning phase, perform padding and segmentation, and divide the original images into sub-images of size 336×336 as the  component. This culminate in the development of LongLLaVA.
,
For multi-modal adaptation, fo



















§ EXPERIMENTS





 §.§ Training Details.
 



For training, we utilize random sampling to concatenate data items into a token length of 40,960, separated by the  token. This approach helps in managing extensive datasets and ensuring diverse coverage of different data segments. Training is executed across three compute nodes, each equipped with eight A800 GPUs, leveraging DeepSpeed Zero-3 as the distributed strategy to enhance scalability and efficiency. We employ a cosine learning rate scheduler with a warm-up rate of , set the training epoch to , and the learning rate to . This configuration is designed to balance learning speed and model convergence effectively. .



 §.§ Evaluation Setup

We evaluate the model's multimodal long-context understanding capabilities using the multi-image evaluation and conduct single-image evaluations to explore basic functionalities. For details on the single-image evaluation, please refer to Appendix <ref>. Both LongLLaVA (single image) and LongLLaVA are evaluated using  quantization with the temperature set to zero to ensure consistency in performance evaluation. The use of  quantization helps in reducing the model size and computational load while maintaining performance accuracy.



  
Benchmarks.
Our evaluation utilizes three multi-image benchmarks: MileBench <cit.> for assessing multimodal long-context scenario performance, and Video-MME <cit.> along with MVBench <cit.> for video analysis capabilities. Detailed descriptions of these benchmarks are available in Appendix <ref>.



  
Baselines.
We compare our model against four commercial models: GPT-4V[] <cit.>, GPT-4o[], Gemini-1.5-Pro[] <cit.>, and Claude3-Opus[], 
as well as five open-source models: Phi-3-Vision[], OmChat <cit.>, LongVILA <cit.>, Video-LLaMA-2 <cit.> and VideoChat2 <cit.>. 




 §.§ Main Results








As shown in Table <ref>, LongLLaVA demonstrates superior performance among open-source models on MileBench, even surpassing Claude3-Opus, and particularly excels in retrieval tasks. This highlights LongLLaVA's impressive capabilities in handling multi-image tasks. Notably, LongLLaVA's effectiveness is further underscored by its performance on video benchmarks such as Video-MME and MVBench. It shows exceptional results, especially in tasks involving medium to long-length videos, outperforming traditional video models like Video-LLaMA2 and VideoChat2.

Remarkably, despite achieving these impressive results, LongLLaVA operates with an order of magnitude fewer FLOPs compared to other models. This efficiency in computational resources not only underscores LongLLaVA's advanced performance but also its optimization in resource management. These results reflect a significant advancement in the research community's efforts to close the performance gap with commercial models.






 §.§ Diagnostic Evaluation of Long-Context MLLMs













Considering that former evaluations cannot adequately capture the abilities of MLLMs over long contexts, we employ a new diagnostic evaluation set, VNBench <cit.>, to further analyze the atomic capabilities of models in long contexts. VNBench is a benchmark construction framework based on synthetic video generation, encompassing tasks such as retrieval, ordering, and counting.

The results, as presented in Table <ref>, indicate that LongLLaVA exhibits performance that is on par with leading closed-source models in tasks such as cross-context retrieval, ordering, and technical capabilities, even outperforms GPT-4V. Among open-source models, LongLLaVA also shows its superior performance. This positions LongLLaVA as a prominent contender in the field, demonstrating its advanced capabilities in managing and interpreting long contexts.

















































 §.§ Ablation Study

































As shown in Table <ref>, significant improvements were observed across all evaluation sets when using the hybrid LLM architecture, Jamba, with identical data and model parameters, demonstrating its potential in multimodal scenarios. For token compression, we choose the 2D pooling, which significantly reduces computational load while keeping performance degradation within acceptable limits (less than 2.2%). Compared to 1D pooling, the 2D pooling method, which pools along the height and width directions to obtain a 12x12 token arrangement, yields better results (0.1∼1.5 improvement). Regarding data construction, after training on our single-image data, the model achieved a 1.5% accuracy improvement on SEEDBench and 12.3% on MileBench. Subsequent multi-image training led to a further 7.4% increase on MileBench, validating the dataset construction's effectiveness.








§ MORE ANALYSIS

In this section, we conduct further analysis to understand the inner workings and multimodal long-context capability of LongLLaVA.




 §.§ On the Motivation for the Hybrid Architecture.


















We explore the strengths and weaknesses of different architectures in terms of ICL capabilities and inference efficiency, highlighting the balanced advantages of multimodal hybrid architectures that combine the strengths of both. For Mamba, we select Cobra <cit.>, which, to the best of our knowledge, is the first and only multimodal architecture to use Mamba as its LLM within the LLaVA framework. For Transformer, we choose the 13B parameter LLaVA-1.6, which has inference parameters consistent with LongLLaVA, to enable a more accurate efficiency comparison.

ICL Analysis. We evaluate the performance on the Matching Image task from VL-ICL benchmark <cit.> for multi-modal in-context learning. This task's inputs contain an image pair x={x1, x2}, and output y
indicates whether a specific relation r holds between them. MLLMs are required to learn the relation
from examples. As shown in the Table <ref>, both Hybrid and Transformer architectures exhibit rapid performance improvements with the increase in examples, whereas the Mamba architecture shows a slower improvement, confirming its ICL shortcomings.

Efficiency Analysis. We focus on three aspects: Prefill Time (first inference latency), Throughput (next tokens per second), and Memory Usage. We control the input text length to 100K and measure time and maximum memory usage for generating outputs of 1 token and 1000 tokens. Throughput is calculated as (1000-1)/(time_1000-time_1). To better simulate real application scenario, Transformer and Hybrid architectures are evaluated using vLLM framework <cit.> with  quantization <cit.>. As shown in the Table <ref>,  Mamba architecture has the fastest Prefill Time, highest throughput. The Hybrid architecture achieves 2.5 times the Throughput, 75% of the Prefill Time, and reduced memory usage compared to the Transformer architecture with similar inference parameters.








 §.§ Scaling Law of the Image Number


With more images processed, it could support more image patches for high-resolution image understanding and more video frames for video understanding. To explore the impact of increasing the number of sub-images and video frames, we evaluate LongLLaVA on the benchmarks V* Bench <cit.> and Video-MME <cit.> respectively.








Scale up Number of SubImages. V* Bench evaluates a model's ability to locate small objects within large images. As shown in Figure <ref>, increasing the number of sub-images initially improves model performance significantly, indicating better understanding of image details. However, we also find that further increasing the number of sub-images slightly degraded the performance, suggesting that an excessive number of sub-images may interfere with performance on this task.

Scale up Number of Frames. Video-MME <cit.> is a benchmark that tests a model's ability to extract information from videos. We can see from Figure <ref> that as the number of sampled frames increases, the model's performance on the benchmark improves significantly, reaching its peak when 256 frames are extracted. This indicates that the model can effectively understand and utilize the information contained in the additionally sampled frames to provide better responses.








§ FURTHER SCALING UP THE IMAGE NUMBER  TO 1000








Using the V-NIAH evaluation framework proposed in LongVA <cit.>, we conduct a needle-in-the-haystack test to evaluate the model’s performance. Given the model’s training sequence length limit of 40,960 tokens, we apply token pooling techniques to reduce the original token count from 144 to 36. This adjustment allows us to test the model’s ability to retrieve relevant information from a large dataset efficiently. As shown in Figure <ref>, LongLLaVA achieves nearly 100% retrieval accuracy on a set of 1000 images without requiring additional training.



However, when we increase the number of images in the test more than 1,000, we observe a decline in retrieval accuracy. This drop in performance may be due to exceeding the model’s training sequence length, which potentially affects its ability to maintain accuracy with more images. In future work, we will extend the training sequence length to 140,000 tokens, the limit for single-card inference with LongLLaVA, to further unlock the model's potential.


































§ CONCLUSION




In this study, we introduce LongLLaVA (Long Context Large Language and Visual Assistant), an innovative hybrid architecture model that excels in long-context multi-modal understanding. The model integrates Mamba and Transformer blocks, leveraging temporal and spatial dependencies between multiple images to construct data, and employs a progressive training strategy. LongLLaVA demonstrates competitive performance across various benchmarks while ensuring efficiency, setting a new standard for long-context Multi-modal Large Language Models (MLLMs).































































iclr2024_conference











§ APPENDIX TABLE OF CONTENTS

[sections]
[sections]l1



§ DETAILS OF BENCHMARKS












  
Single-image Benchmarks.
We select seven commonly used evaluations to assess the model's single-image understanding capabilities. These include:

    
  * GQA <cit.>: A benchmark for real-world visual reasoning and compositional question answering.
    
  * MME <cit.>: A comprehensive benchmark focused on perception and cognition, from which we use the perception component.
    
  * MM-Vet <cit.>: Examines six core visual-linguistic (VL) capabilities and sixteen integrations derived from these capabilities.
    
  * ScienceQA <cit.>: Consists of 4,210 questions across various science topics, with detailed annotations.
    
  * SEED-Bench-v1 <cit.>: Evaluates comprehension across twelve dimensions in both image and video modalities; we use the image set.
    
  * MMBench <cit.>: A systematically-designed benchmark across twenty ability dimensions.
    
  * MMMU <cit.>: Tests multi-modal models on multidisciplinary tasks requiring university-level knowledge, covering 183 subfields and 30 types of images.




  
Multi-image Benchmarks.
To explore multi-image capabilities, we utilized:

    
  * MileBench <cit.>: Assesses long-context scenario performance, focusing on Temporal, Semantic, and Information Retrieval (IR) components.
    
  * Video-MME <cit.>: Covers 30 sub-fields to evaluate video analysis capabilities. We analyze 128 frames extracted uniformly from each video, independent of subtitles.
    
  * MVBench <cit.>: Addresses 20 challenging video tasks that are not effectively solved with a single frame.








§ DETAILS OF SINGLE-IMAGE EVALUATION



The single-image evaluation aims to explore the model's fundamental capabilities and the impact of extended long-context training on single-image understanding. 



 §.§ Benchmarks

We utilized a series of benchmarks, including GQA <cit.>, MME <cit.>, MM-Vet <cit.>, ScienceQA <cit.>, SEED-Bench-v1 <cit.>, MMBench <cit.>, and MMMU <cit.>. These benchmarks assess various aspects of visual understanding and cognitive processing within a single-image context.



 §.§ Comparison Models

For a comprehensive comparison, we selected three commercial models for the single-image evaluation: GPT-4V[] <cit.>, Gemini-1.5[] <cit.>, and Claude3-Opus[]. 
Additionally, we included five open-source models to broaden the scope of evaluation: LLaVA-1.5-13B <cit.>, LLaVA-1.6-13B <cit.>, Phi-3-Vision-4.2B[] and OmChat-8B <cit.>.






 §.§ Results Analysis

As shown in Table <ref>, LongLLaVA (single image) generally outperforms LLaVA-1.6-13B, despite both models having the same inference parameter size. This advantage is particularly notable in the MMMU benchmarks, highlighting LongLLaVA (single image)'s strengths in handling comprehensive knowledge-based questions. Although LongLLaVA (single image)'s performance is slightly lower compared to some recently emerged high-performing models, it still demonstrates the potential of hybrid architectures in multi-model scenarios. To ensure complete reproducibility of our results, we only focus on four representative public datasets. Additionally, we find that LongLLaVA tends to underperform relative to LongLLaVA (single image). Addressing this issue may require incorporating more single-image data during the Multi-image Instruction-tuning phase.


