wren93 commited on
Commit
b452323
·
verified ·
1 Parent(s): 938b37e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -2
README.md CHANGED
@@ -17,12 +17,11 @@ This repo contains model checkpoints for **Vamba-Qwen2-VL-7B**. Vamba is a hybri
17
  The main computation overhead in the transformer-based LMMs comes from the quadratic complexity of the self-attention in the video tokens. To overcome this issue, we design a hybrid Mamba Transformer architecture to process text and video tokens differently. The key idea of our method is to split the expensive self-attention operation over the entire video and text token sequence into two more efficient components. Since video tokens typically dominate the sequence while text tokens remain few, we maintain the self-attention mechanism exclusively for the text tokens and eliminate it for the video tokens. Instead, we add cross-attention layers that use text tokens as queries and video tokens as keys and values. In the meantime, we propose employing Mamba blocks to effectively process the video tokens.
18
 
19
 
20
- # Quick Start
21
  ```python
22
  # git clone https://github.com/TIGER-AI-Lab/Vamba
23
  # cd Vamba
24
  # export PYTHONPATH=.
25
-
26
  from tools.vamba_chat import Vamba
27
  model = Vamba(model_path="TIGER-Lab/Vamba-Qwen2-VL-7B", device="cuda")
28
  test_input = [
 
17
  The main computation overhead in the transformer-based LMMs comes from the quadratic complexity of the self-attention in the video tokens. To overcome this issue, we design a hybrid Mamba Transformer architecture to process text and video tokens differently. The key idea of our method is to split the expensive self-attention operation over the entire video and text token sequence into two more efficient components. Since video tokens typically dominate the sequence while text tokens remain few, we maintain the self-attention mechanism exclusively for the text tokens and eliminate it for the video tokens. Instead, we add cross-attention layers that use text tokens as queries and video tokens as keys and values. In the meantime, we propose employing Mamba blocks to effectively process the video tokens.
18
 
19
 
20
+ ## Quick Start
21
  ```python
22
  # git clone https://github.com/TIGER-AI-Lab/Vamba
23
  # cd Vamba
24
  # export PYTHONPATH=.
 
25
  from tools.vamba_chat import Vamba
26
  model = Vamba(model_path="TIGER-Lab/Vamba-Qwen2-VL-7B", device="cuda")
27
  test_input = [