Update README.md

README.md

@@ -17,7 +17,45 @@ This repo contains model checkpoints for **Vamba-Qwen2-VL-7B**. Vamba is a hybri
 The main computation overhead in the transformer-based LMMs comes from the quadratic complexity of the self-attention in the video tokens. To overcome this issue, we design a hybrid Mamba Transformer architecture to process text and video tokens differently. The key idea of our method is to split the expensive self-attention operation over the entire video and text token sequence into two more efficient components. Since video tokens typically dominate the sequence while text tokens remain few, we maintain the self-attention mechanism exclusively for the text tokens and eliminate it for the video tokens. Instead, we add cross-attention layers that use text tokens as queries and video tokens as keys and values. In the meantime, we propose employing Mamba blocks to effectively process the video tokens.
 
 
+# Quick Start
+```python
+# git clone https://github.com/TIGER-AI-Lab/Vamba
+# cd Vamba
+# export PYTHONPATH=.
 
+from tools.vamba_chat import Vamba
+model = Vamba(model_path="TIGER-Lab/Vamba-Qwen2-VL-7B", device="cuda")
+test_input = [
+    {
+        "type": "video",
+        "content": "assets/magic.mp4",
+        "metadata": {
+            "video_num_frames": 128,
+            "video_sample_type": "middle",
+            "img_longest_edge": 640,
+            "img_shortest_edge": 256,
+        }
+    },
+    {
+        "type": "text",
+        "content": "<video> Describe the magic trick."
+    }
+]
+print(model(test_input))
+
+test_input = [
+    {
+        "type": "image",
+        "content": "assets/old_man.png",
+        "metadata": {}
+    },
+    {
+        "type": "text",
+        "content": "<image> Describe this image."
+    }
+]
+print(model(test_input))
+```
 
 ## Citation
 If you find our paper useful, please cite us with