Update README.md

README.md

@@ -9,19 +9,21 @@ language:
 
 Below is the model card of LLaVa-NeXT-Video model 7b, which is copied from the original Llava model card that you can find [here](https://huggingface.co/liuhaotian/llava-v1.5-13b).
 
-Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing)
+Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CZggLHrjxMReG-FNOmqSOdi4z7NPq6SO?usp=sharing)
 
-Or check out our Spaces demo! [![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-md-dark.svg)](https://huggingface.co/spaces/llava-hf/llava-4bit)
+Disclaimer: The team releasing LLaVa-NeXT-Video did not write a model card for this model so this model card has been written by the Hugging Face team.
 
-
-## Model details
+## 📄 Model details
 
 **Model type:**
 <br>
-LLaVA-Next-Video is an open-source chatbot trained by fine-tuning LLM on multimodal instruction-following data.
+LLaVA-Next-Video is an open-source chatbot trained by fine-tuning LLM on multimodal instruction-following data. The model is buit on top of LLaVa-NeXT by tuning on a mix of video and image data. The videos were sampled uniformly to be 32 frames per clip.
 <br>
  Base LLM: lmsys/vicuna-7b-v1.5
 
+<img src="http://drive.google.com/uc?export=view&id=1fVg-r5MU3NoHlTpD7_lYPEBWH9R8na_4">
+
+
 **Model date:**
 <br>
 LLaVA-Next-Video-7B was trained in April 2024.
@@ -31,7 +33,24 @@ LLaVA-Next-Video-7B was trained in April 2024.
 https://github.com/LLaVA-VL/LLaVA-NeXT
 
 
-## How to use the model
+## 📚 Training dataset
+
+### Image
+- 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP.
+- 158K GPT-generated multimodal instruction-following data.
+- 500K academic-task-oriented VQA data mixture.
+- 50K GPT-4V data mixture.
+- 40K ShareGPT data.
+
+### Video
+- 100K VideoChatGPT-Instruct.
+
+## 📊 Evaluation dataset
+A collection of 4 benchmarks, including 3 academic VQA benchmarks and 1 captioning benchmark.
+
+
+
+## 🚀 How to use the model
 
 First, make sure to have `transformers >= 4.42.0`. 
 The model supports multi-visual and multi-prompt generation. Meaning that you can pass multiple images/videos in your prompt. Make sure also to follow the correct prompt template (`USER: xxx\nASSISTANT:`) and add the token `<image>` or `<video>` to the location where you want to query images/videos:
@@ -39,17 +58,12 @@ The model supports multi-visual and multi-prompt generation. Meaning that you ca
 Below is an example script to run generation in `float16` precision on a GPU device:
 
 ```python
-import requests
-from PIL import Image
 import av
 import torch
 from transformers import LlavaNextVideoProcessor, LlavaNextVideoForConditionalGeneration
 
 model_id = "llava-hf/LLaVA-NeXT-Video-7B-hf"
 
-prompt = "USER: <image>\nWhat are these?\nASSISTANT:"
-image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
-
 model = LlavaNextVideoForConditionalGeneration.from_pretrained(
     model_id, 
     torch_dtype=torch.float16, 
@@ -82,7 +96,7 @@ prompt = "USER: <video>\nWhy is this video funny? ASSISTANT:"
 video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
 container = av.open(video_path)
 
-# sample uniformly 8 frames from the video
+# sample uniformly 8 frames from the video, can sample more for longer videos
 total_frames = container.streams.video[0].frames
 indices = np.arange(0, total_frames, total_frames / 8).astype(int)
 clip = read_video_pyav(container, indices)
@@ -97,6 +111,12 @@ print(processor.decode(output[0][2:], skip_special_tokens=True))
 To generate from images use the below code after loading the model as shown above:
 
 ```python
+import requests
+from PIL import Image
+
+prompt = "USER: <image>\nWhat are these?\nASSISTANT:"
+image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
+
 raw_image = Image.open(requests.get(image_file, stream=True).raw)
 inputs_image = processor(prompt, images=raw_image, return_tensors='pt').to(0, torch.float16)
 
@@ -149,11 +169,12 @@ model = LlavaNextVideoForConditionalGeneration.from_pretrained(
 ).to(0)
 ```
 
-## License
+
+## 🔒 License
 Llama 2 is licensed under the LLAMA 2 Community License, 
 Copyright (c) Meta Platforms, Inc. All Rights Reserved.
 
-## Intended use
+## 🎯 Intended use
 **Primary intended uses:**
 <br>
 The primary use of LLaVA is research on large multimodal models and chatbots.
@@ -162,20 +183,27 @@ The primary use of LLaVA is research on large multimodal models and chatbots.
 <br>
 The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
 
-## Training dataset
 
-### Image
-- 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP.
-- 158K GPT-generated multimodal instruction-following data.
-- 500K academic-task-oriented VQA data mixture.
-- 50K GPT-4V data mixture.
-- 40K ShareGPT data.
-
-### Video
-- 100K VideoChatGPT-Instruct.
-
-## Evaluation dataset
-A collection of 4 benchmarks, including 3 academic VQA benchmarks and 1 captioning benchmark.
+## ✏️ Citation
+If you find our paper and code useful in your research:
 
+```BibTeX
+@misc{zhang2024llavanextvideo,
+  title={LLaVA-NeXT: A Strong Zero-shot Video Understanding Model},
+  url={https://llava-vl.github.io/blog/2024-04-30-llava-next-video/},
+  author={Zhang, Yuanhan and Li, Bo and Liu, haotian and Lee, Yong jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan},
+  month={April},
+  year={2024}
+}
+```
 
+```BibTeX
+@misc{liu2024llavanext,
+    title={LLaVA-NeXT: Improved reasoning, OCR, and world knowledge},
+    url={https://llava-vl.github.io/blog/2024-01-30-llava-next/},
+    author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae},
+    month={January},
+    year={2024}
+}
+```